Skew-aware collective communication for MapReduce shuffling

Harunobu Daikoku, Hideyuki Kawashima, Osamu Tatebe

Research output: Contribution to journalArticle

Abstract

This paper proposes and examines the three in-memory shuffling methods designed to address problems in MapReduce shuffling caused by skewed data. Coupled Shuffle Architecture (CSA) employs a single pairwise all-to-all exchange to shuffle both blocks, units of shuffle transfer, and meta-blocks, which contain the metadata of corresponding blocks. Decoupled Shuffle Architecture (DSA) separates the shuffling of meta-blocks and blocks, and applies different all-to-all exchange algorithms to each shuffling process, attempting to mitigate the impact of stragglers in strongly skewed distributions. Decoupled Shuffle Architecture with Skew-Aware Meta-Shuffle (DSA w/ SMS) autonomously determines the proper placement of blocks based on the memory consumption of each worker process. This approach targets extremely skewed situations where some worker processes could exceed their node memory limitation. This study evaluates implementations of the three shuffling methods in our prototype in-memory MapReduce engine, which employs high performance interconnects such as InfiniBand and Intel Omni-Path. Our results suggest that DSA w/ SMS is the only viable solution for extremely skewed data distributions. We also present a detailed investigation of the performance of CSA and DSA in various skew situations.

Original languageEnglish
Pages (from-to)2389-2399
Number of pages11
JournalIEICE Transactions on Information and Systems
VolumeE102D
Issue number12
DOIs
Publication statusPublished - 2019 Jan 1

Fingerprint

Data storage equipment
Communication
Metadata
Engines

Keywords

  • Intel Omni-Path
  • Libfabric
  • MapReduce
  • Shuffle
  • Skew

ASJC Scopus subject areas

  • Software
  • Hardware and Architecture
  • Computer Vision and Pattern Recognition
  • Electrical and Electronic Engineering
  • Artificial Intelligence

Cite this

Skew-aware collective communication for MapReduce shuffling. / Daikoku, Harunobu; Kawashima, Hideyuki; Tatebe, Osamu.

In: IEICE Transactions on Information and Systems, Vol. E102D, No. 12, 01.01.2019, p. 2389-2399.

Research output: Contribution to journalArticle

Daikoku, Harunobu ; Kawashima, Hideyuki ; Tatebe, Osamu. / Skew-aware collective communication for MapReduce shuffling. In: IEICE Transactions on Information and Systems. 2019 ; Vol. E102D, No. 12. pp. 2389-2399.
@article{cfec0170b5c147a38dd801aaa0ebc443,
title = "Skew-aware collective communication for MapReduce shuffling",
abstract = "This paper proposes and examines the three in-memory shuffling methods designed to address problems in MapReduce shuffling caused by skewed data. Coupled Shuffle Architecture (CSA) employs a single pairwise all-to-all exchange to shuffle both blocks, units of shuffle transfer, and meta-blocks, which contain the metadata of corresponding blocks. Decoupled Shuffle Architecture (DSA) separates the shuffling of meta-blocks and blocks, and applies different all-to-all exchange algorithms to each shuffling process, attempting to mitigate the impact of stragglers in strongly skewed distributions. Decoupled Shuffle Architecture with Skew-Aware Meta-Shuffle (DSA w/ SMS) autonomously determines the proper placement of blocks based on the memory consumption of each worker process. This approach targets extremely skewed situations where some worker processes could exceed their node memory limitation. This study evaluates implementations of the three shuffling methods in our prototype in-memory MapReduce engine, which employs high performance interconnects such as InfiniBand and Intel Omni-Path. Our results suggest that DSA w/ SMS is the only viable solution for extremely skewed data distributions. We also present a detailed investigation of the performance of CSA and DSA in various skew situations.",
keywords = "Intel Omni-Path, Libfabric, MapReduce, Shuffle, Skew",
author = "Harunobu Daikoku and Hideyuki Kawashima and Osamu Tatebe",
year = "2019",
month = "1",
day = "1",
doi = "10.1587/transinf.2019PAP0019",
language = "English",
volume = "E102D",
pages = "2389--2399",
journal = "IEICE Transactions on Information and Systems",
issn = "0916-8532",
publisher = "Maruzen Co., Ltd/Maruzen Kabushikikaisha",
number = "12",

}

TY - JOUR

T1 - Skew-aware collective communication for MapReduce shuffling

AU - Daikoku, Harunobu

AU - Kawashima, Hideyuki

AU - Tatebe, Osamu

PY - 2019/1/1

Y1 - 2019/1/1

N2 - This paper proposes and examines the three in-memory shuffling methods designed to address problems in MapReduce shuffling caused by skewed data. Coupled Shuffle Architecture (CSA) employs a single pairwise all-to-all exchange to shuffle both blocks, units of shuffle transfer, and meta-blocks, which contain the metadata of corresponding blocks. Decoupled Shuffle Architecture (DSA) separates the shuffling of meta-blocks and blocks, and applies different all-to-all exchange algorithms to each shuffling process, attempting to mitigate the impact of stragglers in strongly skewed distributions. Decoupled Shuffle Architecture with Skew-Aware Meta-Shuffle (DSA w/ SMS) autonomously determines the proper placement of blocks based on the memory consumption of each worker process. This approach targets extremely skewed situations where some worker processes could exceed their node memory limitation. This study evaluates implementations of the three shuffling methods in our prototype in-memory MapReduce engine, which employs high performance interconnects such as InfiniBand and Intel Omni-Path. Our results suggest that DSA w/ SMS is the only viable solution for extremely skewed data distributions. We also present a detailed investigation of the performance of CSA and DSA in various skew situations.

AB - This paper proposes and examines the three in-memory shuffling methods designed to address problems in MapReduce shuffling caused by skewed data. Coupled Shuffle Architecture (CSA) employs a single pairwise all-to-all exchange to shuffle both blocks, units of shuffle transfer, and meta-blocks, which contain the metadata of corresponding blocks. Decoupled Shuffle Architecture (DSA) separates the shuffling of meta-blocks and blocks, and applies different all-to-all exchange algorithms to each shuffling process, attempting to mitigate the impact of stragglers in strongly skewed distributions. Decoupled Shuffle Architecture with Skew-Aware Meta-Shuffle (DSA w/ SMS) autonomously determines the proper placement of blocks based on the memory consumption of each worker process. This approach targets extremely skewed situations where some worker processes could exceed their node memory limitation. This study evaluates implementations of the three shuffling methods in our prototype in-memory MapReduce engine, which employs high performance interconnects such as InfiniBand and Intel Omni-Path. Our results suggest that DSA w/ SMS is the only viable solution for extremely skewed data distributions. We also present a detailed investigation of the performance of CSA and DSA in various skew situations.

KW - Intel Omni-Path

KW - Libfabric

KW - MapReduce

KW - Shuffle

KW - Skew

UR - http://www.scopus.com/inward/record.url?scp=85076429499&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85076429499&partnerID=8YFLogxK

U2 - 10.1587/transinf.2019PAP0019

DO - 10.1587/transinf.2019PAP0019

M3 - Article

AN - SCOPUS:85076429499

VL - E102D

SP - 2389

EP - 2399

JO - IEICE Transactions on Information and Systems

JF - IEICE Transactions on Information and Systems

SN - 0916-8532

IS - 12

ER -