On exploring efficient shuffle design for in-memory mapreduce

Harunobu Daikoku, Hideyuki Kawashima, Osamu Tatebe

研究成果: Conference contribution

4 被引用数 (Scopus)

抄録

MapReduce is commonly used as a way of big data analysis in many fields. Shuffling, the inter-node data exchange phase of MapReduce, has been reported as the major bottleneck of the framework. Acceleration of shuffling has been studied in literature, and we raise two questions in this paper. The first question pertains to the effect of Remote Direct Memory Access (RDMA) on the performance of shuffling. RDMA enables one machine to read and write data on the local memory of another and has been known to be an efficient data transfer mechanism. Does the pure use of RDMA affect the performance of shuffling? The second question is the data transfer algorithm to use. There are two types of shuffling algorithms for the conventional MapReduce implementations: Fully-Connected and more sophisticated algorithms such as Pairwise. Does the data transfer algorithm affect the performance of shuffling? To answer these questions, we designed and implemented yet another MapReduce system from scratch in C/C++ to gain the maximum performance and to reserve design flexibility. For the first question, we compared RDMA shuffling based on rsocket with the one based on IPoIB. The results of experiments with GroupBy showed that RDMA accelerates map+shuffle phase by around 50%. For the second question, we first compared our in-memory system with Apache Spark to investigate whether our system performed more efficiently than the existing system. Our system demonstrated performance improvement by a factor of 3.04 on Word Count, and by a factor of 2.64 on BiGram Count as compared to Spark. Then, we compared the two data exchange algorithms, Fully-Connected and Pairwise. The results of experiments with BiGram Count showed that Fully-Connected without RDMA was 13% more efficient than Pairwise with RDMA. We conclude that it is necessary to overlap map and shuffle phases to gain performance improvement. The reason of the relatively small percentage of improvement can be attributed to the time-consuming insertions of key-value pairs into the hash-map in the map phase.

本文言語English
ホスト出版物のタイトルProceedings of the 3rd ACM SIGMOD Workshop on Algorithms and Systems for MapReduce and Beyond, BeyondMR 2016
出版社Association for Computing Machinery
ISBN(印刷版)9781450343114
DOI
出版ステータスPublished - 2016 6 26
外部発表はい
イベント3rd ACM SIGMOD Workshop on Algorithms and Systems for MapReduce and Beyond, BeyondMR 2016, co-located with ACM SIGMOD 2016 - San Francisco, United States
継続期間: 2016 6 262016 7 1

出版物シリーズ

名前Proceedings of the ACM SIGMOD International Conference on Management of Data
ISSN(印刷版)0730-8078

Other

Other3rd ACM SIGMOD Workshop on Algorithms and Systems for MapReduce and Beyond, BeyondMR 2016, co-located with ACM SIGMOD 2016
国/地域United States
CitySan Francisco
Period16/6/2616/7/1

ASJC Scopus subject areas

  • ソフトウェア
  • 情報システム

フィンガープリント

「On exploring efficient shuffle design for in-memory mapreduce」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル