Accelerating spark RDD operations with local and remote GPU devices

Yasuhiro Ohno, Shin Morishima, Hiroki Matsutani

研究成果: Conference contribution

16 被引用数 (Scopus)

抄録

Apache Spark is a distributed processing framework for large-scale data sets, where intermediate data sets are represented as RDDs (Resilient Distributed Datasets) and stored in memory distributed over machines. To accelerate its various computation intensive operations, such as reduction and sort, we focus on GPU devices. We modified Spark framework to invoke CUDA kernels when computation intensive operations are called. RDDs are transformed into array structures and transferred to GPU devices when necessary. Although we need to cache RDDs in GPU device memory as much as possible in order to hide the data transfer overhead, the number of local GPU devices mounted in a host machine is limited. In this paper, we propose to use remote GPU devices which are connected to a host machine via a PCI-Express over 10Gbps Ethernet technology. To mitigate the data transfer overhead for remote GPU devices, we propose three RDD caching policies for local and remote GPU devices. We implemented various reduction programs (e.g., Sum, Max, LineCount) and transformation programs (e.g., SortByKey, PatternMatch, WordConversion) using local and remote GPU devices for Spark. Evaluation results show that Spark with GPU outperforms the original software by up to 21.4x. We also evaluate the RDD caching policies for local and remote GPU devices and show that a caching policy that minimizes the data transfer amount for remote GPU devices achieves the best performance.

本文言語English
ホスト出版物のタイトルProceedings - 22nd IEEE International Conference on Parallel and Distributed Systems, ICPADS 2016
編集者Xiaofei Liao, Robert Lovas, Xipeng Shen, Ran Zheng
出版社IEEE Computer Society
ページ791-799
ページ数9
ISBN(電子版)9781509044573
DOI
出版ステータスPublished - 2016 7月 2
イベント22nd IEEE International Conference on Parallel and Distributed Systems, ICPADS 2016 - Wuhan, Hubei, China
継続期間: 2016 12月 132016 12月 16

出版物シリーズ

名前Proceedings of the International Conference on Parallel and Distributed Systems - ICPADS
0
ISSN(印刷版)1521-9097

Other

Other22nd IEEE International Conference on Parallel and Distributed Systems, ICPADS 2016
国/地域China
CityWuhan, Hubei
Period16/12/1316/12/16

ASJC Scopus subject areas

  • ハードウェアとアーキテクチャ

フィンガープリント

「Accelerating spark RDD operations with local and remote GPU devices」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル