Accelerating Deep Learning using Multiple GPUs and FPGA-Based 10GbE Switch

Tomoya Itsubo, Michihiro Koibuchi, Hideharu Amano, Hiroki Matsutani

研究成果: Conference contribution

4 被引用数 (Scopus)

抄録

A back-propagation algorithm following a gradient descent approach is used for training deep neural networks. Since it iteratively performs a large number of matrix operations to compute the gradients, GPUs (Graphics Processing Units) are efficient especially for the training phase. Thus, a cluster of computers each of which equips multiple GPUs can significantly accelerate the training phase. Although the gradient computation is still a major bottleneck of the training, gradient aggregation and parameter optimization impose both communication and computation overheads, which should also be reduced for further shortening the training time. To address this issue, in this paper, multiple GPUs are interconnected with a PCI Express (PCIe) over 10 Gbit Ethernet (10GbE) technology. Since these remote GPUs are interconnected via network switches, gradient aggregation and optimizers (e.g., SGD, Adagrad, Adam, and SMORMS3) are offloaded to an FPGA-based network switch between a host machine and remote GPUs; thus, the gradient aggregation and optimization are completed in the network. Evaluation results using four remote GPUs connected via the FPGA-based 10GbE switch that implements the four optimizers demonstrate that these optimization algorithms are accelerated by up to 3. 0x and 1. 25x compared to CPU and GPU implementations, respectively. Also, the gradient aggregation throughput by the FPGA-based switch achieves 98.3% of the 10GbE line rate.

本文言語English
ホスト出版物のタイトルProceedings - 2020 28th Euromicro International Conference on Parallel, Distributed and Network-Based Processing, PDP 2020
出版社Institute of Electrical and Electronics Engineers Inc.
ページ102-109
ページ数8
ISBN(電子版)9781728165820
DOI
出版ステータスPublished - 2020 3月
イベント28th Euromicro International Conference on Parallel, Distributed and Network-Based Processing, PDP 2020 - Vasteras, Sweden
継続期間: 2020 3月 112020 3月 13

出版物シリーズ

名前Proceedings - 2020 28th Euromicro International Conference on Parallel, Distributed and Network-Based Processing, PDP 2020

Conference

Conference28th Euromicro International Conference on Parallel, Distributed and Network-Based Processing, PDP 2020
国/地域Sweden
CityVasteras
Period20/3/1120/3/13

ASJC Scopus subject areas

  • 人工知能
  • コンピュータ ネットワークおよび通信
  • 情報システムおよび情報管理
  • 計算数学
  • 制御と最適化
  • 健康情報学

フィンガープリント

「Accelerating Deep Learning using Multiple GPUs and FPGA-Based 10GbE Switch」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル