Horizontal division of deep learning applications with all-to-all communication on a multi-FPGA system

Yugo Yamauchi, Akram Ben Ahmed, Kazuei Hironaka, Kensuke Iizuka, Hideharu Amano

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Citations (Scopus)

Abstract

Although convolutional neural networks (CNNs) have plenty of parallelism, traditional layer-by-layer task division designs for multi-FPGA systems have the following problems: (1) The computational load of each layer is different from each other, so the execution time is dominated with the heaviest one. (2) Each FPGA must be designed independently, it means that we must design, generate and manage various configuration files. To address this problem, we propose a horizontal division method that enables us to use of a single design for each FPGA. All layers are divided horizontal direction of the target CNN, and a set of layers is implemented on an FPGA. It reduces the time of design as well as management costs for the execution. Also, since the weight data can be separated, the usage of local memory can be reduced. The apparent disadvantage of this method is that it requires all-to-all data communication between FPGA boards, and so it is not suitable to traditional multi-FPGA systems with a simple linear network. Here, we tried to apply the method to FiC (Flow-in-Cloud) which has a powerful network to enable efficient broadcasting. A simple CNN LeNet and a matrix multiplication for more practical fully connected layer is implemented on the FiC prototype. As a result of the evaluation, LeNet using 8 FP-GAs achieved 7.5 times faster than that with a single FPGA, and achieved 12.6 times faster than the optimized software of a high-end CPU.

Original languageEnglish
Title of host publicationProceedings - 2020 8th International Symposium on Computing and Networking Workshops, CANDARW 2020
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages277-281
Number of pages5
ISBN (Electronic)9781728199191
DOIs
Publication statusPublished - 2020 Nov
Event8th International Symposium on Computing and Networking Workshops, CANDARW 2020 - Virtual, Naha, Japan
Duration: 2020 Nov 242020 Nov 27

Publication series

NameProceedings - 2020 8th International Symposium on Computing and Networking Workshops, CANDARW 2020

Conference

Conference8th International Symposium on Computing and Networking Workshops, CANDARW 2020
Country/TerritoryJapan
CityVirtual, Naha
Period20/11/2420/11/27

Keywords

  • CNN
  • FPGA
  • multiFPGA

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Computer Science Applications
  • Hardware and Architecture
  • Computational Mathematics
  • Control and Optimization

Fingerprint

Dive into the research topics of 'Horizontal division of deep learning applications with all-to-all communication on a multi-FPGA system'. Together they form a unique fingerprint.

Cite this