Hadoop triggered opt/electrical data-center orchestration architecture for reducing power consumption

Akira Yamashita, Wataru Muro, Masayuki Hirono, Takehiro Sato, Satoru Okamoto, Naoaki Yamanaka, Malathi Veeraraghavan

Research output: Chapter in Book/Report/Conference proceedingConference contribution

4 Citations (Scopus)

Abstract

In this paper, a data-center network (DCN) system that distinguishes Hadoop job types and allocates optical/electrical circuits to data flows depending on the types automatically is proposed. The proposed system calculates the predicted shuffle value (PSV) of the Hadoop job and determines which type of flow is allocated by comparing the PSV and the threshold value. The PSV can be calculated from a part of the input data based on the shuffle ratio, which is known to have a relationship to the heaviness of the shuffle phase of the Hadoop job. The threshold of PSV can be dynamically changed according to the network load condition to exploit the optical circuit efficiently. By orchestrating the DCN and the Hadoop system, the proposed system achieves the reduction of power consumption. In this study, the orchestration part of the proposed DCN system is implemented and the feasibility of switching data flows between optical and electrical circuits is verified.

Original languageEnglish
Title of host publicationICTON 2017 - 19th International Conference on Transparent Optical Networks
PublisherIEEE Computer Society
ISBN (Electronic)9781538608586
DOIs
Publication statusPublished - 2017 Sep 1
Event19th International Conference on Transparent Optical Networks, ICTON 2017 - Girona, Catalonia, Spain
Duration: 2017 Jul 22017 Jul 6

Publication series

NameInternational Conference on Transparent Optical Networks
ISSN (Electronic)2162-7339

Other

Other19th International Conference on Transparent Optical Networks, ICTON 2017
CountrySpain
CityGirona, Catalonia
Period17/7/217/7/6

Keywords

  • Hadoop
  • data-center network
  • optical switch
  • orchestration
  • shuffle-heavy

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Electrical and Electronic Engineering
  • Electronic, Optical and Magnetic Materials

Fingerprint Dive into the research topics of 'Hadoop triggered opt/electrical data-center orchestration architecture for reducing power consumption'. Together they form a unique fingerprint.

Cite this