Designing a Hadoop system based on computational resources and network delay for wide area networks

Tomohiro Matsuno, Bijoy Chand Chatterjee, Nattapong Kitsuwan, Eiji Oki, Malathi Veeraraghavan, Satoru Okamoto, Naoaki Yamanaka

Research output: Contribution to journalArticle

Abstract

This paper proposes a Hadoop system that considers both slave server’s processing capacity and network delay for wide area networks to reduce the job processing time. The task allocation scheme in the proposed Hadoop system divides each individual job into multiple tasks using suitable splitting ratios and then allocates the tasks to different slaves according to the computational capability of each server and the availability of network resources. We incorporate software-defined networking to the proposed Hadoop system to manage path computation elements and network resources. The performance of proposed Hadoop system is experimentally evaluated with fourteen machines located in the different parts of the globe using a scale-out approach. A scale-out experiment using the proposed and conventional Hadoop systems is conducted by executing both single job and multiple jobs. The practical testbed and simulation results indicate that the proposed Hadoop system is effective compared to the conventional Hadoop system in terms of processing time.

Original languageEnglish
Pages (from-to)1-13
Number of pages13
JournalTelecommunication Systems
DOIs
Publication statusAccepted/In press - 2018 Apr 26

Keywords

  • Hadoop
  • Heterogeneous clusters
  • Implementation
  • Jobtracker

ASJC Scopus subject areas

  • Electrical and Electronic Engineering

Fingerprint Dive into the research topics of 'Designing a Hadoop system based on computational resources and network delay for wide area networks'. Together they form a unique fingerprint.

  • Cite this