Designing a Hadoop system based on computational resources and network delay for wide area networks

Tomohiro Matsuno, Bijoy Chand Chatterjee, Nattapong Kitsuwan, Eiji Oki, Malathi Veeraraghavan, Satoru Okamoto, Naoaki Yamanaka

Research output: Contribution to journalArticle

Abstract

This paper proposes a Hadoop system that considers both slave server’s processing capacity and network delay for wide area networks to reduce the job processing time. The task allocation scheme in the proposed Hadoop system divides each individual job into multiple tasks using suitable splitting ratios and then allocates the tasks to different slaves according to the computational capability of each server and the availability of network resources. We incorporate software-defined networking to the proposed Hadoop system to manage path computation elements and network resources. The performance of proposed Hadoop system is experimentally evaluated with fourteen machines located in the different parts of the globe using a scale-out approach. A scale-out experiment using the proposed and conventional Hadoop systems is conducted by executing both single job and multiple jobs. The practical testbed and simulation results indicate that the proposed Hadoop system is effective compared to the conventional Hadoop system in terms of processing time.

Original languageEnglish
Pages (from-to)1-13
Number of pages13
JournalTelecommunication Systems
DOIs
Publication statusAccepted/In press - 2018 Apr 26

Fingerprint

Wide area networks
Servers
Processing
Testbeds
Computer systems
Availability
Experiments

Keywords

  • Hadoop
  • Heterogeneous clusters
  • Implementation
  • Jobtracker

ASJC Scopus subject areas

  • Electrical and Electronic Engineering

Cite this

Designing a Hadoop system based on computational resources and network delay for wide area networks. / Matsuno, Tomohiro; Chatterjee, Bijoy Chand; Kitsuwan, Nattapong; Oki, Eiji; Veeraraghavan, Malathi; Okamoto, Satoru; Yamanaka, Naoaki.

In: Telecommunication Systems, 26.04.2018, p. 1-13.

Research output: Contribution to journalArticle

Matsuno, Tomohiro ; Chatterjee, Bijoy Chand ; Kitsuwan, Nattapong ; Oki, Eiji ; Veeraraghavan, Malathi ; Okamoto, Satoru ; Yamanaka, Naoaki. / Designing a Hadoop system based on computational resources and network delay for wide area networks. In: Telecommunication Systems. 2018 ; pp. 1-13.
@article{b81e2815c44d4d2e994b10f9d4729bff,
title = "Designing a Hadoop system based on computational resources and network delay for wide area networks",
abstract = "This paper proposes a Hadoop system that considers both slave server’s processing capacity and network delay for wide area networks to reduce the job processing time. The task allocation scheme in the proposed Hadoop system divides each individual job into multiple tasks using suitable splitting ratios and then allocates the tasks to different slaves according to the computational capability of each server and the availability of network resources. We incorporate software-defined networking to the proposed Hadoop system to manage path computation elements and network resources. The performance of proposed Hadoop system is experimentally evaluated with fourteen machines located in the different parts of the globe using a scale-out approach. A scale-out experiment using the proposed and conventional Hadoop systems is conducted by executing both single job and multiple jobs. The practical testbed and simulation results indicate that the proposed Hadoop system is effective compared to the conventional Hadoop system in terms of processing time.",
keywords = "Hadoop, Heterogeneous clusters, Implementation, Jobtracker",
author = "Tomohiro Matsuno and Chatterjee, {Bijoy Chand} and Nattapong Kitsuwan and Eiji Oki and Malathi Veeraraghavan and Satoru Okamoto and Naoaki Yamanaka",
year = "2018",
month = "4",
day = "26",
doi = "10.1007/s11235-018-0464-y",
language = "English",
pages = "1--13",
journal = "Telecommunication Systems",
issn = "1018-4864",
publisher = "Springer Netherlands",

}

TY - JOUR

T1 - Designing a Hadoop system based on computational resources and network delay for wide area networks

AU - Matsuno, Tomohiro

AU - Chatterjee, Bijoy Chand

AU - Kitsuwan, Nattapong

AU - Oki, Eiji

AU - Veeraraghavan, Malathi

AU - Okamoto, Satoru

AU - Yamanaka, Naoaki

PY - 2018/4/26

Y1 - 2018/4/26

N2 - This paper proposes a Hadoop system that considers both slave server’s processing capacity and network delay for wide area networks to reduce the job processing time. The task allocation scheme in the proposed Hadoop system divides each individual job into multiple tasks using suitable splitting ratios and then allocates the tasks to different slaves according to the computational capability of each server and the availability of network resources. We incorporate software-defined networking to the proposed Hadoop system to manage path computation elements and network resources. The performance of proposed Hadoop system is experimentally evaluated with fourteen machines located in the different parts of the globe using a scale-out approach. A scale-out experiment using the proposed and conventional Hadoop systems is conducted by executing both single job and multiple jobs. The practical testbed and simulation results indicate that the proposed Hadoop system is effective compared to the conventional Hadoop system in terms of processing time.

AB - This paper proposes a Hadoop system that considers both slave server’s processing capacity and network delay for wide area networks to reduce the job processing time. The task allocation scheme in the proposed Hadoop system divides each individual job into multiple tasks using suitable splitting ratios and then allocates the tasks to different slaves according to the computational capability of each server and the availability of network resources. We incorporate software-defined networking to the proposed Hadoop system to manage path computation elements and network resources. The performance of proposed Hadoop system is experimentally evaluated with fourteen machines located in the different parts of the globe using a scale-out approach. A scale-out experiment using the proposed and conventional Hadoop systems is conducted by executing both single job and multiple jobs. The practical testbed and simulation results indicate that the proposed Hadoop system is effective compared to the conventional Hadoop system in terms of processing time.

KW - Hadoop

KW - Heterogeneous clusters

KW - Implementation

KW - Jobtracker

UR - http://www.scopus.com/inward/record.url?scp=85045917088&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85045917088&partnerID=8YFLogxK

U2 - 10.1007/s11235-018-0464-y

DO - 10.1007/s11235-018-0464-y

M3 - Article

AN - SCOPUS:85045917088

SP - 1

EP - 13

JO - Telecommunication Systems

JF - Telecommunication Systems

SN - 1018-4864

ER -