Task allocation scheme based on computational and network resources for heterogeneous Hadoop clusters

Tomohiro Matsuno, Bijoy Chand Chatterjee, Eiji Oki, Malathi Veeraraghavan, Satoru Okamoto, Naoaki Yamanaka

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

This paper aims to design a Hadoop system and evaluates the performance of a task allocation scheme. The task allocation scheme splits each job into tasks using an appropriate splitting ratio, and assigns tasks to slave servers based on server processing performance and network resource availability. We experimentally evaluate the performance of the scale out of the task allocation scheme with five machines. We focus on the configuration of jobtracker and tasktracker in Hadoop. In cases with heterogeneous Hadoop clusters, we distribute task blocks to high-capability slaves with proportionally larger-sized tasks than to low-capability slaves. We create an environment in which high-capability slaves perform more work than low-capability slaves. The experimental testbed results indicate that the task allocation scheme is effective.

Original languageEnglish
Title of host publication2016 IEEE 17th International Conference on High Performance Switching and Routing, HPSR 2016
PublisherIEEE Computer Society
Pages200-205
Number of pages6
Volume2016-July
ISBN (Electronic)9781479989508
DOIs
Publication statusPublished - 2016 Jul 28
Event17th IEEE International Conference on High Performance Switching and Routing, HPSR 2016 - Yokohama, Japan
Duration: 2016 Jun 142016 Jun 17

Other

Other17th IEEE International Conference on High Performance Switching and Routing, HPSR 2016
CountryJapan
CityYokohama
Period16/6/1416/6/17

Keywords

  • Hadoop
  • heterogeneous clusters
  • implementation
  • jobtracker

ASJC Scopus subject areas

  • Hardware and Architecture
  • Electrical and Electronic Engineering

Fingerprint Dive into the research topics of 'Task allocation scheme based on computational and network resources for heterogeneous Hadoop clusters'. Together they form a unique fingerprint.

  • Cite this

    Matsuno, T., Chatterjee, B. C., Oki, E., Veeraraghavan, M., Okamoto, S., & Yamanaka, N. (2016). Task allocation scheme based on computational and network resources for heterogeneous Hadoop clusters. In 2016 IEEE 17th International Conference on High Performance Switching and Routing, HPSR 2016 (Vol. 2016-July, pp. 200-205). [7525667] IEEE Computer Society. https://doi.org/10.1109/HPSR.2016.7525667