Fast window aggregate on array database by recursive incremental computation

Li Jiang, Hideyuki Kawashima, Osamu Tatebe

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

An array database is effective for managing and analyzing multidimensional scientific big data, and the window aggregate is an important operator in array databases. This paper proposes a method that exploits the scheme of incremental computation to accelerate the execution of window aggregates considerably. Six types of aggregate are improved using different designs of buffer tools to eliminate redundant computation. Our proposed recursive incremental computation method completely eliminates all redundant computation and achieves an improvement factor of the total window size compared with the naive method. This proposed method is fully implemented in SciDB. It improved performance by a factor of 10 on an earth science benchmark and by a factor of 64 on synthetic workloads with a certain data setting when compared with SciDB's built-in window operator.

Original languageEnglish
Title of host publicationProceedings of the 2016 IEEE 12th International Conference on e-Science, e-Science 2016
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages101-110
Number of pages10
ISBN (Electronic)9781509042722
DOIs
Publication statusPublished - 2017 Mar 3
Externally publishedYes
Event12th IEEE International Conference on e-Science, e-Science 2016 - Baltimore, United States
Duration: 2016 Oct 232016 Oct 27

Other

Other12th IEEE International Conference on e-Science, e-Science 2016
CountryUnited States
CityBaltimore
Period16/10/2316/10/27

Fingerprint

Databases
Earth Sciences
Benchmarking
Earth sciences
Earth science
Workload
methodology
workload
Buffers
buffers
method
science
performance
Big data

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Information Systems
  • Environmental Science (miscellaneous)
  • Medicine (miscellaneous)
  • Social Sciences (miscellaneous)
  • Agricultural and Biological Sciences (miscellaneous)
  • Computer Science Applications

Cite this

Jiang, L., Kawashima, H., & Tatebe, O. (2017). Fast window aggregate on array database by recursive incremental computation. In Proceedings of the 2016 IEEE 12th International Conference on e-Science, e-Science 2016 (pp. 101-110). [7870890] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/eScience.2016.7870890

Fast window aggregate on array database by recursive incremental computation. / Jiang, Li; Kawashima, Hideyuki; Tatebe, Osamu.

Proceedings of the 2016 IEEE 12th International Conference on e-Science, e-Science 2016. Institute of Electrical and Electronics Engineers Inc., 2017. p. 101-110 7870890.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Jiang, L, Kawashima, H & Tatebe, O 2017, Fast window aggregate on array database by recursive incremental computation. in Proceedings of the 2016 IEEE 12th International Conference on e-Science, e-Science 2016., 7870890, Institute of Electrical and Electronics Engineers Inc., pp. 101-110, 12th IEEE International Conference on e-Science, e-Science 2016, Baltimore, United States, 16/10/23. https://doi.org/10.1109/eScience.2016.7870890
Jiang L, Kawashima H, Tatebe O. Fast window aggregate on array database by recursive incremental computation. In Proceedings of the 2016 IEEE 12th International Conference on e-Science, e-Science 2016. Institute of Electrical and Electronics Engineers Inc. 2017. p. 101-110. 7870890 https://doi.org/10.1109/eScience.2016.7870890
Jiang, Li ; Kawashima, Hideyuki ; Tatebe, Osamu. / Fast window aggregate on array database by recursive incremental computation. Proceedings of the 2016 IEEE 12th International Conference on e-Science, e-Science 2016. Institute of Electrical and Electronics Engineers Inc., 2017. pp. 101-110
@inproceedings{71c8d633564e42d68afad9fcab8d7b83,
title = "Fast window aggregate on array database by recursive incremental computation",
abstract = "An array database is effective for managing and analyzing multidimensional scientific big data, and the window aggregate is an important operator in array databases. This paper proposes a method that exploits the scheme of incremental computation to accelerate the execution of window aggregates considerably. Six types of aggregate are improved using different designs of buffer tools to eliminate redundant computation. Our proposed recursive incremental computation method completely eliminates all redundant computation and achieves an improvement factor of the total window size compared with the naive method. This proposed method is fully implemented in SciDB. It improved performance by a factor of 10 on an earth science benchmark and by a factor of 64 on synthetic workloads with a certain data setting when compared with SciDB's built-in window operator.",
author = "Li Jiang and Hideyuki Kawashima and Osamu Tatebe",
year = "2017",
month = "3",
day = "3",
doi = "10.1109/eScience.2016.7870890",
language = "English",
pages = "101--110",
booktitle = "Proceedings of the 2016 IEEE 12th International Conference on e-Science, e-Science 2016",
publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

TY - GEN

T1 - Fast window aggregate on array database by recursive incremental computation

AU - Jiang, Li

AU - Kawashima, Hideyuki

AU - Tatebe, Osamu

PY - 2017/3/3

Y1 - 2017/3/3

N2 - An array database is effective for managing and analyzing multidimensional scientific big data, and the window aggregate is an important operator in array databases. This paper proposes a method that exploits the scheme of incremental computation to accelerate the execution of window aggregates considerably. Six types of aggregate are improved using different designs of buffer tools to eliminate redundant computation. Our proposed recursive incremental computation method completely eliminates all redundant computation and achieves an improvement factor of the total window size compared with the naive method. This proposed method is fully implemented in SciDB. It improved performance by a factor of 10 on an earth science benchmark and by a factor of 64 on synthetic workloads with a certain data setting when compared with SciDB's built-in window operator.

AB - An array database is effective for managing and analyzing multidimensional scientific big data, and the window aggregate is an important operator in array databases. This paper proposes a method that exploits the scheme of incremental computation to accelerate the execution of window aggregates considerably. Six types of aggregate are improved using different designs of buffer tools to eliminate redundant computation. Our proposed recursive incremental computation method completely eliminates all redundant computation and achieves an improvement factor of the total window size compared with the naive method. This proposed method is fully implemented in SciDB. It improved performance by a factor of 10 on an earth science benchmark and by a factor of 64 on synthetic workloads with a certain data setting when compared with SciDB's built-in window operator.

UR - http://www.scopus.com/inward/record.url?scp=85016825447&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85016825447&partnerID=8YFLogxK

U2 - 10.1109/eScience.2016.7870890

DO - 10.1109/eScience.2016.7870890

M3 - Conference contribution

AN - SCOPUS:85016825447

SP - 101

EP - 110

BT - Proceedings of the 2016 IEEE 12th International Conference on e-Science, e-Science 2016

PB - Institute of Electrical and Electronics Engineers Inc.

ER -