Fast window aggregate on array database by recursive incremental computation

Li Jiang, Hideyuki Kawashima, Osamu Tatebe

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

An array database is effective for managing and analyzing multidimensional scientific big data, and the window aggregate is an important operator in array databases. This paper proposes a method that exploits the scheme of incremental computation to accelerate the execution of window aggregates considerably. Six types of aggregate are improved using different designs of buffer tools to eliminate redundant computation. Our proposed recursive incremental computation method completely eliminates all redundant computation and achieves an improvement factor of the total window size compared with the naive method. This proposed method is fully implemented in SciDB. It improved performance by a factor of 10 on an earth science benchmark and by a factor of 64 on synthetic workloads with a certain data setting when compared with SciDB's built-in window operator.

Original languageEnglish
Title of host publicationProceedings of the 2016 IEEE 12th International Conference on e-Science, e-Science 2016
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages101-110
Number of pages10
ISBN (Electronic)9781509042722
DOIs
Publication statusPublished - 2017 Mar 3
Externally publishedYes
Event12th IEEE International Conference on e-Science, e-Science 2016 - Baltimore, United States
Duration: 2016 Oct 232016 Oct 27

Other

Other12th IEEE International Conference on e-Science, e-Science 2016
CountryUnited States
CityBaltimore
Period16/10/2316/10/27

    Fingerprint

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Information Systems
  • Environmental Science (miscellaneous)
  • Medicine (miscellaneous)
  • Social Sciences (miscellaneous)
  • Agricultural and Biological Sciences (miscellaneous)
  • Computer Science Applications

Cite this

Jiang, L., Kawashima, H., & Tatebe, O. (2017). Fast window aggregate on array database by recursive incremental computation. In Proceedings of the 2016 IEEE 12th International Conference on e-Science, e-Science 2016 (pp. 101-110). [7870890] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/eScience.2016.7870890