Applying Pwrake Workflow System and Gfarm File System to Telescope Data Processing

Masahiro Tanaka, Osamu Tatebe, Hideyuki Kawashima

研究成果: Conference contribution

抜粋

In this paper, we describe a use case applying a scientific workflow system and a distributed file system to improve the performance of telescope data processing. The application is pipeline processing of data generated by Hyper Suprime-Cam (HSC) which is a focal plane camera mounted on the Subaru telescope. In this paper, we focus on the scalability of parallel I/O and core utilization. The IBM Spectrum Scale (GPFS) used for actual operation has a limit on scalability due to the configuration using storage servers. Therefore, we introduce the Gfarm file system which uses the storage of the worker node for parallel I/O performance. To improve core utilization, we introduce the Pwrake workflow system instead of the parallel processing framework developed for the HSC pipeline. Descriptions of task dependencies are necessary to further improve core utilization by overlapping different types of tasks. We discuss the usefulness of the workflow description language with the function of scripting language for defining complex task dependency. In the experiment, the performance of the pipeline is evaluated using a quarter of the observation data per night (input files: 80 GB, output files: 1.2 TB). Measurements on strong scaling from 48 to 576 cores show that the processing with Gfarm file system is more scalable than that with GPFS. Measurement using 576 cores shows that our method improves the processing speed of the pipeline by 2.2 times compared with the method used in actual operation.

元の言語English
ホスト出版物のタイトルProceedings - 2018 IEEE International Conference on Cluster Computing, CLUSTER 2018
出版者Institute of Electrical and Electronics Engineers Inc.
ページ124-133
ページ数10
ISBN(電子版)9781538683194
DOI
出版物ステータスPublished - 2018 10 29
イベント2018 IEEE International Conference on Cluster Computing, CLUSTER 2018 - Belfast, United Kingdom
継続期間: 2018 9 102018 9 13

出版物シリーズ

名前Proceedings - IEEE International Conference on Cluster Computing, ICCC
2018-September
ISSN(印刷物)1552-5244

Other

Other2018 IEEE International Conference on Cluster Computing, CLUSTER 2018
United Kingdom
Belfast
期間18/9/1018/9/13

ASJC Scopus subject areas

  • Software
  • Hardware and Architecture
  • Signal Processing

フィンガープリント Applying Pwrake Workflow System and Gfarm File System to Telescope Data Processing' の研究トピックを掘り下げます。これらはともに一意のフィンガープリントを構成します。

  • これを引用

    Tanaka, M., Tatebe, O., & Kawashima, H. (2018). Applying Pwrake Workflow System and Gfarm File System to Telescope Data Processing. : Proceedings - 2018 IEEE International Conference on Cluster Computing, CLUSTER 2018 (pp. 124-133). [8514866] (Proceedings - IEEE International Conference on Cluster Computing, ICCC; 巻数 2018-September). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/CLUSTER.2018.00024