Content espresso: A distributed large file sharing system for digital content productions

Research output: Contribution to journalArticle

Abstract

With rapid growth of producing high-resolution digital contents such as Full HD, 4K, and 8K movies, the demand for low cost and high throughput sharing of content files is increasing at digital content productions. In order to meet this demand, we have proposed DRIP (Distributed chunks Retrieval and Integration Procedure), a storage and retrieval mechanism for large file sharing using forward error correction (FEC) and global dispersed storage. DRIP was confirmed that it contributes to low cost and high throughput sharing. This paper describes the design and implementation of Content Espresso, a distributed large file sharing system for digital content productions using DRIP, and presents performance evaluations. We set up experimental environment using 79 physical machines including 72 inexpensive storage servers, and evaluate file metadata access performance, file storage/retrieval performance, FEC block size, and system availability by emulating global environments. The results confirm that Content Espresso has capability to deal with 15,000 requests per second, achieves 1 Gbps for file storage, and achieves more than 3 Gbps for file retrieval. File storage and retrieval performance are not significantly affected by the network conditions. Thus, we conclude that Content Espresso is capable of a global scale file sharing system for digital content productions.

Original languageEnglish
Pages (from-to)2100-2117
Number of pages18
JournalIEICE Transactions on Information and Systems
VolumeE100D
Issue number9
DOIs
Publication statusPublished - 2017 Sep 1

Fingerprint

Forward error correction
Throughput
Metadata
Costs
Servers
Availability

Keywords

  • Distributed storage system
  • FEC
  • File sharing
  • UDP data transmission

ASJC Scopus subject areas

  • Software
  • Hardware and Architecture
  • Computer Vision and Pattern Recognition
  • Artificial Intelligence
  • Electrical and Electronic Engineering

Cite this

Content espresso : A distributed large file sharing system for digital content productions. / Ando, Daisuke; Teraoka, Fumio; Kaneko, Kunitake.

In: IEICE Transactions on Information and Systems, Vol. E100D, No. 9, 01.09.2017, p. 2100-2117.

Research output: Contribution to journalArticle

@article{944f8d18d0a3402eba7443dac291ccc5,
title = "Content espresso: A distributed large file sharing system for digital content productions",
abstract = "With rapid growth of producing high-resolution digital contents such as Full HD, 4K, and 8K movies, the demand for low cost and high throughput sharing of content files is increasing at digital content productions. In order to meet this demand, we have proposed DRIP (Distributed chunks Retrieval and Integration Procedure), a storage and retrieval mechanism for large file sharing using forward error correction (FEC) and global dispersed storage. DRIP was confirmed that it contributes to low cost and high throughput sharing. This paper describes the design and implementation of Content Espresso, a distributed large file sharing system for digital content productions using DRIP, and presents performance evaluations. We set up experimental environment using 79 physical machines including 72 inexpensive storage servers, and evaluate file metadata access performance, file storage/retrieval performance, FEC block size, and system availability by emulating global environments. The results confirm that Content Espresso has capability to deal with 15,000 requests per second, achieves 1 Gbps for file storage, and achieves more than 3 Gbps for file retrieval. File storage and retrieval performance are not significantly affected by the network conditions. Thus, we conclude that Content Espresso is capable of a global scale file sharing system for digital content productions.",
keywords = "Distributed storage system, FEC, File sharing, UDP data transmission",
author = "Daisuke Ando and Fumio Teraoka and Kunitake Kaneko",
year = "2017",
month = "9",
day = "1",
doi = "10.1587/transinf.2017EDP7048",
language = "English",
volume = "E100D",
pages = "2100--2117",
journal = "IEICE Transactions on Information and Systems",
issn = "0916-8532",
publisher = "Maruzen Co., Ltd/Maruzen Kabushikikaisha",
number = "9",

}

TY - JOUR

T1 - Content espresso

T2 - A distributed large file sharing system for digital content productions

AU - Ando, Daisuke

AU - Teraoka, Fumio

AU - Kaneko, Kunitake

PY - 2017/9/1

Y1 - 2017/9/1

N2 - With rapid growth of producing high-resolution digital contents such as Full HD, 4K, and 8K movies, the demand for low cost and high throughput sharing of content files is increasing at digital content productions. In order to meet this demand, we have proposed DRIP (Distributed chunks Retrieval and Integration Procedure), a storage and retrieval mechanism for large file sharing using forward error correction (FEC) and global dispersed storage. DRIP was confirmed that it contributes to low cost and high throughput sharing. This paper describes the design and implementation of Content Espresso, a distributed large file sharing system for digital content productions using DRIP, and presents performance evaluations. We set up experimental environment using 79 physical machines including 72 inexpensive storage servers, and evaluate file metadata access performance, file storage/retrieval performance, FEC block size, and system availability by emulating global environments. The results confirm that Content Espresso has capability to deal with 15,000 requests per second, achieves 1 Gbps for file storage, and achieves more than 3 Gbps for file retrieval. File storage and retrieval performance are not significantly affected by the network conditions. Thus, we conclude that Content Espresso is capable of a global scale file sharing system for digital content productions.

AB - With rapid growth of producing high-resolution digital contents such as Full HD, 4K, and 8K movies, the demand for low cost and high throughput sharing of content files is increasing at digital content productions. In order to meet this demand, we have proposed DRIP (Distributed chunks Retrieval and Integration Procedure), a storage and retrieval mechanism for large file sharing using forward error correction (FEC) and global dispersed storage. DRIP was confirmed that it contributes to low cost and high throughput sharing. This paper describes the design and implementation of Content Espresso, a distributed large file sharing system for digital content productions using DRIP, and presents performance evaluations. We set up experimental environment using 79 physical machines including 72 inexpensive storage servers, and evaluate file metadata access performance, file storage/retrieval performance, FEC block size, and system availability by emulating global environments. The results confirm that Content Espresso has capability to deal with 15,000 requests per second, achieves 1 Gbps for file storage, and achieves more than 3 Gbps for file retrieval. File storage and retrieval performance are not significantly affected by the network conditions. Thus, we conclude that Content Espresso is capable of a global scale file sharing system for digital content productions.

KW - Distributed storage system

KW - FEC

KW - File sharing

KW - UDP data transmission

UR - http://www.scopus.com/inward/record.url?scp=85029444973&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85029444973&partnerID=8YFLogxK

U2 - 10.1587/transinf.2017EDP7048

DO - 10.1587/transinf.2017EDP7048

M3 - Article

AN - SCOPUS:85029444973

VL - E100D

SP - 2100

EP - 2117

JO - IEICE Transactions on Information and Systems

JF - IEICE Transactions on Information and Systems

SN - 0916-8532

IS - 9

ER -