Deployment strategies for crowdsourcing text creation

Ria Mae Borromeo, Thomas Laurent, Motomichi Toyama, Maha Alsayasneh, Sihem Amer-Yahia, Vincent Leroy

Research output: Contribution to journalArticle

6 Citations (Scopus)

Abstract

Automatically generating text of high quality in tasks such as translation, summarization, and narrative writing is difficult as these tasks require creativity, which only humans currently exhibit. However, crowdsourcing such tasks is still a challenge as they are tedious for humans and can require expert knowledge. We thus explore deployment strategies for crowdsourcing text creation tasks to improve the effectiveness of the crowdsourcing process. We consider effectiveness through the quality of the output text, the cost of deploying the task, and the latency in obtaining the output. We formalize a deployment strategy in crowdsourcing along three dimensions: work structure, workforce organization, and work style. Work structure can either be simultaneous or sequential, workforce organization independent or collaborative, and work style either by humans only or by using a combination of machine and human intelligence. We implement these strategies for translation, summarization, and narrative writing tasks by designing a semi-automatic tool that uses the Amazon Mechanical Turk API and experiment with them in different input settings such as text length, number of sources, and topic popularity. We report our findings regarding the effectiveness of each strategy and provide recommendations to guide requesters in selecting the best strategy when deploying text creation tasks.

Original languageEnglish
Pages (from-to)103-110
Number of pages8
JournalInformation Systems
Volume71
DOIs
Publication statusPublished - 2017 Nov 1

Keywords

  • Crowdsourcing
  • Deployment strategies
  • Text creation

ASJC Scopus subject areas

  • Software
  • Information Systems
  • Hardware and Architecture

Fingerprint Dive into the research topics of 'Deployment strategies for crowdsourcing text creation'. Together they form a unique fingerprint.

  • Cite this

    Borromeo, R. M., Laurent, T., Toyama, M., Alsayasneh, M., Amer-Yahia, S., & Leroy, V. (2017). Deployment strategies for crowdsourcing text creation. Information Systems, 71, 103-110. https://doi.org/10.1016/j.is.2017.06.007