Deployment strategies for crowdsourcing text creation

Ria Mae Borromeo, Thomas Laurent, Motomichi Toyama, Maha Alsayasneh, Sihem Amer-Yahia, Vincent Leroy

Research output: Contribution to journalArticle

3 Citations (Scopus)

Abstract

Automatically generating text of high quality in tasks such as translation, summarization, and narrative writing is difficult as these tasks require creativity, which only humans currently exhibit. However, crowdsourcing such tasks is still a challenge as they are tedious for humans and can require expert knowledge. We thus explore deployment strategies for crowdsourcing text creation tasks to improve the effectiveness of the crowdsourcing process. We consider effectiveness through the quality of the output text, the cost of deploying the task, and the latency in obtaining the output. We formalize a deployment strategy in crowdsourcing along three dimensions: work structure, workforce organization, and work style. Work structure can either be simultaneous or sequential, workforce organization independent or collaborative, and work style either by humans only or by using a combination of machine and human intelligence. We implement these strategies for translation, summarization, and narrative writing tasks by designing a semi-automatic tool that uses the Amazon Mechanical Turk API and experiment with them in different input settings such as text length, number of sources, and topic popularity. We report our findings regarding the effectiveness of each strategy and provide recommendations to guide requesters in selecting the best strategy when deploying text creation tasks.

Original languageEnglish
Pages (from-to)103-110
Number of pages8
JournalInformation Systems
Volume71
DOIs
Publication statusPublished - 2017 Nov 1

Fingerprint

Application programming interfaces (API)
Costs
Experiments

Keywords

  • Crowdsourcing
  • Deployment strategies
  • Text creation

ASJC Scopus subject areas

  • Software
  • Information Systems
  • Hardware and Architecture

Cite this

Borromeo, R. M., Laurent, T., Toyama, M., Alsayasneh, M., Amer-Yahia, S., & Leroy, V. (2017). Deployment strategies for crowdsourcing text creation. Information Systems, 71, 103-110. https://doi.org/10.1016/j.is.2017.06.007

Deployment strategies for crowdsourcing text creation. / Borromeo, Ria Mae; Laurent, Thomas; Toyama, Motomichi; Alsayasneh, Maha; Amer-Yahia, Sihem; Leroy, Vincent.

In: Information Systems, Vol. 71, 01.11.2017, p. 103-110.

Research output: Contribution to journalArticle

Borromeo, RM, Laurent, T, Toyama, M, Alsayasneh, M, Amer-Yahia, S & Leroy, V 2017, 'Deployment strategies for crowdsourcing text creation', Information Systems, vol. 71, pp. 103-110. https://doi.org/10.1016/j.is.2017.06.007
Borromeo RM, Laurent T, Toyama M, Alsayasneh M, Amer-Yahia S, Leroy V. Deployment strategies for crowdsourcing text creation. Information Systems. 2017 Nov 1;71:103-110. https://doi.org/10.1016/j.is.2017.06.007
Borromeo, Ria Mae ; Laurent, Thomas ; Toyama, Motomichi ; Alsayasneh, Maha ; Amer-Yahia, Sihem ; Leroy, Vincent. / Deployment strategies for crowdsourcing text creation. In: Information Systems. 2017 ; Vol. 71. pp. 103-110.
@article{cfc5ffdb2d704b808ef5674a8e2b7789,
title = "Deployment strategies for crowdsourcing text creation",
abstract = "Automatically generating text of high quality in tasks such as translation, summarization, and narrative writing is difficult as these tasks require creativity, which only humans currently exhibit. However, crowdsourcing such tasks is still a challenge as they are tedious for humans and can require expert knowledge. We thus explore deployment strategies for crowdsourcing text creation tasks to improve the effectiveness of the crowdsourcing process. We consider effectiveness through the quality of the output text, the cost of deploying the task, and the latency in obtaining the output. We formalize a deployment strategy in crowdsourcing along three dimensions: work structure, workforce organization, and work style. Work structure can either be simultaneous or sequential, workforce organization independent or collaborative, and work style either by humans only or by using a combination of machine and human intelligence. We implement these strategies for translation, summarization, and narrative writing tasks by designing a semi-automatic tool that uses the Amazon Mechanical Turk API and experiment with them in different input settings such as text length, number of sources, and topic popularity. We report our findings regarding the effectiveness of each strategy and provide recommendations to guide requesters in selecting the best strategy when deploying text creation tasks.",
keywords = "Crowdsourcing, Deployment strategies, Text creation",
author = "Borromeo, {Ria Mae} and Thomas Laurent and Motomichi Toyama and Maha Alsayasneh and Sihem Amer-Yahia and Vincent Leroy",
year = "2017",
month = "11",
day = "1",
doi = "10.1016/j.is.2017.06.007",
language = "English",
volume = "71",
pages = "103--110",
journal = "Information Systems",
issn = "0306-4379",
publisher = "Elsevier Limited",

}

TY - JOUR

T1 - Deployment strategies for crowdsourcing text creation

AU - Borromeo, Ria Mae

AU - Laurent, Thomas

AU - Toyama, Motomichi

AU - Alsayasneh, Maha

AU - Amer-Yahia, Sihem

AU - Leroy, Vincent

PY - 2017/11/1

Y1 - 2017/11/1

N2 - Automatically generating text of high quality in tasks such as translation, summarization, and narrative writing is difficult as these tasks require creativity, which only humans currently exhibit. However, crowdsourcing such tasks is still a challenge as they are tedious for humans and can require expert knowledge. We thus explore deployment strategies for crowdsourcing text creation tasks to improve the effectiveness of the crowdsourcing process. We consider effectiveness through the quality of the output text, the cost of deploying the task, and the latency in obtaining the output. We formalize a deployment strategy in crowdsourcing along three dimensions: work structure, workforce organization, and work style. Work structure can either be simultaneous or sequential, workforce organization independent or collaborative, and work style either by humans only or by using a combination of machine and human intelligence. We implement these strategies for translation, summarization, and narrative writing tasks by designing a semi-automatic tool that uses the Amazon Mechanical Turk API and experiment with them in different input settings such as text length, number of sources, and topic popularity. We report our findings regarding the effectiveness of each strategy and provide recommendations to guide requesters in selecting the best strategy when deploying text creation tasks.

AB - Automatically generating text of high quality in tasks such as translation, summarization, and narrative writing is difficult as these tasks require creativity, which only humans currently exhibit. However, crowdsourcing such tasks is still a challenge as they are tedious for humans and can require expert knowledge. We thus explore deployment strategies for crowdsourcing text creation tasks to improve the effectiveness of the crowdsourcing process. We consider effectiveness through the quality of the output text, the cost of deploying the task, and the latency in obtaining the output. We formalize a deployment strategy in crowdsourcing along three dimensions: work structure, workforce organization, and work style. Work structure can either be simultaneous or sequential, workforce organization independent or collaborative, and work style either by humans only or by using a combination of machine and human intelligence. We implement these strategies for translation, summarization, and narrative writing tasks by designing a semi-automatic tool that uses the Amazon Mechanical Turk API and experiment with them in different input settings such as text length, number of sources, and topic popularity. We report our findings regarding the effectiveness of each strategy and provide recommendations to guide requesters in selecting the best strategy when deploying text creation tasks.

KW - Crowdsourcing

KW - Deployment strategies

KW - Text creation

UR - http://www.scopus.com/inward/record.url?scp=85026832860&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85026832860&partnerID=8YFLogxK

U2 - 10.1016/j.is.2017.06.007

DO - 10.1016/j.is.2017.06.007

M3 - Article

AN - SCOPUS:85026832860

VL - 71

SP - 103

EP - 110

JO - Information Systems

JF - Information Systems

SN - 0306-4379

ER -