Developing a scoring rubric for L2 summary writing: a hybrid approach combining analytic and holistic assessment

Hiroyuki Yamanishi, Masumi Ono, Yuko Hijikata

Research output: Contribution to journalArticle

Abstract

Background: In our research project, we have developed a scoring rubric for a second language (L2) summary writing for English as a foreign language (EFL) students in Japanese universities. This study aimed to examine the applicability of our five-dimensional rubric, which features both analytic and holistic assessments, to classrooms in the EFL context. The examination especially focused on a newly added, optional overall quality dimension and two paraphrasing dimensions: paraphrase (quantity) and paraphrase (quality). Methods: Six teacher raters evaluated 16 summaries written by Japanese EFL university student writers using our new rubric. The scoring results were quantitatively compared with the scoring results of a commonly used rubric developed by the Educational Testing Service (ETS). For the qualitative examination, the teacher raters’ retrospective comments on our five-dimensional rubric were analyzed. Results: The quantitative examination demonstrated positive results as follows: (a) adding the optional overall quality dimension improved the reliability of our rubric, (b) the overall quality dimension worked well even if it was used alone, (c) our rubric and the ETS holistic rubric overlapped moderately as L2 summary writing assessments, and (d) the two paraphrasing dimensions covered similar but different aspects of paraphrasing. However, the quantitative analysis using the generalizability theory (G theory) simulated that the reliability (generalizability coefficients) of the rubric was not high when the number of raters decreased. The qualitative examination of the raters’ perceptions of our rubric generally confirmed the quantitative results. Conclusion: This study confirmed the applicability of our rubric to EFL classrooms. This new type of rating scale can be characterized as a “hybrid” approach that offers the user a choice of analytic and holistic measures depending on individual purposes, which can enhance teachers’ explicit instructions.

Original languageEnglish
Article number13
JournalLanguage Testing in Asia
Volume9
Issue number1
DOIs
Publication statusPublished - 2019 Dec 1

Fingerprint

examination
teacher
classroom
university
rating scale
foreign language
research project
student
writer
instruction
Summary
Scoring
language
Raters
Paraphrasing

Keywords

  • Analytic assessment
  • Generalizability theory
  • Holistic assessment
  • Japanese university students
  • L2 summary writing
  • Paraphrasing
  • Rater perception
  • Rubric

ASJC Scopus subject areas

  • Language and Linguistics
  • Linguistics and Language

Cite this

Developing a scoring rubric for L2 summary writing : a hybrid approach combining analytic and holistic assessment. / Yamanishi, Hiroyuki; Ono, Masumi; Hijikata, Yuko.

In: Language Testing in Asia, Vol. 9, No. 1, 13, 01.12.2019.

Research output: Contribution to journalArticle

@article{99fdd49e6eb74eeb99ed7001c6061d4c,
title = "Developing a scoring rubric for L2 summary writing: a hybrid approach combining analytic and holistic assessment",
abstract = "Background: In our research project, we have developed a scoring rubric for a second language (L2) summary writing for English as a foreign language (EFL) students in Japanese universities. This study aimed to examine the applicability of our five-dimensional rubric, which features both analytic and holistic assessments, to classrooms in the EFL context. The examination especially focused on a newly added, optional overall quality dimension and two paraphrasing dimensions: paraphrase (quantity) and paraphrase (quality). Methods: Six teacher raters evaluated 16 summaries written by Japanese EFL university student writers using our new rubric. The scoring results were quantitatively compared with the scoring results of a commonly used rubric developed by the Educational Testing Service (ETS). For the qualitative examination, the teacher raters’ retrospective comments on our five-dimensional rubric were analyzed. Results: The quantitative examination demonstrated positive results as follows: (a) adding the optional overall quality dimension improved the reliability of our rubric, (b) the overall quality dimension worked well even if it was used alone, (c) our rubric and the ETS holistic rubric overlapped moderately as L2 summary writing assessments, and (d) the two paraphrasing dimensions covered similar but different aspects of paraphrasing. However, the quantitative analysis using the generalizability theory (G theory) simulated that the reliability (generalizability coefficients) of the rubric was not high when the number of raters decreased. The qualitative examination of the raters’ perceptions of our rubric generally confirmed the quantitative results. Conclusion: This study confirmed the applicability of our rubric to EFL classrooms. This new type of rating scale can be characterized as a “hybrid” approach that offers the user a choice of analytic and holistic measures depending on individual purposes, which can enhance teachers’ explicit instructions.",
keywords = "Analytic assessment, Generalizability theory, Holistic assessment, Japanese university students, L2 summary writing, Paraphrasing, Rater perception, Rubric",
author = "Hiroyuki Yamanishi and Masumi Ono and Yuko Hijikata",
year = "2019",
month = "12",
day = "1",
doi = "10.1186/s40468-019-0087-6",
language = "English",
volume = "9",
journal = "Language Testing in Asia",
issn = "2229-0443",
publisher = "Springer Open",
number = "1",

}

TY - JOUR

T1 - Developing a scoring rubric for L2 summary writing

T2 - a hybrid approach combining analytic and holistic assessment

AU - Yamanishi, Hiroyuki

AU - Ono, Masumi

AU - Hijikata, Yuko

PY - 2019/12/1

Y1 - 2019/12/1

N2 - Background: In our research project, we have developed a scoring rubric for a second language (L2) summary writing for English as a foreign language (EFL) students in Japanese universities. This study aimed to examine the applicability of our five-dimensional rubric, which features both analytic and holistic assessments, to classrooms in the EFL context. The examination especially focused on a newly added, optional overall quality dimension and two paraphrasing dimensions: paraphrase (quantity) and paraphrase (quality). Methods: Six teacher raters evaluated 16 summaries written by Japanese EFL university student writers using our new rubric. The scoring results were quantitatively compared with the scoring results of a commonly used rubric developed by the Educational Testing Service (ETS). For the qualitative examination, the teacher raters’ retrospective comments on our five-dimensional rubric were analyzed. Results: The quantitative examination demonstrated positive results as follows: (a) adding the optional overall quality dimension improved the reliability of our rubric, (b) the overall quality dimension worked well even if it was used alone, (c) our rubric and the ETS holistic rubric overlapped moderately as L2 summary writing assessments, and (d) the two paraphrasing dimensions covered similar but different aspects of paraphrasing. However, the quantitative analysis using the generalizability theory (G theory) simulated that the reliability (generalizability coefficients) of the rubric was not high when the number of raters decreased. The qualitative examination of the raters’ perceptions of our rubric generally confirmed the quantitative results. Conclusion: This study confirmed the applicability of our rubric to EFL classrooms. This new type of rating scale can be characterized as a “hybrid” approach that offers the user a choice of analytic and holistic measures depending on individual purposes, which can enhance teachers’ explicit instructions.

AB - Background: In our research project, we have developed a scoring rubric for a second language (L2) summary writing for English as a foreign language (EFL) students in Japanese universities. This study aimed to examine the applicability of our five-dimensional rubric, which features both analytic and holistic assessments, to classrooms in the EFL context. The examination especially focused on a newly added, optional overall quality dimension and two paraphrasing dimensions: paraphrase (quantity) and paraphrase (quality). Methods: Six teacher raters evaluated 16 summaries written by Japanese EFL university student writers using our new rubric. The scoring results were quantitatively compared with the scoring results of a commonly used rubric developed by the Educational Testing Service (ETS). For the qualitative examination, the teacher raters’ retrospective comments on our five-dimensional rubric were analyzed. Results: The quantitative examination demonstrated positive results as follows: (a) adding the optional overall quality dimension improved the reliability of our rubric, (b) the overall quality dimension worked well even if it was used alone, (c) our rubric and the ETS holistic rubric overlapped moderately as L2 summary writing assessments, and (d) the two paraphrasing dimensions covered similar but different aspects of paraphrasing. However, the quantitative analysis using the generalizability theory (G theory) simulated that the reliability (generalizability coefficients) of the rubric was not high when the number of raters decreased. The qualitative examination of the raters’ perceptions of our rubric generally confirmed the quantitative results. Conclusion: This study confirmed the applicability of our rubric to EFL classrooms. This new type of rating scale can be characterized as a “hybrid” approach that offers the user a choice of analytic and holistic measures depending on individual purposes, which can enhance teachers’ explicit instructions.

KW - Analytic assessment

KW - Generalizability theory

KW - Holistic assessment

KW - Japanese university students

KW - L2 summary writing

KW - Paraphrasing

KW - Rater perception

KW - Rubric

UR - http://www.scopus.com/inward/record.url?scp=85069530589&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85069530589&partnerID=8YFLogxK

U2 - 10.1186/s40468-019-0087-6

DO - 10.1186/s40468-019-0087-6

M3 - Article

AN - SCOPUS:85069530589

VL - 9

JO - Language Testing in Asia

JF - Language Testing in Asia

SN - 2229-0443

IS - 1

M1 - 13

ER -