Evaluation by native and non-native English teacher-raters of Japanese students’ summaries

Yuko Hijikata-Someya, Masumi Ono, Hiroyuki Yamanishi

研究成果: Article

2 引用 (Scopus)

抄録

Although the importance of summary writing is well documented in prior studies, few have investigated the evaluation of written summaries. Due to the complex nature of L2 summary writing, which requires one to read the original material and summarize its content in the L2, raters often emphasize different features when judging the quality of L2 summaries. Therefore, this study examines the ratings of English-language summaries written by Japanese university students in order to identify differences in EFL instructors’ evaluations. Fifty-one Japanese EFL university students read a passage and then wrote an English summary without receiving any instructions concerning summary composition. The raters included three native English speakers (NESs) and three non-native English speakers (NNESs), who individually evaluated each summary using the Educational Testing Service’s holistic rubric. Analysis of inter-rater reliability revealed a lower Cronbach’s alpha coefficient for NNES raters (α = .39) when compared to NES raters (α = .77). Comments were collected from raters regarding the difficulty of evaluating summaries, and the causes of such difficulties were examined. Comments from NNES raters more concerned vocabulary use and paraphrasing, whereas the NES raters concentrated on content and language. This study also explores ways to potentially improve the holistic rubric by examining feedback from raters regarding their rating experiences.

元の言語English
ページ(範囲)1-12
ページ数12
ジャーナルEnglish Language Teaching
8
発行部数7
DOI
出版物ステータスPublished - 2015

Fingerprint

rating
university
teacher
evaluation
English language
vocabulary
instructor
student
instruction
cause
language
experience
Japanese Students
Summary
Raters
Evaluation
English Teachers
English Speakers
Non-native English Speakers

ASJC Scopus subject areas

  • Education
  • Language and Linguistics
  • Linguistics and Language

これを引用

Evaluation by native and non-native English teacher-raters of Japanese students’ summaries. / Hijikata-Someya, Yuko; Ono, Masumi; Yamanishi, Hiroyuki.

:: English Language Teaching, 巻 8, 番号 7, 2015, p. 1-12.

研究成果: Article

Hijikata-Someya, Yuko ; Ono, Masumi ; Yamanishi, Hiroyuki. / Evaluation by native and non-native English teacher-raters of Japanese students’ summaries. :: English Language Teaching. 2015 ; 巻 8, 番号 7. pp. 1-12.
@article{2a75e1ca282642ddbafeee24f6ce6232,
title = "Evaluation by native and non-native English teacher-raters of Japanese students’ summaries",
abstract = "Although the importance of summary writing is well documented in prior studies, few have investigated the evaluation of written summaries. Due to the complex nature of L2 summary writing, which requires one to read the original material and summarize its content in the L2, raters often emphasize different features when judging the quality of L2 summaries. Therefore, this study examines the ratings of English-language summaries written by Japanese university students in order to identify differences in EFL instructors’ evaluations. Fifty-one Japanese EFL university students read a passage and then wrote an English summary without receiving any instructions concerning summary composition. The raters included three native English speakers (NESs) and three non-native English speakers (NNESs), who individually evaluated each summary using the Educational Testing Service’s holistic rubric. Analysis of inter-rater reliability revealed a lower Cronbach’s alpha coefficient for NNES raters (α = .39) when compared to NES raters (α = .77). Comments were collected from raters regarding the difficulty of evaluating summaries, and the causes of such difficulties were examined. Comments from NNES raters more concerned vocabulary use and paraphrasing, whereas the NES raters concentrated on content and language. This study also explores ways to potentially improve the holistic rubric by examining feedback from raters regarding their rating experiences.",
keywords = "Evaluation, Holistic rubric, Native English speaker, Non-native English speaker, Paraphrasing, Reliability, Summary writing",
author = "Yuko Hijikata-Someya and Masumi Ono and Hiroyuki Yamanishi",
year = "2015",
doi = "10.5539/elt.v8n7p1",
language = "English",
volume = "8",
pages = "1--12",
journal = "English Language Teaching",
issn = "1916-4742",
publisher = "Canadian Center of Science and Education",
number = "7",

}

TY - JOUR

T1 - Evaluation by native and non-native English teacher-raters of Japanese students’ summaries

AU - Hijikata-Someya, Yuko

AU - Ono, Masumi

AU - Yamanishi, Hiroyuki

PY - 2015

Y1 - 2015

N2 - Although the importance of summary writing is well documented in prior studies, few have investigated the evaluation of written summaries. Due to the complex nature of L2 summary writing, which requires one to read the original material and summarize its content in the L2, raters often emphasize different features when judging the quality of L2 summaries. Therefore, this study examines the ratings of English-language summaries written by Japanese university students in order to identify differences in EFL instructors’ evaluations. Fifty-one Japanese EFL university students read a passage and then wrote an English summary without receiving any instructions concerning summary composition. The raters included three native English speakers (NESs) and three non-native English speakers (NNESs), who individually evaluated each summary using the Educational Testing Service’s holistic rubric. Analysis of inter-rater reliability revealed a lower Cronbach’s alpha coefficient for NNES raters (α = .39) when compared to NES raters (α = .77). Comments were collected from raters regarding the difficulty of evaluating summaries, and the causes of such difficulties were examined. Comments from NNES raters more concerned vocabulary use and paraphrasing, whereas the NES raters concentrated on content and language. This study also explores ways to potentially improve the holistic rubric by examining feedback from raters regarding their rating experiences.

AB - Although the importance of summary writing is well documented in prior studies, few have investigated the evaluation of written summaries. Due to the complex nature of L2 summary writing, which requires one to read the original material and summarize its content in the L2, raters often emphasize different features when judging the quality of L2 summaries. Therefore, this study examines the ratings of English-language summaries written by Japanese university students in order to identify differences in EFL instructors’ evaluations. Fifty-one Japanese EFL university students read a passage and then wrote an English summary without receiving any instructions concerning summary composition. The raters included three native English speakers (NESs) and three non-native English speakers (NNESs), who individually evaluated each summary using the Educational Testing Service’s holistic rubric. Analysis of inter-rater reliability revealed a lower Cronbach’s alpha coefficient for NNES raters (α = .39) when compared to NES raters (α = .77). Comments were collected from raters regarding the difficulty of evaluating summaries, and the causes of such difficulties were examined. Comments from NNES raters more concerned vocabulary use and paraphrasing, whereas the NES raters concentrated on content and language. This study also explores ways to potentially improve the holistic rubric by examining feedback from raters regarding their rating experiences.

KW - Evaluation

KW - Holistic rubric

KW - Native English speaker

KW - Non-native English speaker

KW - Paraphrasing

KW - Reliability

KW - Summary writing

UR - http://www.scopus.com/inward/record.url?scp=84936887268&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84936887268&partnerID=8YFLogxK

U2 - 10.5539/elt.v8n7p1

DO - 10.5539/elt.v8n7p1

M3 - Article

AN - SCOPUS:84936887268

VL - 8

SP - 1

EP - 12

JO - English Language Teaching

JF - English Language Teaching

SN - 1916-4742

IS - 7

ER -