“I can see that”: Developing shared rubric category interpretations through score negotiation

Jonathan Trace, Valerie Meier, Gerriet Janssen

Research output: Contribution to journalArticle

6 Citations (Scopus)

Abstract

Performance assessments using raters will always contain some subjectivity, and disagreement among raters necessitates reliable methods for resolving scores. Negotiation is one effective method to guide scoring decisions and reduce raters’ tendencies to be unexpectedly severe or lenient when scoring specific rubric categories or examinees. Beyond its utility for scoring, however, negotiation is also a resource for raters to co-construct interpretations about the language constructs being measured. This study uses quantitative and qualitative methods to trace how negotiation impacts raters’ scoring decisions and examine in detail how raters develop joint interpretations of rubric category criteria. Scores from the writing section of a high stakes English language placement exam (n = 60) were analyzed using ANOVA and many-faceted Rasch measurement to determine which categories were frequently assigned discrepant scores and to estimate rater severity. Discourse analysis of six audiotaped negotiation sessions was then used to examine how raters’ understanding of rubric criteria converged over time. Our results indicate that through negotiation, raters used shared terminology and justifications to clarify ambiguous constructs and work to establish shared values. The results suggest that score negotiation influences scoring inferences and also creates affordances for raters to ground those inferences in shared constructions of meaning.

Original languageEnglish
Pages (from-to)32-43
Number of pages12
JournalAssessing Writing
Volume30
DOIs
Publication statusPublished - 2016 Oct 1

Keywords

  • Academic writing
  • Negotiation
  • Performance assessment
  • Raters
  • Rubrics
  • Sociocultural learning

ASJC Scopus subject areas

  • Language and Linguistics
  • Education
  • Linguistics and Language

Fingerprint Dive into the research topics of '“I can see that”: Developing shared rubric category interpretations through score negotiation'. Together they form a unique fingerprint.

  • Cite this