“I can see that”: Developing shared rubric category interpretations through score negotiation

Jonathan Trace, Valerie Meier, Gerriet Janssen

研究成果: Article

5 引用 (Scopus)

抄録

Performance assessments using raters will always contain some subjectivity, and disagreement among raters necessitates reliable methods for resolving scores. Negotiation is one effective method to guide scoring decisions and reduce raters’ tendencies to be unexpectedly severe or lenient when scoring specific rubric categories or examinees. Beyond its utility for scoring, however, negotiation is also a resource for raters to co-construct interpretations about the language constructs being measured. This study uses quantitative and qualitative methods to trace how negotiation impacts raters’ scoring decisions and examine in detail how raters develop joint interpretations of rubric category criteria. Scores from the writing section of a high stakes English language placement exam (n = 60) were analyzed using ANOVA and many-faceted Rasch measurement to determine which categories were frequently assigned discrepant scores and to estimate rater severity. Discourse analysis of six audiotaped negotiation sessions was then used to examine how raters’ understanding of rubric criteria converged over time. Our results indicate that through negotiation, raters used shared terminology and justifications to clarify ambiguous constructs and work to establish shared values. The results suggest that score negotiation influences scoring inferences and also creates affordances for raters to ground those inferences in shared constructions of meaning.

元の言語English
ページ(範囲)32-43
ページ数12
ジャーナルAssessing Writing
30
DOI
出版物ステータスPublished - 2016 10 1

Fingerprint

interpretation
performance assessment
quantitative method
discourse analysis
qualitative method
technical language
subjectivity
English language
Raters
language
resources
Scoring
Values

ASJC Scopus subject areas

  • Language and Linguistics
  • Education
  • Linguistics and Language

これを引用

“I can see that” : Developing shared rubric category interpretations through score negotiation. / Trace, Jonathan; Meier, Valerie; Janssen, Gerriet.

:: Assessing Writing, 巻 30, 01.10.2016, p. 32-43.

研究成果: Article

@article{bd7dfaa4a1874fc080dc173bbac34c2d,
title = "“I can see that”: Developing shared rubric category interpretations through score negotiation",
abstract = "Performance assessments using raters will always contain some subjectivity, and disagreement among raters necessitates reliable methods for resolving scores. Negotiation is one effective method to guide scoring decisions and reduce raters’ tendencies to be unexpectedly severe or lenient when scoring specific rubric categories or examinees. Beyond its utility for scoring, however, negotiation is also a resource for raters to co-construct interpretations about the language constructs being measured. This study uses quantitative and qualitative methods to trace how negotiation impacts raters’ scoring decisions and examine in detail how raters develop joint interpretations of rubric category criteria. Scores from the writing section of a high stakes English language placement exam (n = 60) were analyzed using ANOVA and many-faceted Rasch measurement to determine which categories were frequently assigned discrepant scores and to estimate rater severity. Discourse analysis of six audiotaped negotiation sessions was then used to examine how raters’ understanding of rubric criteria converged over time. Our results indicate that through negotiation, raters used shared terminology and justifications to clarify ambiguous constructs and work to establish shared values. The results suggest that score negotiation influences scoring inferences and also creates affordances for raters to ground those inferences in shared constructions of meaning.",
keywords = "Academic writing, Negotiation, Performance assessment, Raters, Rubrics, Sociocultural learning",
author = "Jonathan Trace and Valerie Meier and Gerriet Janssen",
year = "2016",
month = "10",
day = "1",
doi = "10.1016/j.asw.2016.08.001",
language = "English",
volume = "30",
pages = "32--43",
journal = "Assessing Writing",
issn = "1075-2935",
publisher = "Elsevier Limited",

}

TY - JOUR

T1 - “I can see that”

T2 - Developing shared rubric category interpretations through score negotiation

AU - Trace, Jonathan

AU - Meier, Valerie

AU - Janssen, Gerriet

PY - 2016/10/1

Y1 - 2016/10/1

N2 - Performance assessments using raters will always contain some subjectivity, and disagreement among raters necessitates reliable methods for resolving scores. Negotiation is one effective method to guide scoring decisions and reduce raters’ tendencies to be unexpectedly severe or lenient when scoring specific rubric categories or examinees. Beyond its utility for scoring, however, negotiation is also a resource for raters to co-construct interpretations about the language constructs being measured. This study uses quantitative and qualitative methods to trace how negotiation impacts raters’ scoring decisions and examine in detail how raters develop joint interpretations of rubric category criteria. Scores from the writing section of a high stakes English language placement exam (n = 60) were analyzed using ANOVA and many-faceted Rasch measurement to determine which categories were frequently assigned discrepant scores and to estimate rater severity. Discourse analysis of six audiotaped negotiation sessions was then used to examine how raters’ understanding of rubric criteria converged over time. Our results indicate that through negotiation, raters used shared terminology and justifications to clarify ambiguous constructs and work to establish shared values. The results suggest that score negotiation influences scoring inferences and also creates affordances for raters to ground those inferences in shared constructions of meaning.

AB - Performance assessments using raters will always contain some subjectivity, and disagreement among raters necessitates reliable methods for resolving scores. Negotiation is one effective method to guide scoring decisions and reduce raters’ tendencies to be unexpectedly severe or lenient when scoring specific rubric categories or examinees. Beyond its utility for scoring, however, negotiation is also a resource for raters to co-construct interpretations about the language constructs being measured. This study uses quantitative and qualitative methods to trace how negotiation impacts raters’ scoring decisions and examine in detail how raters develop joint interpretations of rubric category criteria. Scores from the writing section of a high stakes English language placement exam (n = 60) were analyzed using ANOVA and many-faceted Rasch measurement to determine which categories were frequently assigned discrepant scores and to estimate rater severity. Discourse analysis of six audiotaped negotiation sessions was then used to examine how raters’ understanding of rubric criteria converged over time. Our results indicate that through negotiation, raters used shared terminology and justifications to clarify ambiguous constructs and work to establish shared values. The results suggest that score negotiation influences scoring inferences and also creates affordances for raters to ground those inferences in shared constructions of meaning.

KW - Academic writing

KW - Negotiation

KW - Performance assessment

KW - Raters

KW - Rubrics

KW - Sociocultural learning

UR - http://www.scopus.com/inward/record.url?scp=84988449061&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84988449061&partnerID=8YFLogxK

U2 - 10.1016/j.asw.2016.08.001

DO - 10.1016/j.asw.2016.08.001

M3 - Article

AN - SCOPUS:84988449061

VL - 30

SP - 32

EP - 43

JO - Assessing Writing

JF - Assessing Writing

SN - 1075-2935

ER -