The ability to measure how closely raters agree when providing subjective evaluations is a need common to many fields of research. A main criticism arising in agreement studies concerns the dependency of the commonly adopted κ-type agreement coefficients on the frequency distribution of ratings over classification categories. This study aims at investigating the robustness of two inferential benchmarking procedures adopted for characterizing the extent of inter-rater agreement when applied to different κ-type coefficients.
Robustness of κ-type coefficients as measures of rater consistency / Vanacore, A.; Pellegrino, M. S.. - (2019), pp. 363-366. (Intervento presentato al convegno 9th International Conference IES 2019 - Innovation & Society - Statistical evaluation systems at 360°: techniques, technologies and new frontiers tenutosi a Università Europea di Roma nel Luglio 2019).
Robustness of κ-type coefficients as measures of rater consistency
Vanacore A.
;Pellegrino M. S.
2019
Abstract
The ability to measure how closely raters agree when providing subjective evaluations is a need common to many fields of research. A main criticism arising in agreement studies concerns the dependency of the commonly adopted κ-type agreement coefficients on the frequency distribution of ratings over classification categories. This study aims at investigating the robustness of two inferential benchmarking procedures adopted for characterizing the extent of inter-rater agreement when applied to different κ-type coefficients.I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
