Challenges and Misinterpretations of Cohen's Kappa in Agreement Studies in Ophthalmology.

Cohen's kappa is prevalence-dependent; use supplementary indices like PABAK when interpreting agreement in skewed ophthalmology screening datasets.

This study examines the challenges and potential misinterpretations of Cohen's kappa, a commonly used statistical measure of inter-rater agreement, in the context of veterinary ophthalmology. Using an existing dataset of 60 client-owned dogs undergoing pre-breeding gonioscopy examinations, two examiners independently classified iridocorneal angle abnormalities as either 'breeding-YES' or 'breeding-NO' according to the 2022 ECVO-HED grading scheme. The study demonstrated a striking paradox: despite both examiners disagreeing on 7/60 left eyes in both the real and simulated datasets, Cohen's kappa values differed dramatically—0.18 for the imbalanced real data versus 0.77 for the artificially balanced dataset. This discrepancy arose because Cohen's kappa is highly sensitive to outcome prevalence; when one category predominates (as is common in screening populations where most animals are classified as 'breeding-YES'), kappa values are artificially deflated even when actual agreement is clinically acceptable. By contrast, right eye data showed near-perfect kappa (0.91) with only 1 disagreement out of 58 eyes, illustrating how low disagreement rates with balanced prevalence yield high kappa values. The authors recommend that veterinary ophthalmologists supplement Cohen's kappa with additional indices such as the prevalence index, bias index, prevalence-and-bias-adjusted kappa (PABAK), and maxKappa when data distributions are skewed. These supplementary tools do not replace Cohen's kappa but provide critical context for accurate interpretation, particularly in screening scenarios where disease prevalence is inherently low.

This summary was distilled by AI and may occasionally misinterpret data. Confirm critical details with the primary literature before clinical application.