Global inter-rater reliability, scale validity, and local perception: Panss ratings and reactions from 4 Countries

Abstract

Introduction/objectives/aims

We compared cohorts of raters from different countries who received training on the PANSS. We attempted to determine if there was any consistent by-country impact on specific items, factors, or subscales. We also queried raters about their perceptions of the instrument they were asked to use vis-à-vis their local patient population.

Methods

The data set comes from standardized rater training events involving raters from four countries: India (n = 83), Russia (n = 59), the US (n = 63), and Romania (n = 76). Raters scored interviews of schizophrenic patients using the PANSS. Scores were compared and intra-class correlation coefficients (ICCs) and rater agreement with “gold standard” scores were evaluated. The results were viewed against raters’ responses to questions about how well the PANSS items correlated to the presentation of symptoms.

Results

Raters from the US and Russia demonstrated a higher level of inter-rater consistency with ICCs of 0.883 and 0.835, respectively. For eight PANSS items, all raters demonstrated at least 80% agreement with the gold standard scores. For ten PANSS items, there was at least one country whose raters scored below 60% agreement. The PANSS items with the lower inter-rater reliability were the same items raters indicated as problematic in local settings.

Conclusion

The differences in rater performance indicate that standardized rater training is broadly effective but that there are some important differences in the way in which different groups conceptualize symptomatology and corresponding PANSS items. This suggests a need to tailor training to ensure reliability and validity in the use of this instrument.