Triage reliability studies typically use hypothetical scenarios and weighted kappa scores where agreement within one level is considered satisfactory. But if triage category is used to help define ED case-mix groups for comparative or benchmarking processes, agreement on exact triage level and major system involved is important. Our hypothesis was that a computerized menu that links presenting complaints to preferred triage levels (PC-linked triage) would provide high triage reliability.

Our objective was to assess inter-rater reliability of PC-linked triage using the Canadian Emergency Department Triage and Acuity Scale (CTAS) in a real-time clinical setting, considering agreement on exact triage level and primary body system involved.

On duty triage nurses entered patient presenting complaint and PC-linked triage level as per standard procedure. In a convenience sample of patients, a second nurse, blinded to triage assignment, observed the triage interaction and independently entered presenting complaint and triage level on a dummy terminal.

During the study, 15 nurse pairs triaged 266 patients. Study patients matched actual emergency department case mix closely. Triage nurses agreed exactly in 74% of cases and within one level in 94% of cases. The unweighted kappa value was 0.66 (95% confidence interval [CI], 0.60–0.73) and the quadratic weighted kappa value was 0.75 (95% CI, 0.68–0.81). Kappa for agreement on major system involved was 0.80 (95% CI, 0.69–0.91).

PC-linked triage has high inter-rater reliability in a real-time clinical setting. PC-linked triage may be useful as one factor in defining case-mix groups for benchmarking and comparative purposes.