Triage is a critical process in most emergency departments (EDs) and is defined as the first point of a dynamic process chain when a patient arrives to the ED. Reference Grosgurin, Gayet-Ageron and Suppan1,Reference Aacharya, Gastmans and Denier2 At this point, complaints of the patients and injured people are assessed according to the priority of urgency to provide adequate medical attention. Reference Morley, Unwin and Peterson3 An initial clinical assessment is essential to distinguish the patients and recognize the ones in need of immediate care. This triage decision-making critically affects patient safety and their access to timely care in the ED. Reference Varndell, Hodge and Ryan4 In this context, a variety of 5-point-triage scales have been developed to assess patient urgency. Reference Grosgurin, Gayet-Ageron and Suppan1 Manchester triage scale (MTS) in the United Kingdom, the Canadian triage acuity scale (CTAS) in Canada, the Australian Triage Scale (ATS) in Australia, and the Emergency Severity Index (ESI) in the United States of America are currently in use and have been nationally adopted. Reference Christ, Grossmann and Winter5,Reference Robertson-Steel6 However, although above-mentioned systems share common characteristics, their approaches have substantial differences; CTAS, ATS, and MTS use the vital signs and clinical discriminators combinations, whereas ESI focuses on prediction of resource requirements in ED. Reference Ebrahimi, Heydari and Mazlom7–Reference Parenti, Reggiani and Iannone10 Hence, traditional triage assessments mostly rely on the triage guidelines and experience of the health professional in charge. Reference Dong, Bullard and Meurer11 A well-validated and accurate triage system is pivotal for an optimal assessment of patients’ urgency; therefore, the development of an electronic triage decision process may guide identification of patients’ condition and level of urgency. Reference Schuetz, Hausfater and Amin12 Moreover, there is still an ongoing debate about whether adopting an international system or developing new triage methods at national level, because it is important that the same system is used throughout health-care organizations. Reference Lahdet, Suserud and Jonsson13 Triage is not well developed in Turkey and is performed by physicians, nurses, or paramedics.Reference Kahveci, Demircan and Keles14 Most of the time, nurses or paramedics are in charge of triage. In Turkey, a paramedic is a health-care professional that provides emergency medical services after fulfilling a 2-y full-time program.
The emergency triage instrument mandated by the Ministry of Health, is routinely used in our institution. Reference Erimsah, Yaka and Yilmaz15 The instrument is a 3-level scale that allocated patients to either the red, yellow, or green codes in descending order of acuity; each color code is then divided into 2 categories. The instrument is paper-based and differs from international triage systems with its 6 levels. This triage system is not algorithmic and not useful as it does not include all complaints and consists of only a few identified examples (Appendix 1).
Therefore, we aimed to develop a 5-level computer-based software that is able to display key elements of complaints to identify patient’s urgency. To expand use of such software, the triage support tool in question must be feasible and effective to perform triage in a standardized and simple manner within the framework of measurable criteria.
In this study, we examined the reliability and validity of a new computerized triage decision support tool, ANKUTRIAGE.
Methods
Study Design
This was a prospective study conducted in real time in an academic ED environment. The study was approved by the Health Research Ethics Board of our institution (number: 12-682-17).
Study Setting and Population
The study was conducted in a large tertiary care hospital with an annual approximate number of 60,000 ED admissions. Considering the density of the patients admitted to the ED, weekdays, weekends, day-night shifts were randomized and the time intervals for the study were determined. All adult patients admitted to the ED with any complaint were eligible for inclusion. Informed consent was obtained from patients or legal guardians. The need for consent for critically ill patients was waived and patient care was not altered. Critically ill patients were sent directly to the appropriate area in the ED to initiate urgent treatment or lifesaving procedures.
The ANKUTRIAGE System
The ANKUTRIAGE system was developed between June 1, 2018, and June 30, 2019, by a project team of 12 people consisting of 6 emergency physicians, 2 pediatric emergency physicians, 1 pediatrician, 1 computer engineer, and 2 software development experts.
The engineer and software experts had no medical knowledge but were familiar with hospital software design. The ANKUTRIAGE system was established to categorize patients presenting to the ED in a scale ranging from 1 to 5. Triage scores are given according to 5 urgency levels as follows: Level 1, emergency; Level 2, high urgency; Level 3, urgency; Level 4, low urgency; Level 5, no urgency. Two major factors are considered while determining the urgency of patients: patient’s vital signs and characteristics of the complaints at the admission. The clinicians in the project team determined valid vital parameters for each level of urgency examining the worldwide accepted triage systems such CTAS, MTS, and ATS (Table 1). Complaints and related discriminative questions were defined by considering those triage systems and the International Classification of Diseases 10th Revision (ICD-10). 16 Discriminators like vital signs and surrogate questions are factors that allocate patients to 1 of the 5 triage levels. Vital signs (body temperature, pulse, respiratory rate, oxygen saturation, systolic blood pressure, diastolic blood pressure), general appearance of the patient and level of consciousness assessments are recorded by the triage officer. In case of problem regarding consciousness, blood sugar assessment is advised by the system. Similarly, if any high blood pressure values are determined, the triage officer is warned to ask any symptoms related to hypertension. Moreover, 4 important points regarding triage as forensic issues, need of isolation, need of decontamination and agitated patient status can also be checked. In addition, definitions of “general condition of the patient” and “critical patient”, which are key features during triage process, were made. “Critical patient” status was defined as life-threatening conditions such as airway compromise, inadequate breathing, shock, uncontrollable major hemorrhage, and unresponsive patient.
Table 1. Variables of vital signs according to triage levels
Abbreviations: BT, body temperature; DBP, diastolic blood pressure; SBP, systolic blood pressure.
*Sickle cell anemia or absence of spleen, venticuloperitoneal shunt, catheter, known or suspected neutropenia, chemotherapy, steroid use, immunosuppressive therapy, transplant and oncology (cancer).
If any critical value is detected, the triage officer is warned by the system. Then, the major complaint of the patient is selected from a standardized complaint list and relevant discriminative questions that lead the user to assess the appropriate triage score are displayed. Critical features and questions impelling clues to the urgency of the patient during triage assessment are listed under the main complaint. For each complaint, general and specific discriminators were determined. For example, as surrogate questions for a patient presenting with “chest pain”, pain severity and radiation, cardiac history, and other related symptoms are listed under the main complaint. The assessment is carried out by finding the highest level at which the answer posed by the discriminator question is positive. An example of “chest pain” complaint triage is given in Figure 1. Hence, an effective and complete questioning is made independent of factors such as experience or emotional state of the triage officer.
Figure 1. Example of the “chest pain” complaint in the ANKUTRIAGE system (screenshot translated after discriminators appear).
To ensure accurate and effective use of the software, health-care professionals (doctors, nurses, and paramedics) were trained on how to apply ANKUTRIAGE. A 2-h lecture about the software system was provided by 2 emergency physicians experienced in emergency triage, and all participants were given 20 scenarios to improve their practice.
ANKUTRIAGE system was applied twice on all patients admitted to the ED simultaneously at the point of ED arrival, first by the regular triage nurse on duty working at triage (ANKU1) and by the researchers conducting the study (ANKU2), who were last year emergency medicine residents. Both ANKU1 and ANKU2 listened to the presenting patient’s complaint at the same time; the 2 assessors independently determined the triage level and were blinded to each other’s decision. Besides ANKU1 and ANKU2, a final triage level was assessed by an emergency triage nurse experienced in ED triage (Expert1) and an emergency physician (Expert2) in a separate area. The 2 experts made their decision about a patient’s urgency level, without using any triage decision support tool, according to their knowledge and experience.
Measurements and Statistical Analysis
The sample size of the study, in order to find the difference between kappa 81 and 85 with 80% power at 5% margin of error, was calculated by assuming a percentage of patients for each triage level; level 1, 10%; level 2, 15%; level 3, 35%; level 4, 20%; and level 5, 20%. A total of 1023 patients were planned to be included in the study.
To measure inter-rater reliability between triage scores (ANKU1/ANKU2) (Expert 1/Expert 2), quadratic-weighted kappa coefficients (Kw) were calculated. K values were defined as follows: (K > 0.8), excellent; (0.6 < K < 0.8), good; (0.4 < K < 0.6), moderate; (0.2 < K < 0.4), fair; or (K < 0.2), poor.
For the validity, surrogate markers as clinical outcomes, and resource use were determined. Diagnostic resource or therapeutic procedures and lifesaving procedures were recorded. Diagnostic resources included blood and urine examinations, X-ray, electrocardiogram (ECG), ultrasound (US), computed tomography (CT), and magnetic resonance imaging (MRI). Therapeutic procedures included intravenous (IV), intramuscular (IM) or nebulized drug therapy and IV fluid administration. All lifesaving procedures including mechanical ventilation, defibrillation, intubation, cardioversion, external pacing, needle decompression, pericardiocentesis, and thoracotomy were registered. Clinical outcomes were determined as 1-mo mortality, admission to intensive care unit (ICU), hospitalization, and discharge from ED.
Differences among 3 or more groups for ordinal variables or nonnormally distributed continuous variables were evaluated by Kruskal-Wallis variance analysis. When a P value lower than 0.05 was achieved after Kruskal-Wallis test, Dunn’s multiple test was used for intergroup differences. Statistical differences between 2 groups for ordinal variables or nonnormally distributed continuous variables were assessed by Mann-Whitney U test. Statistical differences between 2 groups in terms of categorical variables were analyzed by using Chi-Square test. A P-value less than 0.05 was considered statistically significant.
Results
Population
A total of 1100 patients were enrolled between July 1, 2019, and October 31, 2019. A total of 519 (47.1%) of the patients included in the study were male and the mean age of the patients was 46.1 ± 19.5 y (min-max = 18-95). The characteristics of the study cohort are given in Table 1. The number of patients assigned to each triage level by all triage officers is shown in Figure 2. Level 3 was the most commonly assigned score by all health-care professionals independent of method used.
Figure 2. Triage level score distribution according to all triagers.
Reliability
Agreement between users of ANKUTRIAGE was excellent with a Kw greater than 0.8 in all groups (Table 3). Inter-rater reliability showed a weighted K of 0.97 (95% confidence interval [CI]: 0.9707-0.9843) between triage practitioners (ANKU1 vs ANKU1) who determined the level with ANKUTRIAGE. Agreement between ANKU1 and expert opinions (Expert 1-Expert 2) was determined as 0.9279 and 0.947, respectively. Agreement between ANKU2 and expert opinions (Expert1-Expert 2) was determined as 0.9393 and 0.9484, respectively.
Table 2. Characteristics of the study cohort
Table 3. Agreement between triagers
Validity
Because ANKU1 was the first health-care professional to assign a triage level to the patient with the software ANKUTRIAGE, levels determined by ANKU1 were used for validity assessment. ANKU1 assigned 26 of 1100 patients to level 1, 127 to level 2, 381 to level 3, 267 to level 4, and 299 to level 5 (Figure 2).
Resources
Resource consumption was significantly proportional to urgency levels of the patients (P < 0.001). Distribution of therapeutic and lifesaving procedures was significantly comparable with regard to the severity of patients as reflected by the triage levels. There was no significant difference between level 4 and 5 in terms of resource use (Figure 3). Patients with emergent or urgent levels of triage were more likely to undergo a lifesaving procedure (34.6% of level 1 patients and 11.8% of level 2 patients), whereas only 1.85% of level 3 patients and none of level 4 and level 5 patients underwent lifesaving procedures (P < 0.05).
Figure 3. Box plot representation. A: Resource use by triage level. B: Resource use by patient clinical outcomes. Bars represent the median; boxes represent the interquartile range.
Clinical Outcomes
Of 1100 patients included in the study, 11 died within a month, 37 were admitted to the ICU, 62 were hospitalized, 964 were discharged from ED, and 37 patients left without authorization. A total of 37 patients that left without authorization were omitted from the outcome results.
When all patients were compared, as the level of urgency decreased, intensive care hospitalization was found to be significantly reduced (P < 0.05). A total of 16.7% of level 1 patients were admitted to the ICU, whereas only 1.4% of level 5 patients were admitted to ICU. Similarly, as the triage level decreased, the rate of hospitalization of patients decreased significantly with regard to the urgency levels (P < 0.05) (Table 4). A total of 11 patients died: 1 patient (3.8%) was assigned to level 1, 3 (2.4%) to level 2, 6 (1.6 %) to level 3, none of the patients in level 4 died within a month, and a 30-y-old patient who was assigned to level 5 died. Of the 11 patients who died, 9 had a malignant medical history (Table 5). On the other hand, the patient who died in level 5 was admitted to the ED with nonspecific abdominal pain assessed as 2 of 10 on the visual analogue scale, his vital signs were normal, and he had no medical history. During his course in the ED, the patient was diagnosed with peptic ulcer perforation, and transferred to the operating room. He died on postoperative day 17 because of “abdominal sepsis”.
Table 4. Intensive care unit admission, hospitalization, and discharge from emergency department rates among all patients by urgency levels (* P <0.05)
Abbreviation: ED, emergency department.
Table 5. Characteristics of patients with 1-mo mortality
Resource use and outcomes relationships were also evaluated. Resource consumption was higher in patients admitted to the ICU compared with the patients discharged from the ED (Figure 3).
Discussion
This is the first study to evaluate the reliability and validity of the ANKUTRIAGE.
ANKU1 triage officer was a regular triage nurse on duty, and we found an excellent agreement between ANKU1 AND ANKU2, who was a senior emergency resident. Reliability between different professional groups (doctor and nurse) can be challenging due to differences in knowledge, experience, and patient approach. The reliability of a measuring instrument is mainly measured by the reproducibility of test results and the ability of the instrument to repeatedly measure the same results. Reference Pourasghar, Daemi and Sadegh Tabrizi17 Many studies have reported poor agreement between nurses and physicians with a Kw less than 0.5. Reference Brillman, Doezema and Tandberg18,Reference Loke, Liaw and Tiong19 Because the agreement between ANKU1 and ANKU2 (Kw:0.9775) was found to be the highest, our results suggest that the actual profession did not affect the triage decision when using the decision-making support. Because agreement between ANKU1 versus ANKU2 was higher than Expert1 versus Expert2 (Kw:0.9409), our results demonstrate that, even if triage is performed by health professionals with triage experience like Expert1 and Expert2, using the ANKUTRIAGE system leads to higher reliable decisions for patients’ urgency levels (Table 2). These results may also suggest that using a standardized triage support tool increase reliability between triage officers by avoiding effects like personal mood or knowledge.
Major limitations of previous studies examining the inter-rater reliability of various triage scales were that the majority of these studies were conducted using triage scenarios or clinical data. Reference Taboulet, Moreira and Haas20,Reference Jobe, Ghuysen and Gerard21 There are very few studies conducted with real patients. Reference Tanabe, Gimbel and Yarnold22–Reference van Veen, Teunen-van der Walle and Steyerberg24 Our study revealed the reliability with real-life patients, instead of simulated scenarios commonly preferred in other studies. Reference Grosgurin, Gayet-Ageron and Suppan1,Reference Parenti, Reggiani and Iannone10 We believe that the simulated clinical scenarios cannot reflect the triage process.
There are various main elements that conduce to inaccuracy in ED triage. Reference Dong, Bullard and Meurer25 The first 1 is that the health-care professional (HCP) in charge of triage has to anticipate resource needs of the existent medical condition. ESI algorithm relies on the basis of acuity and resource needs. This estimation may vary according to HCP’s own experience and knowledge and also according to medical resources availability which is subject to differ from hospital to hospital. Another factor is the intricacy of reference triage systems. For example, MTS presents 53 charts of several clinical presentation that can assist in streaming patients to the most appropriate pathway of care, while CTAS refers to complaint-based triage that could be translated changeably in routine practice comparing to the Canada. Reference Mirhaghi, Heydari and Mazlom9,Reference Zhiting, Jingfen and Shuihong26 Khorram-Manesh et al. aimed to evaluate the possibility of creating a translational triage tool, using criteria used in most common pre-existing prehospital triage systems. They were able to demonstrate that common characteristics are repeated in all current prehospital triage systems, and, that it is feasible to combine various triage systems in order to develop a unified global system. Reference Karjala and Eriksson27 Even if ANKUTRIAGE is not a translational triage tool, during the first step of the tool’s development, current reference triage systems were reviewed by the project team. Translation or adaptations of triage scales are difficult because of language induced errors and major differences in medical care management due to local and regional specifities. Reference Taboulet, Moreira and Haas20 Therefore, implementing new triage methods at national level is important to uniform and standardize admission systems throughout health-care organizations. Previously, an online application of CTAS, eTRIAGE has been shown to be easy-to-use, reliable, and more valid than approaches relying on memory. Reference Dong, Bullard and Meurer11 Considering ED triage realities such as overcrowding, long hours of high patient volume, patients urge to be seen, a reliable and valid triage system is necessary to manage ED patients demand while trying to assure minimum patient harm. Moreover, meticulously established discriminators for each complaint according to local characteristics and factors without language constraints are essential for accurate triage assessment.
ANKUTRIAGE is easy to use, the HCP in charge of triage does not have to anticipate resource consumption and the tool supports the triager in decision-making by asking adequate questions according to the complaint.
Surrogate markers as resource consumption, hospital admission and mortality have been considered to be validity criteria and they have been widely used with this purpose in similar triage studies. Reference Loke, Liaw and Tiong19,Reference Taboulet, Moreira and Haas20 However, both mortality and admission are issues that complicate accurate assessment of predictive validity. For instance, a metastatic cancer patient presenting with non-urgent conditions to the ED becomes a potential inpatient whereas some patients with urgent condition at admission like anaphylaxis or joint dislocation will probably be discharged directly from the ED. Reference Khorram-Manesh, Nordling and Carlström28 Therefore, because definitive reference standard for validity has not been established, correlation between urgency for medical attention and resource use, clinical outcome, or hospital admission remains challenging to interpret. Reference Tanabe, Gimbel and Yarnold22,Reference Mitchell, Bue and Nou29 However, ANKUTRIAGE was designed to precisely describe the difficulties and variability of patients’ assessments. Resource consumption and therapeutic procedures had good correlation with the triage ranking because patients with high urgency level were more likely to undergo lifesaving and therapeutic procedures. Storm-Versloot et al. reported that the number of resources used for a patient increases with the higher level of urgency of MTS. Reference Kuriyama, Urushidani and Nakayama30 In other studies, it was stated that the rate of hospital admission increases with a higher level of urgency and more than 80% of admitted patients who were triaged were urgency level 1 or 2. Reference Storm-Versloot, Ubbink and Kappelhof31,Reference Martins, Cuña and Freitas32 In our study, we were also able to demonstrate a good correlation between clinical outcomes and triage categories; our triage system can reliably predict outcomes such as ICU or hospital admission outcomes (Table 4). Validity of triage systems depends on their ability to distinguish levels of urgency. Accurate classification of high-urgency patient is closely related to patient safety, while correct classification of low-urgency patients increases the efficiency of the ED flow and shortens waiting times. Reference Roukema, Steyerberg and van Meurs33,Reference Gilboy, Tanabe and Travers34 Therefore, an ideal triage system must balance between safety and accuracy. Thus, ANKUTRIAGE system’s level assignment was significantly correlated with the number of resources used, the need of lifesaving procedures, ICU and hospital admission rates.
Triage remains a central process for safe management of patients under circumstances of high demand that is common in many EDs. Increases in patient volume in countries with new evolving emergency care systems strongly suggests the need for more accurate and reliable triage. Emergency Department triage used to be intuitive rather than methodological, and rather than being based on clinical evidence, it was based on the practitioners’ own experience and clinical judgement; making it neither consistent nor auditable. Reference Zachariasse, van der Hagen and Seiger35 Advancements in decision support have opportunity to better control variability and enhance triage performance. Over the years, triage took its own place in hospitals and became a well-adopted process; however, a more integrated triage with a common language is still needed for optimal results. Considering characteristic components of EDs, such as increased demand and limitations of the staff, it is pertinent to implement a computerized decision support system to make the process of providing health care more organized and efficient. The main benefit of using this computer-based triage instrument was to ensure that triage was not person-dependent. We have demonstrated that ANKUTRIAGE is not affected by experience, mood, and conditions of the working environment of the triage staff, and ensures the application of triage in a standardized manner within the framework of measurable criteria.
Limitations
Our study had several limitations. First, this study was conducted at a single academic ED with a limited number of patients; therefore, the results need to be validated in a prospective multicenter study. Second, although ANKUTRIAGE provides decision support for pediatric patients, we included only patients’ ≥ 18 y old in this study, because the study took place in an ED admitting adult patients. Finally, we could not record the time taken to perform triage; therefore, further studies are needed to evaluate the effects of ANKUTRIAGE on triage time.
Conclusions
ANKUTRIAGE proved to be a valid and reliable tool in an academic ED. It showed that displaying the key discriminator for each complaint to assist decision led to high inter-rater agreement with good correlation between urgency levels and clinical outcomes, as well as between urgency levels and resource consumption. Accuracy in triage is a matter of assessing patients objectively according to a set of parameters of vital signs or admission complaint, without being affected by environmental or personal biases. The importance of implementing the triage methodology in a standard format cannot be stressed enough. This computerized tool can promote accurate triage; however, a multicenter study is necessary to corroborate these preliminary findings, indicate the adjustments needed for different health contexts, and assess the external validity of the instrument.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/dmp.2022.101.
Acknowledgments
All authors have made substantial contributions to all of the following the conception and design of the study, acquisition of data, or analysis and interpretation of data, drafting the article and revising it critically for important intellectual content, all authors give final approval of the version to be submitted.
Funding statement
This study was supported by grants from the Scientific and Technological Research Council of Turkey (TÜBİTAK) Project Number: 277S788, Ankara, Turkey.
Conflicts of interest
The authors declare that there is no conflict of interest.
Ethical standards
All procedures followed were in accordance with the ethical standards of the responsible committee on human experimentation (number: 12-682-17) and with the Helsinki Declaration of 1975. Informed consent was obtained from all patients for being included in the study.
Open access
