LO22: Risk-stratification of emergency department syncope by artificial intelligence using machine learning: human, statistics or machine

Abstract

Introduction: The Canadian Syncope Risk Score (CSRS) is a validated risk tool developed using the best practices of conventional biostatistics, for predicting 30-day serious adverse events (SAE) after an Emergency Department (ED) visit for syncope. We sought to improve on the prediction ability of the CSRS and compared it to physician judgement using artificial intelligence (AI) research with modern machine learning (ML) methods. Methods: We used the prospective multicenter cohort data collected for the CSRS derivation and validation at 11 EDs across Canada over an 8-year period. The same 43 candidate variables considered for CSRS development were used to train and validate the four classes of ML models to predict 30-day SAE (death, arrhythmias, MI, structural heart disease, pulmonary embolism, hemorrhage) after ED disposition. Physician judgement was modeled using the two variables, referral for consultation and hospitalization. We compared the area under the curve (AUC) for the three models. Results: The proportion of patients who suffered 30-day SAE in the derivation cohort (N = 4030) was 3.6% and in validation phase (N = 2290) was 3.4%. Characteristics of the both cohorts were similar with no shift. The best performing ML model, a gradient boosting tree-based model used all 43 variables as predictors as opposed to the 9 final CSRS predictors. The AUC for the three models on the validation data were: best ML model 0.91 (95% CI 0.87–0.93), CSRS 0.87 (95% CI 0.83–0.90) and physician judgment 0.79 (95% CI 0.74 - 0.84). The most important predictors in the ML model were the same as the CSRS predictors. Conclusion: A ML model developed using AI method for risk-stratification of ED syncope performed with slightly better discrimination ability though not significantly different when compared to the CSRS. Both the ML model and the CSRS were better predictors of poor outcomes after syncope than physician judgement. ML models can perform with similar discrimination abilities when compared to traditional statistical models and outperform physician judgement given their ability to use all candidate variables.