Systematic review of risk prediction scores for surgical site infection or periprosthetic joint infection following joint arthroplasty

Summary

Accurate identification of individuals at high risk of surgical site infections (SSIs) or periprosthetic joint infections (PJIs) influences clinical decisions and development of preventive strategies. We aimed to determine progress in the development and validation of risk prediction models for SSI or PJI using a systematic review. We searched for studies that have developed or validated a risk prediction tool for SSI or PJI following joint replacement in MEDLINE, EMBASE, Web of Science and Cochrane databases; trial registers and reference lists of studies up to September 2016. Nine studies describing 16 risk scores for SSI or PJI were identified. The number of component variables in a risk score ranged from 4 to 45. The C-index ranged from 0·56 to 0·74, with only three risk scores reporting a discriminative ability of >0·70. Five risk scores were validated internally. The National Healthcare Safety Network SSIs risk models for hip and knee arthroplasties (HPRO and KPRO) were the only scores to be externally validated. Except for HPRO which shows some promise for use in a clinical setting (based on predictive performance and external validation), none of the identified risk scores can be considered ready for use. Further research is urgently warranted within the field.

INTRODUCTION

Surgical site infections (SSIs) which can be classified as superficial wound infections, deep wound infections, or periprosthetic joint infections (PJIs) [1], are uncommon but serious complications of total joint replacements [2, 3]. PJIs can result in severe pain, functional deficits and even death [4–6]; and their management is a huge financial burden to health care systems [7, 8]. With increasing life expectancy and a growing indication for primary joint replacements [9], there will be a proportionate rise in the number of patients who will be affected by PJIs. An approach to tackle the increasing incidence of PJIs is to identify those people at high risk and offer appropriate interventions. Early and accurate identification of individuals at high risk of PJI influences clinical decisions and development of targeted preventive strategies, and helps to optimise resources required for detection of PJI. Several factors such as characteristics of the patient, surgical procedure and postoperative care, have been found to influence the risk of developing PJI [10, 11], however their potential utility for PJI risk assessment remains uncertain.

A risk score or prognostic model is a statistical equation that predicts an individual's disease risk based on a combination of the values of multiple predictors or risk factors [12]. Risk prediction scores are ideally developed using data from long-term follow-up of large population-based cohorts of individuals without a history of the event of interest (SSI or PJI in this case) at baseline. The dataset is used to identify important predictors and the model equation is developed [13]. Using the derivation sample, the score's apparent performance is evaluated in a process known as internal evaluation. The next stage is external validation, which examines the generalisability of the model using new data. Risk prediction scores first emerged in the area of cardiovascular disease (CVD) prevention and have been widely used globally in clinical and public health practice. Well known amongst them is the Framingham CVD risk score [14] (a risk score which assesses an individual's risk of a cardiovascular event within 10 years), which is a commonly used algorithm in clinical practice and accepted tool in preventive medicine.

Prevention of SSIs or PJI is a high policy priority and there has been an increasing interest in the development of risk prediction tools for SSI or PJI over the last decade. However, unlike the substantial progress made in CVD prevention using risk scores, the amount of progress made in the area of SSIs or PJIs is uncertain. There is therefore a need for objective data on the development of risk scores (including their component variables), their discriminative abilities, whether they have been externally validated, and whether their clinical effectiveness have been assessed in well-designed randomised controlled trials (RCTs). In this context, using systematic review methodology, we aimed to: (i) identify and summarise studies reporting the development of risk prediction scores for SSI or PJI; (ii) assess clinical variables selected for model inclusion and the predictive performance of these models; (iii) assess if identified models have been externally validated and their performances compared; (iv) assess if the impact or clinical effectiveness of these risk scores have been evaluated in appropriate RCTs and (v) finally to identify gaps in the existing evidence and whether further research is needed in the field.

METHODS

This review was conducted using a predefined protocol, which has been registered in the PROSPERO prospective register of systematic reviews (CRD42016042158), and in line with PRISMA guidelines [15] (Supplementary Material 1). We searched MEDLINE, EMBASE, Web of Science and the Cochrane Library electronic databases up to 30 September 2016. The publicly available trial registers ClinicalTrials.gov, UKCRN (UK Clinical Research Network) Study Portfolio Database, and the WHO International Clinical Trials Registry Platform were also searched. The search strategy combined free and MeSH search terms and combination of key words relating to risk prediction (e.g., ‘predict’, ‘risk score’, ‘sensitivity’), SSI or PJI (e.g., ‘periprosthetic joint infection’, ‘deep infection’, ‘surgical site infection’), and joint replacement (e.g., ‘hip replacement’, ‘knee replacement’, ‘hip arthroplasty’, ‘knee arthroplasty’). No restrictions were placed on publication dates and only articles published in English were considered. Reference lists of retrieved articles and relevant review articles identified on the topic were manually scanned for all relevant additional studies. Detailed description of all Materials and Methods, as well as the Literature Search Strategy are available in Supplementary Materials 2 and 3.

RESULTS

Study identification and selection

Figure 1 shows the flow of studies through the review. Our literature search strategy identified 1802 potentially relevant articles. After the initial screening of titles and abstracts, 15 articles remained for further evaluation. Following detailed evaluation which included full-text reviews, six articles were excluded because (i) they were studies of diagnostic scores (n = 2) and (ii) they were studies of risk scores for outcomes such as readmission, infection eradication and treatment outcome of PJI (n = 4). The remaining nine articles met the inclusion criteria and were included in the review [16–24].

Fig. 1. PRISMA flow diagram.

Study characteristics and quality assessment

Table 1 summarises characteristics of the studies in the sample. Studies were published between 2006 and 2016, with all but one appearing in 2011–2016. One study was reported as a published conference abstract [16]. Overall, the studies involved 482 877 joint replacements, including 6968 SSIs or PJIs. For studies that reported age data, the baseline age of participants ranged from 56 to 81 years. The sample size of cohorts ranged from 217 to 172 055 and follow-up for infection outcomes ranged from 30 days to 2 years. For the assessment of infections, the majority of the studies used Centre for Disease Control or Infectious Diseases Society of America criteria. Studies classified infection outcomes as SSI or PJI specifically. One study employed both SSI and PJI outcomes [21] and another study used PJI recurrence [24]. Quality assessment using PROBAST showed evidence of high overall risk of bias throughout the included studies. Five risk scores had unclear concern for overall applicability and only two scores were deemed to be usable in the targeted individuals and context (the National Healthcare Safety Network (NHSN) SSIs risk models for hip and knee arthroplasties (HPRO and KPRO)) [18] (Supplementary Material 4).

Table 1. Summary characteristics of studies included in the review

ATC, Anatomic, Therapeutic and Chemical Classification; CDC, Centre for Disease Control; HPRO, National Healthcare Safety Network surgical site infections risk model for hip arthroplasty; KPRO, National Healthcare Safety Network surgical site infections risk model for knee arthroplasty; NHSN, National Healthcare Safety Network; NR, not reported; NS, not stated; PJI, periprosthetic joint infection; SSI, surgical site infections; THA, total hip arthroplasty; TKA, total knee arthroplasty; ICD-10-AM, International Statistical Classification of Diseases and Related Health Problems, Tenth Revision, Australian Modification.

* Indicates the total number of SSIs for both THA and TKA.

Model description and development

Table 2 provides details of risk scores included in eligible studies: their component predictors, statistical properties, measures of discrimination and/or calibration, and reports of any validation and performance comparisons made. A total of 16 risk scores were described in the nine eligible studies. Five of these scores had separate models for hip and knee replacement patients [18, 22, 23]. Four studies described the development of two or more risk scores [18, 19, 22, 23]. All 16 risk scores were derivations of risk models on a base population and two of them were also externally validated on new populations [21]. Except for one study that developed the risk score based on a cohort recruited prospectively for the surveillance of SSIs [17], all studies used datasets retrospectively that had been established for different purposes. Except for the scores that were developed in both knee and hip replacement patients, the component predictors varied from score to score. However, age, sex and type of primary surgery featured in the majority of risk scores. Except for one score that was mainly based on invasive data such as ESR (erythrocyte sedimentation rate), CRP (C-reactive protein) and microbial aetiology [24], all scores were based on data that can be assessed non-invasively such as demographics, anthropometrics, medical and surgical histories, and surgical procedures. The number of component variables in a single score ranged from 4 to 45 (n = 16, median 19, interquartile range 6·5–32·5). Seven out of the 16 risk scores had 10 or fewer components. Of the 16 risk scores, 15 used regression techniques (logistic or Cox) to develop the score and one used a classification tree [24].

Table 2. Key characteristics of risk prediction scores for PJI included in the review

ASA, American Society of Anesthesiologist; BMI, body mass index; CRP, C-reactive protein; ESR, erythrocyte sedimentation rate; HL, Hosmer–Lemeshow; HPRO, National Healthcare Safety Network surgical site infections risk model for hip arthroplasty; KPRO, National Healthcare Safety Network surgical site infections risk model for knee arthroplasty; IDI, Integrated Discrimination Index; N/A, not applicable; NNIS, National Nosocomial Infection Surveillance; NHSN, National Healthcare Safety Network; NR, not reported; NS, not stated; PJI, periprosthetic joint infection; SSI, surgical site infection; THA, total hip arthroplasty; TKA, total knee arthroplasty.

† It was an external validation study of the risk models HPRO and KPRO.

Model diagnostics

Except for three studies (comprising of three risk scores) [16, 20, 24], the C-statistic was reported for 13 risk scores. The C-index ranged from 0·56 to 0·74. Only three risk scores were reported to have a discriminative ability of >0·70 and these were the baseline Mayo and 1-month-postsurgery Mayo PJI risk scores as reported by Berbari et al. [19] and HPRO which was externally validated by Lewallen et al. [21], Calibration measures were presented for 11 risk scores (including the baseline Mayo PJI risk score) and each was reported to have satisfactory model calibration. Two studies did not report on any measures of discrimination or calibration [20, 24] (Table 2).

Model validation

Only five of the risk scores were validated internally using resampling techniques such as bootstrapping and cross-validation [17, 18, 23]. These included (i) a total hip arthroplasty (THA)-specific risk model for SSI, developed using data collected from 62 acute care hospitals within the Dutch surveillance network for nosocomial infections [17]; (ii) the HPRO and KPRO [18] and (iii) claims-based risk models for THA and total knee arthroplasty (TKA) [23]. Only the HPRO and KPRO risk scores were externally validated using an independent dataset in a different study [18]. Although the HPRO score performed better in the external cohort compared with the internal validation cohort, the KPRO risk score performed much less well when tested on the external cohort compared with the internal cohort (Table 2).

Performance comparisons

The performances of five risk scores were compared with existing models in three studies [17–19]. Geubbels et al. compared the predictive performance of their newly developed THA-specific risk score for SSI with the NNIS (National Nosocomial Infection Surveillance) system risk index (which incorporates three risk factors of equal weight namely wound contamination class, American Society of Anesthesiologists (ASA) score, and duration of surgery), and reported better predictive performance for the new risk score (C-index: 0·64 vs. 0·56; P < 0·001) [17]. Mu et al. also reported statistically significantly better performances for the HPRO and KPRO risk scores when compared with the traditional NHSN SSI risk model, though the C-statistics were generally low (<0·70) [18]. The baseline Mayo and 1-month-postsurgery Mayo PJI risk scores also performed well compared with the traditional NHSN SSI risk score (C-index: 0·72 vs. 0·64; P < 0·001) and (C-index: 0·72 vs. 0·63; P < 0·001), respectively [19]. Two studies assessed the incremental prognostic value of adding additional risk factors to their existing models [21, 23]. Lewallen et al. externally validated the HPRO and KPRO risk scores and reported that addition of information on morbid obesity and diabetes mellitus to each score modestly improved discrimination [21]. On addition of four clinical risk factors (morbid obesity, prior non-arthroplasties on the same joint, ASA score and operative time) to their claims-based risk models for THA and TKA, Maradit Kremers et al. reported improved performance (by C-statistics) for both models, though the THA model showed better performance than the TKA model [23]. There was however no noticeable improvement in calibration for both models. Finally, whiles there was an improvement in IDI (Integrated Discrimination Index) for the THA score, no significant improvement was seen for the TKA score: 0·37% (0·12% to 0·62%) and 0·09% (−0·02% to 0·21%), respectively.

Clinical evaluation of risk scores

None of the studies described the evaluation of the clinical effectiveness of a score in an intervention study or as part of an impact study aimed at changing patient outcomes.

DISCUSSION

Key findings

Using systematic review methods, we have reported the first overview of available risk assessment scores for SSI or PJI following joint replacement. Based on established quality criteria for risk scores [25, 26], none of the risk scores in our review were judged to be promising for use in clinical settings or public health practice, except for the HPRO. The HPRO is a procedure-specific risk score which was adapted from the traditional NHSN risk index using NHSN data and its purpose is for predicting SSI or PJI within 1 year of hip replacement [18]. The HPRO was found to perform better than the traditional NHSN risk index and external validation in an independent cohort showed high discriminative ability [21]. The HPRO also showed higher accuracy for predicting PJI compared with SSI. The data also show that risk prediction models for SSI or PJI have only been developed over the past 5 years. Of the 16 risk scores identified, only seven had 10 or few components included in the final score, with a number of scores having between 30 and 45 components. Although all 11 risk scores reporting calibration measures exhibited satisfactory calibration, only three of these risk scores were reported to have a discriminative ability of >0·70. Of all 16 risk scores, HPRO and KPRO were the only risk scores externally validated in an independent population. Quality assessment of the risk scores’ development and validation criteria showed all scores to have a high risk of bias. This was mainly due to the methodology used in assessment of predictors and outcomes, inappropriate handling of missing data, and lack of external validation.

Explanations and implications of findings

Our findings highlight the limited evidence available on appropriate risk scores for predicting SSI or PJI after joint replacement. Given the absence of an ideal risk score which can be used in a routine clinical setting, it appears that the potential value of risk scores in preventing SSI or PJI may have been underestimated in orthopaedic practice. The findings also highlight the use of poor methodology in the development of some of these risk scores. Although cross-sectional study designs were not included, the included studies were not free from bias and confounding. The majority of the designs were based on retrospective cohorts instead of prospective cohort designs, which are ideal for risk score modelling as predictor information can be ascertained blindly in relation to the outcome or disease [13]. None of the risk scores was developed in a cohort recruited for this sole purpose, which introduced an inherent selection bias. A key methodological issue was the absence of clear and detailed reporting of the treatment of missing data in all studies, which is of utmost importance prior to the development of risk scores [12]. Included studies used complete case analysis in the presence of missing data, which does not represent the entire population and reduces the sample size [13]. It has been shown that risk scores that use multiple imputation, produce more valid results and have better discrimination than tools that ignore such additional analyses [27]. There were also concerns with usability of the risk scores, as the majority of the risk scores had more than 10 variables. It is recognised that the simplicity of the model is an important criteria for developing clinically useful risk scores [28, 29]. Evidence suggests that complex models are more likely to provide overoptimistic predictions, especially when extensive variable selection has been performed [30]. Only five of the risk scores were validated internally using resampling methods, which are techniques which give a good indication of how optimistic the risk score may be [31]. Although internal validation is helpful, it cannot provide information on the model's performance elsewhere or its generalisability. Before a risk prediction tool can be used in clinical practice or in real-world settings, evaluation of its generalisability (or transportability) requires data from elsewhere – also known as external validation [12]. However, only two risk scores were externally validated in our sample [21]. Finally, none of the risk scores was reported to have been used in an impact study aimed at changing patient outcomes. Before a risk score can be implemented, a vital criterion that needs to be fulfilled is its impact on clinical practice [12]. Among the identified risk scores, only the HPRO was found to be potentially promising for use in a clinical setting. However, it cannot be considered ready for use as its clinical effectiveness is still yet to be evaluated. The unavailability of appropriate existing risk scores for use in the clinical setting is extremely concerning. To add to this challenge is the lack of established uniform criteria for the diagnosis of infection especially PJI, which actually makes it difficult to conduct diagnostic or risk prediction studies for infection. Although hip and knee replacements are successful elective procedures, with SSIs or PJIs being rare complications of these procedures [3, 32]; the incidence of these infections will increase in conjunction with growing healthcare burden due to osteoarthritis [33] and a predicted large rise in the numbers of arthroplasty procedures [34, 35]. To meet this challenge, there should be a clinical drive towards identification of individuals at high risk of SSIs or PJIs using risk prediction engines. The current findings should stimulate research groups to develop and evaluate appropriate infection outcome-specific risk prediction algorithms using robust methodology. The clinical effectiveness of the HPRO also needs to be evaluated before it is implemented. Within our 5 year INFORM (INFection ORthopaedic Management) Programme, the aim is to develop and establish optimum strategies for the prevention and treatment of PJIs within the UK National Health System [36], and which may include the development of appropriate risk prediction engines when the data allows.

Study strengths and limitations

To the best of our knowledge, this is the first systematic review to identify limited progress in the development and validation of risk prediction models for SSI or PJI following joint replacement, using robust systematic methodology. It is also the first review to assess the validity of existing risk scores based on risk of bias and applicability. Our search strategy was comprehensive and spanned multiple databases, making it unlikely that any relevant study was missed. There was variation in the definition of SSIs in the included studies, which did not allow for a head-to-head comparison of risk scores across studies. We were unable to harmonise data from contributing studies to perform a quantitative analysis, due to the heterogeneity in study designs and populations, predictors used, model types, and measures reported. Even though we tried to present the data as robustly as possible using established criteria, our conclusions might be limited due to the quality of published research and the large variability across study characteristics and methodologies.

CONCLUSION

In conclusion, available risk scores to predict SSI or PJI have been developed using poor methodology and have several limitations. The majority of these risk scores have not been externally validated and are not ideal for use in clinical settings. The HPRO is the only risk prediction tool identified to show some promise for use in a clinical setting (based on its predictive performance and having some external validation); however, it needs further validation using new data and its clinical effectiveness should be evaluated using a RCT design. A potentially effective way of tackling the increasing incidence of SSIs is early and accurate identification of individuals at high risk using established risk prediction scores, an approach which has been very effective in the area of CVD prevention. Further research is urgently warranted within the field to develop and test appropriate outcome-specific risk prediction tools.