Using machine learning to examine drivers of inappropriate outpatient antibiotic prescribing in acute respiratory illnesses

Abstract

Using a machine-learning model, we examined drivers of antibiotic prescribing for antibiotic-inappropriate acute respiratory illnesses in a large US claims data set. Antibiotics were prescribed in 11% of the 42 million visits in our sample. The model identified outpatient setting type, patient age mix, and state as top drivers of prescribing.

Despite recent decreases, outpatient antibiotics are frequently prescribed for acute respiratory illnesses (ARIs) for which they are not indicated: influenza, bronchitis, bronchiolitis, asthma, allergy, nonsuppurative otitis media, and viral upper respiratory infection. In 2014–2015, there were >14 million unnecessary antibiotic prescriptions annually for these conditions.¹ Clinician specialty,² outpatient setting,³ and region^4,5 have previously been associated with inappropriate antibiotic prescribing. However, these relationships are likely complex; machine-learning models may elucidate nuanced relationships and stewardship targets. Our primary objective was to examine relationships between clinician-related factors and antibiotic prescribing for antibiotic-inappropriate ARIs in a large convenience sample in the United States. Our secondary objective was to pilot test machine-learning methods for evaluating antibiotic prescribing.

Methods

Data source

We identified visits and antibiotic prescriptions for antibiotic-inappropriate ARIs using the IQVIA Medical Claims Data (Dx) data set and the IQVIA Longitudinal Prescription Data (LRx) data set from October 1, 2018, to September 30, 2019. The IQVIA LRx contains data from 92% of retail pharmacy transactions. The IQVIA Dx data set includes >1.3 billion pre-adjudicated outpatient medical claims per year. We linked visits with prescriptions within a 3-day postvisit window using deidentified patient and clinician codes.

We defined visits for antibiotic-inappropriate ARIs as those with International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) diagnosis codes for asthma, allergy, bronchitis, bronchiolitis, influenza, viral upper respiratory infection, and non-suppurative otitis media without diagnoses for which antibiotics are or may be indicated (Supplementary Table 1), following a previously established categorization.⁶ We included only visits to primary care specialties to exclude complex cases requiring specialty care. We included nurse practitioners (NPs) and physician assistants (PAs); in these data, NPs and PAs are not categorized by specialty. We excluded clinicians with <10 captured visits for stability. We categorized clinician caseload by patient sex (>50% male, >50% female, or balanced) and age group: >50% children 0–19 years, >50% adults 20–64 years, >50% adults ≥65 years, or balanced. We included all systemic antibiotics except urinary anti-infectives.

Machine-learning model development and analysis

We calculated the Prescriber Inappropriate Antibiotic Prescription Index (PIAPI) as the proportion of a clinician’s visits for antibiotic-inappropriate ARIs with associated antibiotic prescriptions overall and by stratum. We used a predictive machine-learning model to identify drivers of PIAPI from a set of input features: clinician age and sex, training, state, outpatient setting, and caseload age, and gender mix.

The data set was partitioned into training (80%) and holdout (20%) sets. A gradient-boosting decision tree (GBDT) algorithm⁷ used the training set to train a machine-learning model to predict PIAPI using the input features through a generalized linear model. The GBDT algorithm constructs an ensemble of decision trees that are combined to obtain a single predictive model. Our machine-learning model consisted of 300 decision trees. The first decision tree was built similarly to the conventional decision tree approach; input features were split to minimize the error between the prediction and the dependent variable (ie, PIAPI). The GBDT algorithm sequentially built additional trees, reducing the error obtained from the preceding trees in each subsequent tree.

GBDT algorithms typically contain many more model parameters than a simple decision tree (eg, the number of trees in the ensemble is a model parameter). Our model parameters were automatically configured using a Bayesian optimizer.⁸ We applied SHAP,⁹ a state-of-the-art interpretability framework, to the trained model to specify feature impact on predicted clinician PIAPI. The aggregate impact across all clinicians was used to extract prescribing drivers. Final machine-learning model performance was reported on the holdout set. The importance determined by our machine-learning model was validated on the holdout data; the relative importance of the key drivers identified by the model on the training data was replicated on the holdout data.

This study was approved as non–human-subjects research by the Centers for Disease Control and Prevention National Center for Emerging and Zoonotic Infectious Diseases and did not require institutional review board review.

Results

In our study population, there were 41.97 million visits for antibiotic-inappropriate ARIs from October 1, 2018, to September 30, 2019 (Table 1). The average PIAPI was 11%, meaning that clinicians prescribed antibiotics in 11% of visits (N = 4.41 million). PIAPI values ranged from 0% to 99% and were highly skewed: 45% of clinicians prescribed antibiotics in ≤4% of antibiotic-inappropriate ARI visits while only 2% of clinicians prescribed antibiotics in ≥50% of visits (Supplementary Fig. 1).

Table 1. Clinicians, Visits, and Average PIAPI by Clinician Characteristics

Note. PIAPI, Prescriber Inappropriate Antibiotic Prescription Index.

^a Volumes and percents may not sum to total due to missing values and rounding.

^b Other includes assisted living facility, birthing center, community mental health center, comprehensive rehabilitation facility, facility, custodial care facility, group home, home, homeless shelter, hospice, Indian Health Service free-standing facility, Indian Health Service provider-based facility, inpatient psychiatric facility, intermediate care facility or individuals with intellectual disabilities, military treatment facility, mobile unit, nonresidential substance abuse treatment facility, other place of service, place of employment work site, prison or correctional facility, psychiatric facility–partial hospitalization, psychiatric residential treatment center, residential substance abuse treatment facility, school, temporary lodging, tribal 8 free-standing facility, tribal 638 provider-based facility, dialysis facility.

The machine-learning model identified outpatient setting, patient age mix, and state as the strongest predictors of PIAPI (Table 1). Among settings, average PIAPI ranged from 5% in outpatient hospital clinics to 21% in urgent care facilities. Clinicians who saw predominantly children had a lower average PIAPI (7%) than those who saw adults or balanced age mixes (11%–13%). We detected wide variation by state (Table 1), with the highest average PIAPIs in Mississippi (17%) and Alabama (18%).

The machine-learning model allowed us to examine nuanced relationships between the factors included in this study. Among urgent care clinicians who saw predominantly children, average PIAPI was 12%. In contrast, the average PIAPI was >20% for those who saw mostly adults or had balanced patient age mixes. Among urgent-care clinicians, the average PIAPI was >14% across all states, with the highest value in Alabama (36%). In outpatient hospital clinics, 68% of clinicians had PIAPI values ≤4%. The average PIAPI among clinicians in this setting was <9% across states, except in South Dakota and Maine, where the average PIAPI was 12%.

The machine-learning model identified complex relationships between state, outpatient setting, and provider. In Alabama, aside from state, the most highly ranked driver was outpatient setting. In Alabama, the highest average PIAPI was in urgent care and physician offices, and in these 2 settings PAs and NPs had the highest average PIAPI among all specialties, 24% and 23%, respectively. In California, the average PIAPI was 19% among urgent care clinicians, while all other PIAPI values were relatively low.

Although specialty was not identified as a major driver of PIAPI overall, we observed wide variation in average PIAPI by specialty, with the highest average PIAPI among NPs and PAs and the lowest among pediatricians (Table 1). Notably, in urgent care and retail health settings, where high proportions of clinicians were NPs or PAs, the machine-learning model identified specialty as a major predictor of PIAPI (after patient age mix and state).

Discussion

On average, clinicians prescribed antibiotics in 11% of antibiotic-inappropriate ARI visits in our sample. However, PIAPI distribution was highly skewed, suggesting prescribing practice heterogeneity. Using a supervised machine-learning model, we found that urgent care, states in the South region, and older patient age mix were the strongest predictors of inappropriate antibiotic prescribing.

Our major findings align with previous studies. We found that the urgent care setting was the strongest driver of inappropriate prescribing; average urgent care PIAPI was almost double overall PIAPI, consistent with previous findings.^3,10 Patient age mix also strongly predicted PIAPI, mirroring previous findings that overall and unnecessary outpatient antibiotic prescribing is lower among children.^1,5,6 We also detected high variation in average PIAPI by state, with highest average PIAPIs in the South region and lowest in the West region, similar to previously observed trends.^4–6 Despite wide variation in PIAPI, the machine-learning model did not identify specialty as a major driver of inappropriate antibiotic prescribing. This may be partially related to relationships between specialty and setting type; clinician specialty was a major driver of PIAPI in urgent care and retail health settings. Another potential explanation may be the correlation between pediatrics specialty and patients aged 0–19 years. Because the machine-learning model assigned high importance to the patient age group, it may have deflated the importance assigned to clinician specialty; it was designed to not overstate the combined impact of correlated factors.

In addition to accounting for interaction, our machine-learning approach allows us to evaluate feature importance across a range of dynamically selected subcohorts. This contrasts with traditional approaches, which assign static importance to modeling features across subcohorts. Machine-learning approaches may offer opportunities to identify and target antibiotic stewardship interventions at both macro and micro levels. For example, in Alabama, where one of the most highly ranked drivers was outpatient setting and where we observed high average PIAPI values among NPs and PAs, stewardship efforts could target mid-level providers in physician office and urgent care settings. In contrast, in California, stewardship efforts could focus on all providers in urgent care settings, where the highest PIAPI values were observed.

Our study has several limitations. First, this study was based on a convenience sample; findings may not be generalizable. Second, we relied on diagnostic codes from claims and could not evaluate their validity. Third, NPs and PAs practicing in non–primary-care specialties may have been included as NP and PA specialty was not available. Finally, we included all antibiotics except urinary anti-infectives, not just agents commonly used for respiratory infections, in this analysis, which may have resulted in overestimates of prescribing.

In conclusion, in this study of 42 million antibiotic-inappropriate ARI visits, our machine-learning model identified outpatient setting, state, and patient age-mix as top predictors of prescribing for antibiotic-inappropriate ARIs. However, feature importance varied by strata. This project demonstrated that machine-learning may be valuable in targeting antibiotic stewardship interventions.