A prognostic model for predicting functional impairment in youth mental health services

Abstract

Background

Functional impairment is a major concern among those presenting to youth mental health services and can have a profound impact on long-term outcomes. Early recognition and prevention for those at risk of functional impairment is essential to guide effective youth mental health care. Yet, identifying those at risk is challenging and impacts the appropriate allocation of indicated prevention and early intervention strategies.

Methods

We developed a prognostic model to predict a young person’s social and occupational functional impairment trajectory over 3 months. The sample included 718 young people (12–25 years) engaged in youth mental health care. A Bayesian random effects model was designed using demographic and clinical factors and model performance was evaluated on held-out test data via 5-fold cross-validation.

Results

Eight factors were identified as the optimal set for prediction: employment, education, or training status; self-harm; psychotic-like experiences; physical health comorbidity; childhood-onset syndrome; illness type; clinical stage; and circadian disturbances. The model had an acceptable area under the curve (AUC) of 0.70 (95% CI, 0.56–0.81) overall, indicating its utility for predicting functional impairment over 3 months. For those with good baseline functioning, it showed excellent performance (AUC = 0.80, 0.67–0.79) for identifying individuals at risk of deterioration.

Conclusions

We developed and validated a prognostic model for youth mental health services to predict functional impairment trajectories over a 3-month period. This model serves as a foundation for further tool development and demonstrates its potential to guide indicated prevention and early intervention for enhancing functional outcomes or preventing functional decline.

Introduction

Functional impairment is a major concern among those presenting to youth mental health services. It typically involves disruptions to education or work, social interactions, and daily functioning, which often indicate or exacerbate serious underlying mental health issues [1]. According to the 2019 Global Burden of Disease study, around one-fifth of all disease-related disability (from all causes) is attributable to mental disorders in children, adolescents, and young adults aged 5–24 [2]. A further concern is that functional impairment can have a profound impact on longer-term outcomes in adulthood due to the foundational importance of adolescence and young adulthood for social, emotional, and physical development [3]. Thus, early recognition and prevention for those at risk of functional impairment in youth mental health settings is essential to guide effective care that aims to improve immediate and long-term outcomes.

Current primary youth mental health care often falls short of effectively addressing functional impairment. Available data from these settings show that many young people engaged in care experience poor functional outcomes [4–6], which can typically be characterized as persistently low or volatile [7], and in some cases, only 37.8% experience clinically significant improvement [5]. This is critical given the bidirectional relationship between functioning and clinical symptoms. Indeed, the impact of functional impairment extends beyond the functional domain as it also predicts illness progression [8, 9] and can have causal effects on suicidal thoughts and behaviors, self-harm, and psychosis-like experiences [10]. Conversely, clinical symptoms such as depression and anxiety can directly impact young people’s participation in school and work. Effective functional recovery in youth requires interventions specifically designed to enhance engagement with education or employment, life skills, and social functioning [11–15]. Without direct intervention targeting these areas, many young people may continue to experience significant challenges in their daily lives, even if their mental health symptoms have been addressed.

An effective approach to allocate specific interventions for functional impairment involves the use of prognostic stratification. This method involves stratifying young people based on the risk of functional impairment to identify those who would benefit most from early and targeted interventions focused on functioning [16, 17]. This stratified approach ensures that resources and interventions are efficiently directed toward those who need them most, potentially preventing or mitigating the progression of functional impairment. This approach also allows for a more tailored, proactive approach in mental health care, focusing not only on symptom reduction but also on enhancing overall functioning and quality of life from an early stage [18].

Prognostic models are commonly used in other areas of medicine (e.g., breast cancer and cardiovascular disease), and there are some promising examples in mental health for the onset and course of major mood, anxiety, and psychotic disorders [19–30]. Though, few studies have focused on functional impairment as an outcome [31], despite its importance as a consequential outcome of interest for those with lived experience, its widespread impact on other outcomes, and the need for specific intervention [32]. This is particularly important in youth since current diagnostic approaches fail to capture the potential longer-term impact of illness even among those with subthreshold disorders [33, 34]. Unfortunately, the range of factors involved in functional impairment makes prediction and early intervention challenging which can lead to delayed access to effective interventions and highlights the need for prognostic models to identify those at risk earlier [4].

The goal of this study was to develop a prognostic model to predict future functional impairment among young people presenting for youth mental health care. Given the detrimental impact of functional impairment for those presenting to mental health services there is a need for tools that can be used to guide indicated prevention and early intervention efforts. Specifically, we sought to combine basic demographic and clinical information commonly available in mental health services to predict functional outcomes over the subsequent 3 months. This time period was chosen since most change occurs early in the course of care, and to align with our focus on prevention and early intervention by detecting functional change before impairment becomes entrenched.

Methods

The study was approved by the University of Sydney Human Research Ethics Committee (2008/5453, 2012/1626) and participants (and/or guardians) gave written informed consent for the use of routinely collected clinical data for research purposes. The study is reported according to the STROBE reporting guidelines.

Participants

Participants were drawn from a cohort of 6743 individuals aged 12–30 years who presented to the Brain and Mind Centre’s youth mental health clinics in Sydney, Australia, and were recruited to a research register between June 2008 and July 2018 [35]. These clinics include primary care services (i.e., headspace [36]) as well as more specialized mental health services. All participants received ongoing clinician-based case management involving treatment planning and coordination of psychological, social, and/or medical interventions for the duration of their care with the services. This may have involved contact with a psychiatrist, psychologist, occupational therapist, support worker, or hospitalization for those whose needs exceeded the capacity of the primary care services.

Eligibility criteria

By December 2019, phase two of data entry was completed, and so baseline data were available for 2901 participants. To be eligible for the current analysis, we selected individuals who had at least one follow-up visit to the clinic within the first 3 months after their initial visit (baseline). Application of these criteria reduced the sample to 718 individuals included for analyses.

Data collection

Research staff were trained through individual and group training sessions to extract key data from clinical and research files and code inputs according to a specifically designed clinical research proforma [35]. Clinical files included all available notes and records from standard clinical care (recorded by treating clinicians), and research files included various assessments as part of participation in substudies (which may include structured or unstructured clinical interviews and the use of symptom rating scales). Data were extracted from clinical files and code inputs according to the standardized proforma which records information at predetermined timepoints. The first available clinical assessment at the service was taken as the baseline timepoint for each participant and the date of this assessment was used to determine each of the follow-up timepoints. If there was no clinical information available for any given timepoint (i.e., the participant did not attend the service during that time) then that entry was left missing. All clinical notes from the preceding timepoints, up to and including the current timepoint were used to inform and complete the current proforma entry.

Assessments

The proforma recorded specific illness characteristics. More detailed descriptions about the proforma, including the interrater reliability, are reported in the supplement and cohort paper [35]. The measures used here include (Supplementary Materials): demographics, social and occupational functioning, using the Social and Occupational Functioning Assessment Scale (SOFAS), and “Not in Education, Employment or Training” (NEET) as a measure of participation with education or work.

Mental disorder diagnoses (DSM-5) were recorded for each participant and labeled as either primary, secondary, or tertiary based, however, these were collapsed for this study. At-risk mental states, including psychotic- and manic-like experiences and circadian disturbances, were also recorded.

A clinical stage was assigned to each young person alongside their diagnoses using the established criteria [37]. Young people at stage 1a (“help-seeking”) typically have fewer clinical symptoms and milder impairment than those at stage 1b (“attenuated syndrome”). Stage 2+ (stages 2, 3, and 4) includes young people with full-threshold disorders that are major, discrete, and persistent. One of three illness subtypes was also coded for each participant, neurodevelopmental-psychosis (psychotic symptoms, cognitive impairment, and/or childhood neurodevelopmental disorder), hyperarousal-anxious depression (heightened stress sensitivity and/or depressive features), or circadian-bipolar spectrum (disrupted sleep and circadian rhythms, symptoms of atypical or bipolar spectrum). “N/A” was recorded if the clinical researchers determined no clear illness subtype.

Other clinical information included self-harm, suicidal thoughts and behaviors, physical health comorbidities (e.g., diabetes and respiratory illnesses), personal mental illness history, and treatment utilization.

Statistical analyses

Analyses were conducted using Python 3.9.12. A Bayesian growth mixture model [38, 39] was designed to predict whether an individual’s SOFAS score would improve, deteriorate, or stay the same during the first 3 months of care, and estimate the amount of change in SOFAS score, if any, over that same period. For these purposes, we adopted a latent class linear mixed effects model where individuals were assigned to one of three fixed classes: (i) constant, (ii) improve, and (iii) decline. Individual factors were used as predictors of class membership, which is jointly inferred with the rest of the model parameters. The inference is performed following a Bayesian approach via Markov chain Monte Carlo, where we use a no-U-turn sampler (NUTS) to estimate continuous model parameters within a Gibbs sampling scheme to sample the discrete random variables (i.e., the latent class for each individual) [40]. To estimate our results, we ran the NUTS for 6000 iterations in 2 parallel chains. The first 4000 samples of each chain were discarded as burn-in, while the remaining 2000 samples of each chain were combined to form the model posterior samples, totaling 4000 posterior samples for each model. The resulting Gelman-Rubin R-hat statistic for each parameter’s posterior distribution was below 1.1 across all models, indicating convergence of the sampling algorithm. An overview of the model is presented in Figure 1 and further details can be found in the Supplementary Materials.

Figure 1. An overview of the predictive model. An individual’s clinical and demographic characteristics serve as inputs to the cluster assignment model, which predicts the probabilities of whether an individual’s score is going to remain the same, improve, or deteriorate. Combined with the cluster’s prediction, the individual’s initial score informs a trajectory prediction model which predicts an individual’s response over time. The magenta crosses indicate the actual scores given by clinicians. Note that the trajectory model is only informed of the individual’s initial score at baseline, and not the score at their second visit, which the model must predict.

A total of 31 factors were available for these analyses. These included basic demographic and clinical factors commonly collected in mental health services when someone presents for care. We considered three different combinations of these factors for the model: (i) a model with all available factors; (ii) a model with no factors; and (iii) a model with a subset of eight factors that were deemed to be the optimal set for prediction (decided by expert knowledge and confirmatory analyses). The eight factors included in the model were NEET status, self-harm, psychotic-like experiences, physical health comorbidity, childhood-onset syndrome, illness type, clinical stage, and sleep–wake and/or circadian disturbances. Factor selection was evaluated using the Watanabe-Akaike information criterion (WAIC), a Bayesian estimate of the predictive performance of a model on unseen future data via an approximation based on the model’s posterior distribution given all available data [41, 42]. In general, a lower WAIC value indicates a better out-of-sample predictive performance when comparing models.

We used the traditional interpretation of the c-statistic/area under the curve (AUC) to evaluate the performance of the model in two prediction tasks relevant to clinical practice. The first task consisted of predicting whether an individual’s SOFAS score would significantly drop over the course of 3 months, based solely on the information accessible at baseline. A predicted difference of 10 or more points in SOFAS was deemed significant [43]. To provide true labels for this assessment, the score at 3 months was computed based on the available data for an individual via linear extrapolation.

The second task consisted of predicting functional impairment at the next timepoint for individuals with initially good functioning (SOFAS above 70), again using only the information available at baseline. This task was designed to assist with indicated prevention and early intervention by identifying those individuals who are at risk of becoming functionally impaired.

To assess predictive performance on unseen data in both tasks, we used 5-fold cross-validation, where 80% of the data is used to estimate the model parameters posterior, and the remaining 20% is used as test data to assess the model’s predictions.

Results

Baseline characteristics for the 718 young people (mean age = 18.1 ± 3.3; 63% female) who met the criteria for this study are presented in Table 1.

Table 1. Baseline characteristics of the sample selected for the analysis (N = 718)

Each row presents the count of individuals presenting with a certain covariate in the dataset, except for the case of age, which shows the mean and standard deviation of the participants’ age. Abbreviations: NEET, not in education, employment or training; No., number of participants; SD, standard deviation.

Overall model performance

The model using all factors had the highest WAIC (WAIC = 1942; AUC = 0.65) and the model with no factors had the lowest AUC (WAIC = 1802; AUC = 0.50), indicating that each of these models was the worst performing on at least one of the evaluation metrics. By contrast, the model using the subset of eight factors performed best with the lowest WAIC (1777) and best AUC with “acceptable” predictive accuracy for predicting whether a person’s functioning would significantly drop over the next 3 months (AUC = 0.70, 95% Credible Interval [CI], 0.56–0.81; Figure 2A).

Figure 2. Model performance metrics and AUC. Panel A presents results for the overall model predicting whether an individual’s SOFAS score would significantly drop by 10 points over the course of 3 months. Panel B presents results for the model predicting functional impairment at the next consultation for individuals with initially good functioning (SOFAS above 70). Abbreviations: NPV, negative predictive value; PPV, positive predictive value.

Characteristics of change

The constant and decline clusters were comprised of individuals with similar initial SOFAS scores (constant: M = 64.1, SD = 9.0; decline: M = 64.7, SD = 7.9), though individuals in the decline cluster had a decrease of 8.8 SOFAS points on average (SD = 6.6) after 3 months. Individuals in the improve cluster started at an initially lower score (M = 58.6, SD = 8.3) and then progressed to an average increase of 7.8 SOFAS points (SD = 5.4) after 3 months. The time to a significant change in SOFAS score (10 points) was 5.0 months (95% CI, 1.7–11.0) for the improve cluster and 4.6 months (95% CI, 1.7–9.5) for the decline cluster.

The marginal probabilities for the whole sample indicate that in their first 3 months, most individuals had no change in score (58%), 16% deteriorated, and 26% improved. Being NEET shifted this marginal probability with an increase of 52% in the probability of improvement when present at baseline (Odds Ratio [OR], 1.52; 95% CI, 1.11–2.00). Conversely, a history of self-harm (OR, 1.49; 95% CI, 1.10–1.94) and having a physical health comorbidity (OR, 1.98; 95% CI, 1.43–2.69) led to a significant increase in the probability of deterioration (Supplementary Materials).

Typical cases for each cluster (i.e., individuals who maximize each cluster probability given their factors at baseline) are presented in Table 2. Typical cases across each cluster do not change very much between baseline and follow-up. The first noticeable difference between clusters is in the presence of medical and childhood history, with individuals in the decline cluster presenting with both at baseline. The typical case for each cluster also differed based on specific illness characteristics such as the type of mood disorder, presence of sleep–wake and/or circadian disturbances, clinical stage, and initial functioning.

Table 2. Baseline characteristics of typical individuals in each cluster

Predicting impairment among those with good functioning (SOFAS above 70)

The model demonstrated “excellent” accuracy for predicting functional impairment among those who present with good functioning (AUC = 0.80, 95% CI, 0.67–0.89; Figure 2B). The model has high sensitivity (0.82, 95% CI, 0.71–0.97), while maintaining high specificity (0.74, 95% CI, 0.68–0.78), indicating good discriminative power. Table 3 presents odds ratios for the posterior probabilities of drops below 70 in the follow-up visit as a function of the factors and the SOFAS score at baseline. NEET status did not emerge as a predictor, which likely reflects how many of those with a SOFAS score above 70 are engaged in employment, education, or training. Consistent with the overall model results, physical health comorbidity and self-harm were associated with the greatest increase in the probability of future functional impairment among those who are functioning well, however psychotic-like experiences and circadian disturbances were also identified as key predictors of future impairment (Table 3).

Table 3. Factors that influence a drop in SOFAS score to below 70 at the next visit to the clinics given different initial score levels

The table presents odds ratios for the probability of the next SOFAS score being below 70 computed with respect to a baseline individual with no mood disorders or any comorbidities based on the model’s posterior predictions. The 95% credible interval for the predictions is also shown within brackets. Bold text indicates significance.

Discussion

This study describes the development and preliminary validation of a brief prognostic model for functional outcomes among young people presenting for youth mental health care. Overall model performance was “acceptable” indicating its potential utility for predicting functional outcomes over 3 months, and model performance was “excellent” for identifying those at risk of deterioration from good functioning to impairment. This work has clear implications for the future development of prognostic models. These models can inform individual-level decision-making about the type and intensity of indicated prevention and early intervention required to improve functional outcomes or prevent functional decline in youth mental health.

The goal of indicated prevention in youth mental health services is to provide those at risk for poor outcomes with the appropriate supports [44]. The heterogeneity of presentations makes this type of prediction difficult. Here, we demonstrate how basic demographic and clinical information could be leveraged to identify those at risk and who could benefit from specific preventative approaches needed to prevent functional impairment (e.g., vocational support and school support).

Incorporating new predictive models into clinical services represents a transformative shift in youth mental health care. Early interventions guided by prognostic models hold promise for reducing the burden of mental health disorders in young people. By identifying those at risk before the onset of severe illness, preventive strategies can be deployed earlier, when people are more likely to be responsive to interventions [45]. This shift aligns with a broader trend toward proactive, individualized care that addresses mental health challenges early in their course rather than after they have progressed significantly. While further validation and testing are required before this model can be recommended for clinical practice [46], the present model demonstrates utility as a prognostic model to identify those most at risk of deteriorating mental health and function [47].

Critical factors associated with poor trajectories of functional impairment included a history of self-harm, physical health comorbidity, circadian rhythm disruptions, and psychotic-like experiences. This is consistent with previous evidence about the complicating role of complexity and comorbidity in diagnosis, treatment, and the course of illness [48]. While the relationship between these factors and impairment is likely bidirectional, their predictive utility among those functioning well points to some potential critical mechanisms.

Disrupted sleep and circadian rhythms can impair cognitive and mood regulation, affecting academic and social engagement [49–51], and self-harm may result in social withdrawal and academic struggles [52, 53]. Further, psychotic-like experiences, such as hallucinations or delusions, can significantly disrupt reality perception, leading to challenges in social interactions and daily activities [54, 55]. These factors collectively underscore the need for tailored, multifaceted treatment approaches to address these challenges and reiterate the importance of large-scale health service trials focused on developing service models that facilitate this level of personalized care [56]. For example, behavioral interventions to improve the alignment between someone’s biological clocks with external environmental cues can foster a more regulated sleep–wake cycle [57]. This, in turn, can lead to improved mood, higher energy levels, and better cognitive functioning.

Prediction in mental health is inherently complex due to the variability in individual trajectories [58] and multiple causes of poor outcomes. This makes the communication of uncertainty vital in decision-support models, and ensures clinicians make informed, nuanced choices. For some individuals, the model’s predictions were highly concentrated around a particular trajectory which reflects the predictive power of some factors at baseline. However, for most people, predicted trajectories had higher degrees of uncertainty (see Figure 3 for examples). Acknowledging this uncertainty in prognostic models is crucial so health professionals can weigh the potential outcomes and risks appropriately, fostering a more transparent and collaborative decision-making process. Further, the use of digital technologies for measurement-based care may also help to reduce uncertainty and improve informed decision-making with greater involvement from young people [59]. These technologies can be leveraged to deliver patient-reported outcome measures regularly to provide a more detailed understanding of individuals’ trajectories across a range of domains [60].

Figure 3. Model outputs for an individual who improved over the course of 3 months (Panel A), and for another individual who deteriorated over the course of 3 months (Panel B). The “cluster” graph shows the probability of each change cluster (“constant,” “up,” and “down”) for both individuals. The “predicted trajectories” graph shows the simulated trajectories based on the cluster model and the individuals initial score (~50 for the person in panel A and ~60 for the person in panel B). “Up” trajectories are shaded green, “constant” trajectories are shaded red, and “down” trajectories are shaded blue. The dotted black line and cross show the actual observed trajectory and SOFAS score for both individuals over the follow-up period.

We recognize that this study has limitations. The data may not be missing at random as individuals were not obliged to participate in follow-up, which could lead to biases in data available for these models as well as differences in characteristics of individuals who were retained versus those who dropped out. Similarly, the specific sample characteristics of those included in this study may limit the generalizability to other youth settings where the types of disorders and comorbidities differ. The demographic and clinical predictors used in this study are not exhaustive and rely on clinician input rather than standardized instruments, so there may be critical factors omitted and potential biases introduced by clinician reports. For example, relevant missing variables include factors such as neurocognition, social cognition, and level of educational attainment, among others (e.g., cultural status and socio-economic status). Importantly, however, the factors used here are basic clinical variables that should be easily generalized to other services, which means this work is more accessible to be validated in other samples and has the potential to be scaled more rapidly. Finally, the use of SOFAS as a measure of functional impairment, while common practice in many mental health settings, may be limited in its ability to fully capture and characterize the large and multifaceted construct of functioning. Future research should look to more detailed methods of characterizing functional impairment by using multiple perspectives or modalities, questionnaires, and repeat measurements.

The use of evidence-based clinical decision-support models holds promise to transform youth mental health care, assist in identifying those at heightened risk, and inform indicated prevention and early intervention. Our development of a brief, prognostic model for functional impairment is an example of this advance and should serve as a foundation for future model development. It offers a potential means to guide individual-level decision-support about the type and intensity of intervention required to improve functional outcomes. This type of prognostic stratification is integral to the evolution toward personalized psychiatry, enhancing the efficacy of health care resource allocation and ultimately improving real-world outcomes for young people.