Detailed clinical phenotyping and generalisability in prognostic models of functioning in at-risk populations

Summary

Personalised prediction of functional outcomes is a promising approach for targeted early intervention in psychiatry. However, generalisability and resource efficiency of such prognostic models represent challenges. In the PRONIA study (German Clinical Trials Register: DRKS00005042), we demonstrate excellent generalisability of prognostic models in individuals at clinical high-risk for psychosis or with recent-onset depression, and substantial contributions of detailed clinical phenotyping, particularly to the prediction of role functioning. These results indicate that it is possible that functioning prediction models based only on clinical data could be effectively applied in diverse healthcare settings, so that neuroimaging data may not be needed at early assessment stages.

Young adults in a clinical high-risk state for psychosis (CHR) or with recent-onset depression (ROD) represent risk populations for severe mental illness (psychosis or recurrent depression).¹ Besides some shared symptom phenomenology, these two groups show comparable substantial and persistent deficits in functional outcomes,² which account for much of the immense individual and socioeconomic burden of mental disorders. Therefore, personalised prediction of functional outcomes in at-risk populations is a major target for prevention in psychiatry. Aiming at this objective, we recently reported machine learning models with a balanced accuracy (BAC) of up to 83% for the prediction of social and role functioning,³ i.e. the extent to which an individual is able to deal with social interactions and occupational and other demands of daily life respectively. However, as these models combined only data of previous functioning with structural neuroimaging data, an ongoing discussion was stimulated about selection of data included as predictors in such models.⁴ In particular, conclusions for clinical practice could be biased such that costly diagnostics would be recommended without testing whether more cost-efficient clinical data have a similar predictive potential.⁵ The optimal strategy to develop models that are accurate and generalisable but also efficient in a clinical context remains unclear. Detailed phenotyping by clinical data alone might have essential advantages over neuroimaging data owing to cost-efficiency and the potential to inform clinical interventions.⁵ Still, adding more and more data to prediction models might reduce generalisability to new patients or settings.⁶ As external performance of psychiatric prediction models is often low and highly heterogeneous,⁷ generalisability is crucial in the translation to clinical practice.

To determine the role of predictor parsimony and data type in prediction of psychosocial functioning, we conducted the current study, building on and reviewing our previous report of multi-modal models for prediction of functioning.³ We specifically tested (a) whether detailed clinical phenotyping improves prediction performance compared with parsimonious models based on previous functioning alone and (b) the so far unknown generalisability of prediction models for functional outcome to new patients from different healthcare settings and countries.

Method

For comparability, we used participants (CHR group: n = 114; ROD group: n = 106) from the observational multicentre Personalised Prognostic Tools for Early Psychosis Management (PRONIA) study (German Clinical Trials Register identifier DRKS00005042; for detailed sample description see supplementary Table 1, available at https://doi.org/10.1192/bjp.2021.141) and an analogous analysis approach following our study published in 2018.³ We trained different machine learning models using baseline variables for the prediction of social and role functioning after 1 year as assessed by the Global Functioning Scales: Social and Role⁸ separately for CHR and ROD individuals.

The first parsimonious model (A) is a replication of a basic model based only on previous functioning measures we reported earlier.³ A second group of models (B) includes n = 176 clinical variables comprising additional aspects of previous functioning and detailed characterisation of disease severity by core psychopathology of the ROD and CHR participants (supplementary Table 2). To identify the most predictive and eliminate non-informative variables, we implemented feature selection based on greedy feature elimination (B₁) and greedy feature elimination combined with a principal component analysis (PCA) (B₂), consistent with our previous analysis.³ In addition, we implemented two alternative modelling strategies based on an L₁-regularised support-vector machine (SVM) algorithm (B₃) and based on sparse PCA (B₄) that might be more appropriate for clinical data, yielding an informative, parsimonious set of the most predictive features. A third group of models (C₁₋₄) was based on a stacked ensemble model by combining model A with B_1–4 respectively. For detailed description of machine learning models, see the methods section in the supplementary material. We assessed the geographical generalisability of the models to patients in the different sites of the PRONIA study using nested leave-site-out cross-validation (LSO-CV). To test the generalisability to new patients, we applied all models to truly unseen data of additional patients (CHR: n = 97; ROD: n = 61) from the PRONIA sample of all original study sites plus three additional study sites that were not part of the training sample (supplementary Table 1). Differences in model performances were determined by Quade's test at the omnibus level, followed by post hoc comparisons with model A.

Ethics approval and consent

The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008. The multicentre PRONIA study was approved by all local research ethics committees. Written informed consent was obtained from all patients. For minors (defined as participants younger than 18 years), guardians also provided written informed assent.

Results

All models showed robust generalisability, i.e. similar prediction accuracy in the training and independent test sample (Table 1). The different dimensionality reduction methods (B₁₋₄) performed equally well across different data domains and risk populations. For prediction of role functioning, clinical phenotyping (B₁₋₄ and C₁₋₄) yielded significant, clinically relevant improvements (gain in BAC up to 10.9% for ROD participants in the training set and up to 10.0% for ROD participants and 8.3% for CHR participants in the independent test data) compared with models based on functioning measures alone (A). For social functioning, detailed clinical data did not improve prediction in CHR participants and revealed only slight, non-significant improvement for ROD participants (gain in BAC of up to 5.8% in the training set).

Table 1 Classification performances using leave-site-out cross-validation and validation in an independent sample of machine-learning predictors of global functioning social scale or global functioning role scale outcomes in individuals at clinical high-risk of psychosis and individuals with recent-onset depression

LSO-CV, leave-site-out cross-validation; sens., sensitivity; spec., specificity; BAC, balanced accuracy; PPV, positive predictive value; NPV, negative predictive value; CHR, clinical high-risk of psychosis; PCA, principal component analysis; L₁, L₁ regularisation; ROD, recent-onset depression.

a. Quade's test for statistical comparison of all models at omnibus level and post hoc comparisons with model A based on folds and repetitions of the outer cycle in the cross-validation process (reported classification performance measures are not weighted by varying size of folds).

Discussion

The present work demonstrates that detailed clinical phenotyping is valuable for prediction of functional outcomes, in particular for role functioning, which might be less biologically determined, influenced by a complex set of factors and more closely connected to the disease trajectory. Prognostication of social functioning, in contrast, did not substantially benefit from additional clinical data beyond previous functioning, and was generally more accurate in CHR participants than in ROD participants. These results are in line with previous findings showing that social impairment was more constant than role functioning across time.⁸ Furthermore, in CHR individuals deficits in social cognition resulting in social impairment are more persistent,² whereas social cognition in people with depression tends to be affected by symptom severity.⁹

Independent of the quantity of included clinical predictors, all models proved robust when applied to new patients. This underlines the reliability and validity of established clinical assessments included into the multi-predictor models for clinical phenotyping. Furthermore, different feature reduction strategies seemed to be equally effective, emphasising that results are primarily determined by included information about symptoms and aspects of functioning, and not by a specific analysis approach.

Compared descriptively with our previously reported models complemented by neuroimaging data³ (average difference in BAC Δ = 0.44; supplementary Table 4), the current results lead to a re-evaluation and suggest no absolute need for an extension of clinical phenotyping by another prognostic data modality such as costlier magnetic resonance imaging (MRI) in early stages of preventive care. Clinical data, such as different aspects of previous functioning and detailed characterisation of disease severity, are commonly collected in clinical contexts, facilitating translation into clinical practice. Moreover, prediction models based on clinical data yield informative insights for efficient targeted interventions to prevent disabilities in psychosocial functioning. However, costlier assessments, such as MRI, might be valuable at later stages in the context of sequential testing, allowing for more precise predictions in participants identified to be at risk for poor outcomes via prediction models based on clinical data.¹⁰

Given that we showed excellent generalisability of our personalised prognostic models based on easily accessible clinical data for functional outcome in CHR and ROD individuals across geographically and structurally diverse healthcare systems,⁴ our study represents an important step for clinical application of prognostic models and towards improvement of targeted early intervention and personalised psychiatric care.