Hostname: page-component-78c5997874-fbnjt Total loading time: 0 Render date: 2024-11-11T06:56:15.911Z Has data issue: false hasContentIssue false
  • English
  • Français

Prediction of transition to psychosis in patients with a clinical high risk for psychosis: a systematic review of methodology and reporting

Published online by Cambridge University Press:  16 January 2017

E. Studerus ,
A. Ramyead  and
A. Riecher-Rössler
E. Studerus
Affiliation:
University of Basel Psychiatric Hospital, Center for Gender Research and Early Detection, Basel, Switzerland
A. Ramyead
Affiliation:
Department of Psychiatry, Weill Institute for Neurosciences, University of California (UCSF), San Francisco, CA, USA
A. Riecher-Rössler*
Affiliation:
University of Basel Psychiatric Hospital, Center for Gender Research and Early Detection, Basel, Switzerland
*
*Address for correspondence: A. Riecher-Rössler, M.D., University of Basel Psychiatric Clinics, Center for Gender Research and Early Detection, Kornhausgasse 7, CH-4051 Basel, Switzerland. (Email: anita.riecher@upkbs.ch)
Get access Rights & Permissions [Opens in a new window]

Abstract

Background

To enhance indicated prevention in patients with a clinical high risk (CHR) for psychosis, recent research efforts have been increasingly directed towards estimating the risk of developing psychosis on an individual level using multivariable clinical prediction models. The aim of this study was to systematically review the methodological quality and reporting of studies developing or validating such models.

Method

A systematic literature search was carried out (up to 14 March 2016) to find all studies that developed or validated a clinical prediction model predicting the transition to psychosis in CHR patients. Data were extracted using a comprehensive item list which was based on current methodological recommendations.

Results

A total of 91 studies met the inclusion criteria. None of the retrieved studies performed a true external validation of an existing model. Only three studies (3.5%) had an event per variable ratio of at least 10, which is the recommended minimum to avoid overfitting. Internal validation was performed in only 14 studies (15%) and seven of these used biased internal validation strategies. Other frequently observed modeling approaches not recommended by methodologists included univariable screening of candidate predictors, stepwise variable selection, categorization of continuous variables, and poor handling and reporting of missing data.

Conclusions

Our systematic review revealed that poor methods and reporting are widespread in prediction of psychosis research. Since most studies relied on small sample sizes, did not perform internal or external cross-validation, and used poor model development strategies, most published models are probably overfitted and their reported predictive accuracy is likely to be overoptimistic.

Keywords

Clinical high riskpredictionprognostic modelspsychosisschizophrenia
Type
Review Article
Information
Psychological Medicine , Volume 47 , Issue 7 , May 2017 , pp. 1163 - 1178
Copyright
Copyright © Cambridge University Press 2017 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Altman, DG, McShane, LM, Sauerbrei, W, Taube, SE (2012). Reporting Recommendations for Tumor Marker Prognostic Studies (REMARK): explanation and elaboration. PLoS Medicine 9, e1001216.Google Scholar
Altman, DG, Vergouwe, Y, Royston, P, Moons, KG (2009). Prognosis and prognostic research: validating a prognostic model. BMJ 338, b605.Google Scholar
Amminger, GP, Leicester, S, Yung, AR, Phillips, LJ, Berger, GE, Francey, SM, Yuen, HP, McGorry, PD (2006). Early-onset of symptoms predicts conversion to non-affective psychosis in ultra-high risk individuals. Schizophrenia Research 84, 67–76.Google Scholar
Austin, PC, Steyerberg, EW (2014). Events per variable (EPV) and the relative performance of different strategies for estimating the out-of-sample validity of logistic regression models. Statistical Methods in Medical Research. Published online 19 November 2014. doi:10.1177/0962280214558972.Google ScholarPubMed
Bang, M, Kim, KR, Song, YY, Baek, S, Lee, E, An, SK (2015). Neurocognitive impairments in individuals at ultra-high risk for psychosis: who will really convert? Australian and New Zealand Journal of Psychiatry 49, 462–470.Google Scholar
Bedi, G, Carrillo, F, Cecchi, GA, Slezak, DF, Sigman, M, Mota, NB, Ribeiro, S, Javitt, DC, Copelli, M, Corcoran, CM (2015). Automated analysis of free speech predicts psychosis onset in high-risk youths. NPJ Schizophrenia 1, 15030.Google Scholar
Bouwmeester, W, Zuithoff, NP, Mallett, S, Geerlings, MI, Vergouwe, Y, Steyerberg, EW, Altman, DG, Moons, KG (2012). Reporting and methods in clinical prediction research: a systematic review. PLoS Medicine 9, 1–12.CrossRefGoogle ScholarPubMed
Cannon, TD, Cadenhead, K, Cornblatt, B, Woods, SW, Addington, J, Walker, E, Seidman, LJ, Perkins, D, Tsuang, M, McGlashan, T, Heinssen, R (2008). Prediction of psychosis in youth at high clinical risk: a multisite longitudinal study in North America. Archives of General Psychiatry 65, 28–37.CrossRefGoogle ScholarPubMed
Chan, MK, Krebs, MO, Cox, D, Guest, PC, Yolken, RH, Rahmoune, H, Rothermundt, M, Steiner, J, Leweke, FM, van Beveren, NJ, Niebuhr, DW, Weber, NS, Cowan, DN, Suarez-Pinilla, P, Crespo-Facorro, B, Mam-Lam-Fook, C, Bourgin, J, Wenstrup, RJ, Kaldate, RR, Cooper, JD, Bahn, S (2015). Development of a blood-based molecular biomarker test for identification of schizophrenia before disease onset. Translational Psychiatry 5, e601.Google Scholar
Collins, GS, Mallett, S, Omar, O, Yu, LM (2011). Developing risk prediction models for type 2 diabetes: a systematic review of methodology and reporting. BMC Medicine 9, 103.Google Scholar
Collins, GS, Ogundimu, EO, Cook, JA, Manach, YL, Altman, DG (2016). Quantifying the impact of different approaches for handling continuous predictors on the performance of a prognostic model. Statistics in Medicine 35, 4124–4135.Google Scholar
Collins, GS, Omar, O, Shanyinde, M, Yu, LM (2013). A systematic review finds prediction models for chronic kidney disease were poorly reported and often developed using inappropriate methods. Journal of Clinical Epidemiology 66, 268–277.Google Scholar
Collins, GS, Reitsma, JB, Altman, DG, Moons, KGM (2015). Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis Or Diagnosis (TRIPOD): the TRIPOD statement. Journal of Clinical Epidemiology 68, 112–121.Google Scholar
Cornblatt, BA, Carrion, RE, Auther, A, McLaughlin, D, Olsen, RH, John, M, Correll, CU (2015). Psychosis prevention: a modified clinical high risk perspective from the Recognition and Prevention (RAP) Program. American Journal of Psychiatry 172, 986–994.Google Scholar
D'Amico, G, Malizia, G, D'Amico, M (2016). Prognosis research and risk of bias. Internal and Emergency Medicine 11, 251–260.Google Scholar
Demjaha, A, Valmaggia, L, Stahl, D, Byrne, M, McGuire, P (2012). Disorganization/cognitive and negative symptom dimensions in the at-risk mental state predict subsequent transition to psychosis. Schizophrenia Bulletin 38, 351–359.CrossRefGoogle ScholarPubMed
Derksen, S, Keselman, HJ (1992). Backward, forward and stepwise automated subset selection algorithms: frequency of obtaining authentic and noise variables. British Journal of Mathematical and Statistical Psychology 45, 265–282.Google Scholar
DeVylder, JE, Muchomba, FM, Gill, KE, Ben-David, S, Walder, DJ, Malaspina, D, Corcoran, CM (2014). Symptom trajectories and psychosis onset in a clinical high-risk cohort: the relevance of subthreshold thought disorder. Schizophrenia Research 159, 278–283.Google Scholar
Fusar-Poli, P, Bechdolf, A, Taylor, MJ, Bonoldi, I, Carpenter, WT, Yung, AR, McGuire, P (2013 a). At risk for schizophrenic or affective psychoses? A meta-analysis of DSM/ICD diagnostic outcomes in individuals at high clinical risk. Schizophrenia Bulletin 39, 923–932.Google Scholar
Fusar-Poli, P, Bonoldi, I, Yung, AR, Borgwardt, S, Kempton, MJ, Valmaggia, L, Barale, F, Caverzasi, E, McGuire, P (2012). Predicting psychosis: meta-analysis of transition outcomes in individuals at high clinical risk. Archives of General Psychiatry 69, 220–229.Google Scholar
Fusar-Poli, P, Borgwardt, S, Bechdolf, A, Addington, J, Riecher-Rössler, A, Schultze-Lutter, F, Keshavan, M, Wood, S, Ruhrmann, S, Seidman, LJ, Valmaggia, L, Cannon, T, Velthorst, E, De Haan, L, Cornblatt, B, Bonoldi, I, Birchwood, M, McGlashan, T, Carpenter, W, McGorry, P, Klosterkötter, J, McGuire, P, Yung, A (2013 b). The psychosis high-risk state: a comprehensive state-of-the-art review. JAMA Psychiatry 70, 107–120.Google Scholar
Fusar-Poli, P, Cappucciati, M, Borgwardt, S, Woods, SW, Addington, J, Nelson, B, Nieman, DH, Stahl, DR, Rutigliano, G, Riecher-Rössler, A, Simon, AE, Mizuno, M, Lee, TY, Kwon, JS, Lam, MM, Perez, J, Keri, S, Amminger, P, Metzler, S, Kawohl, W, Rössler, W, Lee, J, Labad, J, Ziermans, T, An, SK, Liu, CC, Woodberry, KA, Braham, A, Corcoran, C, McGorry, P, Yung, AR, McGuire, PK (2016). Heterogeneity of psychosis risk within individuals at clinical high risk: a meta-analytical stratification. JAMA Psychiatry 73, 113–120.CrossRefGoogle Scholar
Fusar-Poli, P, Schultze-Lutter, F (2016). Predicting the onset of psychosis in patients at clinical high risk: practical guide to probabilistic prognostic reasoning. Evidence-Based Mental Health 19, 10–15.Google Scholar
Gorelick, MH (2006). Bias arising from missing data in predictive models. Journal of Clinical Epidemiology 59, 1115–1123.Google Scholar
Harrell, FE (2001). Regression Modeling Strategies with Applications to Linear Models, Logistic Regression, and Survival Analysis. Springer: New York.CrossRefGoogle Scholar
Healey, KM, Penn, DL, Perkins, D, Woods, SW, Addington, J (2013). Theory of mind and social judgments in people at clinical high risk of psychosis. Schizophrenia Research 150, 498–504.CrossRefGoogle ScholarPubMed
Hidalgo, B, Goodman, M (2013). Multivariate or multivariable regression? American Journal of Public Health 103, 39–40.Google Scholar
Hollander, N, Sauerbrei, W, Schumacher, M (2004). Confidence intervals for the effect of a prognostic factor after selection of an ‘optimal’ cutpoint. Statistics in Medicine 23, 1701–1713.CrossRefGoogle ScholarPubMed
Huang, JT, Leweke, FM, Tsang, TM, Koethe, D, Kranaster, L, Gerth, CW, Gross, S, Schreiber, D, Ruhrmann, S, Schultze-Lutter, F, Klosterkötter, J, Holmes, E, Bahn, S (2007). CSF metabolic and proteomic profiles in patients prodromal for psychosis. PLoS ONE 2, e756.Google Scholar
Kempton, MJ, Bonoldi, I, Valmaggia, L, McGuire, P, Fusar-Poli, P (2015). Speed of psychosis progression in people at ultra-high clinical risk: a complementary meta-analysis. JAMA Psychiatry 72, 622–623.Google Scholar
Keshavan, MS, Nasrallah, HA, Tandon, R (2011). Schizophrenia, “Just the Facts” 6. Moving ahead with the schizophrenia concept: from the elephant to the mouse. Schizophrenia Research 127, 3–13.Google Scholar
Koutsouleris, N, Borgwardt, S, Meisenzahl, EM, Bottlender, R, Moller, HJ, Riecher-Rössler, A (2012 a). Disease prediction in the at-risk mental state for psychosis using neuroanatomical biomarkers: results from the FePsy study. Schizophrenia Bulletin 38, 1234–1246.Google Scholar
Koutsouleris, N, Davatzikos, C, Bottlender, R, Patschurek-Kliche, K, Scheuerecker, J, Decker, P, Gaser, C, Moller, HJ, Meisenzahl, EM (2012 b). Early recognition and disease prediction in the at-risk mental states for psychosis using neurocognitive pattern classification. Schizophrenia Bulletin 38, 1200–1215.CrossRefGoogle ScholarPubMed
Koutsouleris, N, Kambeitz, J (2016). Pattern recognition methods in the prediction of psychosis. In Early Detection and Intervention in Psychosis – State of the Art and Future Perspectives (ed. Riecher-Rössler, A and McGorry, PD), pp. 95–102. Karger: Basel.Google Scholar
Koutsouleris, N, Meisenzahl, EM, Davatzikos, C, Bottlender, R, Frodl, T, Scheuerecker, J, Schmitt, G, Zetzsche, T, Decker, P, Reiser, M, Moller, HJ, Gaser, C (2009). Use of neuroanatomical pattern classification to identify subjects in at-risk mental states of psychosis and predict disease transition. Archives of General Psychiatry 66, 700–712.Google Scholar
Koutsouleris, N, Riecher-Rössler, A, Meisenzahl, EM, Smieskova, R, Studerus, E, Kambeitz-Ilankovic, L, von Saldern, S, Cabral, C, Reiser, M, Falkai, P, Borgwardt, S (2015). Detecting the psychosis prodrome across high-risk populations using neuroanatomical biomarkers. Schizophrenia Bulletin 41, 471–482.CrossRefGoogle ScholarPubMed
Krstajic, D, Buturovic, LJ, Leahy, DE, Thomas, S (2014). Cross-validation pitfalls when selecting and assessing regression and classification models. Journal of Cheminformatics 6, 10.Google Scholar
Leeflang, MM, Moons, KG, Reitsma, JB, Zwinderman, AH (2008). Bias in sensitivity and specificity caused by data-driven selection of optimal cutoff values: mechanisms, magnitude, and solutions. Clinical Chemistry 54, 729–737.Google Scholar
Lencz, T, Smith, CW, McLaughlin, D, Auther, A, Nakayama, E, Hovey, L, Cornblatt, BA (2006). Generalized and specific neurocognitive deficits in prodromal schizophrenia. Biological Psychiatry 59, 863–871.Google Scholar
Mallett, S, Royston, P, Dutton, S, Waters, R, Altman, DG (2010). Reporting methods in studies developing prognostic models in cancer: a review. BMC Medicine 8, 20.Google Scholar
Mason, O, Startup, M, Halpin, S, Schall, U, Conrad, A, Carr, V (2004). Risk factors for transition to first episode psychosis among individuals with ‘at-risk mental states’. Schizophrenia Research 71, 227–237.Google Scholar
Michel, C, Ruhrmann, S, Schimmelmann, BG, Klosterkötter, J, Schultze-Lutter, F (2014). A stratified model for psychosis prediction in clinical practice. Schizophrenia Bulletin 40, 1533–1542.CrossRefGoogle ScholarPubMed
Mittal, VA, Walker, EF, Bearden, CE, Walder, D, Trottman, H, Daley, M, Simone, A, Cannon, TD (2010). Markers of basal ganglia dysfunction and conversion to psychosis: neurocognitive deficits and dyskinesias in the prodromal period. Biological Psychiatry 68, 93–99.Google Scholar
Moher, D, Liberati, A, Tetzlaff, J, Altman, DG, Group, P (2009). Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Medicine 6, e1000097.CrossRefGoogle ScholarPubMed
Moons, KG, Altman, DG, Reitsma, JB, Ioannidis, JP, Macaskill, P, Steyerberg, EW, Vickers, AJ, Ransohoff, DF, Collins, GS (2015). Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Annals of Internal Medicine 162, W1–W73.Google Scholar
Moons, KG, Altman, DG, Vergouwe, Y, Royston, P (2009 a). Prognosis and prognostic research: application and impact of prognostic models in clinical practice. BMJ 338, b606.Google Scholar
Moons, KG, de Groot, JA, Bouwmeester, W, Vergouwe, Y, Mallett, S, Altman, DG, Reitsma, JB, Collins, GS (2014). Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist. PLoS Medicine 11, e1001744.CrossRefGoogle ScholarPubMed
Moons, KG, Kengne, AP, Woodward, M, Royston, P, Vergouwe, Y, Altman, DG, Grobbee, DE (2012). Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker. Heart 98, 683–690.Google Scholar
Moons, KG, Royston, P, Vergouwe, Y, Grobbee, DE, Altman, DG (2009 b). Prognosis and prognostic research: what, why, and how? BMJ 338, b375.Google Scholar
Mushkudiani, NA, Hukkelhoven, CW, Hernandez, AV, Murray, GD, Choi, SC, Maas, AI, Steyerberg, EW (2008). A systematic review finds methodological improvements necessary for prognostic models in determining traumatic brain injury outcomes. Journal of Clinical Epidemiology 61, 331–343.Google Scholar
Nelson, B, Yuen, HP, Wood, SJ, Lin, A, Spiliotacopoulos, D, Bruxner, A, Broussard, C, Simmons, M, Foley, DL, Brewer, WJ, Francey, SM, Amminger, GP, Thompson, A, McGorry, PD, Yung, AR (2013). Long-term follow-up of a group at ultra high risk (“prodromal”) for psychosis: the PACE 400 study. JAMA Psychiatry 70, 793–802.Google Scholar
Nieman, DH, Ruhrmann, S, Dragt, S, Soen, F, van Tricht, MJ, Koelman, JH, Bour, LJ, Velthorst, E, Becker, HE, Weiser, M, Linszen, DH, de Haan, L (2014). Psychosis prediction: stratification of risk estimation with information-processing and premorbid functioning variables. Schizophrenia Bulletin 40, 1482–1490.Google Scholar
Nieman, DH, Velthorst, E, Becker, HE, de Haan, L, Dingemans, PM, Linszen, DH, Birchwood, M, Patterson, P, Salokangas, RK, Heinimaa, M, Heinz, A, Juckel, G, von Reventlow, HG, Morrison, A, Schultze-Lutter, F, Klosterkotter, J, Ruhrmann, S, group E (2013). The Strauss and Carpenter Prognostic Scale in subjects clinically at high risk of psychosis. Acta Psychiatrica Scandinavica 127, 53–61.Google Scholar
Núñez, E, Steyerberg, EW, Núñez, J (2011). [Regression modeling strategies]. Revista Española de Cardiología 64, 501–507.Google Scholar
O'Donoghue, B, Nelson, B, Yuen, HP, Lane, A, Wood, S, Thompson, A, Lin, A, McGorry, P, Yung, AR (2015). Social environmental risk factors for transition to psychosis in an ultra-high risk population. Schizophrenia Research 161, 150–155.Google Scholar
Perkins, DO, Jeffries, CD, Addington, J, Bearden, CE, Cadenhead, KS, Cannon, TD, Cornblatt, BA, Mathalon, DH, McGlashan, TH, Seidman, LJ, Tsuang, MT, Walker, EF, Woods, SW, Heinssen, R (2015 a). Towards a psychosis risk blood diagnostic for persons experiencing high-risk symptoms: preliminary results from the NAPLS project. Schizophrenia Bulletin 41, 419–428.Google Scholar
Perkins, DO, Jeffries, CD, Cornblatt, BA, Woods, SW, Addington, J, Bearden, CE, Cadenhead, KS, Cannon, TD, Heinssen, R, Mathalon, DH, Seidman, LJ, Tsuang, MT, Walker, EF, McGlashan, TH (2015 b). Severity of thought disorder predicts psychosis in persons at clinical high-risk. Schizophrenia Research 169, 169–177.Google Scholar
Perlis, RH (2013). A clinical risk stratification tool for predicting treatment resistance in major depressive disorder. Biological Psychiatry 74, 7–14.CrossRefGoogle ScholarPubMed
Pettersson-Yeo, W, Benetti, S, Marquand, AF, Dell'acqua, F, Williams, SC, Allen, P, Prata, D, McGuire, P, Mechelli, A (2013). Using genetic, cognitive and multi-modal neuroimaging data to identify ultra-high-risk and first-episode psychosis at the individual level. Psychological Medicine 43, 2547–2562.Google Scholar
Piskulic, D, Addington, J, Cadenhead, KS, Cannon, TD, Cornblatt, BA, Heinssen, R, Perkins, DO, Seidman, LJ, Tsuang, MT, Walker, EF, Woods, SW, McGlashan, TH (2012). Negative symptoms in individuals at clinical high risk of psychosis. Psychiatry Research 196, 220–224.Google Scholar
Raballo, A, Nelson, B, Thompson, A, Yung, A (2011). The comprehensive assessment of at-risk mental states: from mapping the onset to mapping the structure. Schizophrenia Research 127, 107–114.Google Scholar
Ramyead, A, Studerus, E, Kometer, M, Uttinger, M, Gschwandtner, U, Fuhr, P, Riecher-Rössler, A (2016). Prediction of psychosis using neural oscillations and machine learning in neuroleptic-naive at-risk patients. World Journal of Biological Psychiatry 17, 285–295.Google Scholar
Riecher-Rössler, A, Pflueger, MO, Aston, J, Borgwardt, SJ, Brewer, WJ, Gschwandtner, U, Stieglitz, RD (2009). Efficacy of using cognitive status in predicting psychosis: a 7-year follow-up. Biological Psychiatry 66, 1023–1030.Google Scholar
Royston, P, Altman, DG (2013). External validation of a Cox prognostic model: principles and methods. BMC Medical Research Methodology 13, 33.Google Scholar
Royston, P, Altman, DG, Sauerbrei, W (2006). Dichotomizing continuous predictors in multiple regression: a bad idea. Statistics in Medicine 25, 127–141.Google Scholar
Royston, P, Moons, KG, Altman, DG, Vergouwe, Y (2009). Prognosis and prognostic research: developing a prognostic model. BMJ 338, b604.Google Scholar
Ruhrmann, S, Klosterkötter, J, Bodatsch, M, Nikolaides, A, Julkowski, D, Hilboll, D, Schultz-Lutter, F (2012). Chances and risks of predicting psychosis. European Archives of Psychiatry and Clinical Neuroscience 262 (Suppl. 2), S85–S90.Google Scholar
Ruhrmann, S, Schultze-Lutter, F, Salokangas, RK, Heinimaa, M, Linszen, D, Dingemans, P, Birchwood, M, Patterson, P, Juckel, G, Heinz, A, Morrison, A, Lewis, S, von Reventlow, HG, Klosterkötter, J (2010). Prediction of psychosis in adolescents and young adults at high risk: results from the prospective European Prediction of Psychosis Study. Archives of General Psychiatry 67, 241–251.Google Scholar
Rüsch, N, Heekeren, K, Theodoridou, A, Muller, M, Corrigan, PW, Mayer, B, Metzler, S, Dvorsky, D, Walitza, S, Rossler, W (2015). Stigma as a stressor and transition to schizophrenia after 1 year among young people at risk of psychosis. Schizophrenia Research 166, 43–48.Google Scholar
Schultze-Lutter, F, Klosterkötter, J, Michel, C, Winkler, K, Ruhrmann, S (2012). Personality disorders and accentuations in at-risk persons with and without conversion to first-episode psychosis. Early Intervention in Psychiatry 6, 389–398.Google Scholar
Schultze-Lutter, F, Klosterkötter, J, Picker, H, Steinmeyer, E-M, Ruhrmann, S (2007). Predicting first-episode psychosis by basic symptom criteria. Clinical Neuropsychiatry 4, 11–22.Google Scholar
Schultze-Lutter, F, Michel, C, Schmidt, SJ, Schimmelmann, BG, Maric, NP, Salokangas, RK, Riecher-Rössler, A, van der Gaag, M, Nordentoft, M, Raballo, A, Meneghelli, A, Marshall, M, Morrison, A, Ruhrmann, S, Klosterkötter, J (2015). EPA guidance on the early detection of clinical high risk states of psychoses. European Psychiatry 30, 405–416.Google Scholar
Seel, RT, Steyerberg, EW, Malec, JF, Sherer, M, Macciocchi, SN (2012). Developing and evaluating prediction models in rehabilitation populations. Archives of Physical Medicine and Rehabilitation 93, S138–S153.Google Scholar
Seidman, LJ, Giuliano, AJ, Meyer, EC, Addington, J, Cadenhead, KS, Cannon, TD, McGlashan, TH, Perkins, DO, Tsuang, MT, Walker, EF, Woods, SW, Bearden, CE, Christensen, BK, Hawkins, K, Heaton, R, Keefe, RS, Heinssen, R, Cornblatt, BA; North American Prodrome Longitudinal Study (NAPLS) Group (2010). Neuropsychology of the prodrome to psychosis in the NAPLS consortium: relationship to family history and conversion to psychosis. Archives of General Psychiatry 67, 578–588.CrossRefGoogle ScholarPubMed
Simon, AE, Borgwardt, S, Riecher-Rössler, A, Velthorst, E, de Haan, L, Fusar-Poli, P (2013). Moving beyond transition outcomes: meta-analysis of remission rates in individuals at high clinical risk for psychosis. Psychiatry Research 209, 266–272.Google Scholar
Sterne, JA, White, IR, Carlin, JB, Spratt, M, Royston, P, Kenward, MG, Wood, AM, Carpenter, JR (2009). Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ 338, b2393.Google Scholar
Steyerberg, EW (2009). Clinical Prediction Models: a Practical Approach to Development, Validation, and Updating. Springer: New York.Google Scholar
Steyerberg, EW, Eijkemans, MJ, Habbema, JD (1999). Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis. Journal of Clinical Epidemiology 52, 935–942.Google Scholar
Steyerberg, EW, Eijkemans, MJ, Harrell, FE Jr., Habbema, JD (2001). Prognostic modeling with logistic regression analysis: in search of a sensible strategy in small data sets. Medical Decision Making 21, 45–56.Google Scholar
Steyerberg, EW, van der Ploeg, T, Van Calster, B (2014). Risk prediction with machine learning and regression methods. Biometrical Journal 56, 601–606.Google Scholar
Steyerberg, EW, Vergouwe, Y (2014). Towards better clinical prediction models: seven steps for development and an ABCD for validation. European Heart Journal 35, 1925–1931.Google Scholar
Steyerberg, EW, Vickers, AJ, Cook, NR, Gerds, T, Gonen, M, Obuchowski, N, Pencina, MJ, Kattan, MW (2010). Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 21, 128–138.Google Scholar
Stowkowy, J, Liu, L, Cadenhead, KS, Cannon, TD, Cornblatt, BA, McGlashan, TH, Perkins, DO, Seidman, LJ, Tsuang, MT, Walker, EF, Woods, SW, Bearden, CE, Mathalon, DH, Addington, J (2016). Early traumatic experiences, perceived discrimination and conversion to psychosis in those at clinical high risk for psychosis. Social Psychiatry and Psychiatric Epidemiology 51, 497–503.Google Scholar
Strobl, EV, Eack, SM, Swaminathan, V, Visweswaran, S (2012). Predicting the risk of psychosis onset: advances and prospects. Early Intervention in Psychiatry 6, 368–379.Google Scholar
Sun, GW, Shook, TL, Kay, GL (1996). Inappropriate use of bivariable analysis to screen risk factors for use in multivariable analysis. Journal of Clinical Epidemiology 49, 907–916.Google Scholar
Thompson, A, Nelson, B, Yung, A (2011). Predictive validity of clinical variables in the “at risk” for psychosis population: international comparison with results from the North American Prodrome Longitudinal Study. Schizophrenia Research 126, 51–57.Google Scholar
Tibshirani, R (1997). The lasso method for variable selection in the Cox model. Statistics in Medicine 16, 385–395.Google Scholar
van der Net, JB, Janssens, ACJW, Eijkemans, MJC, Kastelein, JJP, Sijbrands, EJG, Steyerberg, EW (2008). Cox proportional hazards models have more statistical power than logistic regression models in cross-sectional genetic association studies. European Journal of Human Genetics 16, 1111–1116.Google Scholar
van der Ploeg, T, Austin, PC, Steyerberg, EW (2014). Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints. BMC Medical Research Methodology 14, 137.Google Scholar
van der Ploeg, T, Nieboer, D, Steyerberg, EW (2016). Modern modeling techniques had limited external validity in predicting mortality from traumatic brain injury. Journal of Clinical Epidemiology 78, 83–89.Google Scholar
van Oort, L, van den Berg, T, Koes, BW, de Vet, RH, Anema, HJ, Heymans, MW, Verhagen, AP (2012). Preliminary state of development of prediction models for primary care physical therapy: a systematic review. Journal of Clinical Epidemiology 65, 1257–1266.Google Scholar
Velthorst, E, Derks, EM, Schothorst, P, Becker, H, Durston, S, Ziermans, T, Nieman, DH, de Haan, L (2013 a). Quantitative and qualitative symptomatic differences in individuals at ultra-high risk for psychosis and healthy controls. Psychiatry Research 210, 432–437.Google Scholar
Velthorst, E, Nelson, B, Wiltink, S, de Haan, L, Wood, SJ, Lin, A, Yung, AR (2013 b). Transition to first episode psychosis in ultra high risk populations: does baseline functioning hold the key? Schizophrenia Research 143, 132–137.CrossRefGoogle ScholarPubMed
Walder, DJ, Holtzman, CW, Addington, J, Cadenhead, K, Tsuang, M, Cornblatt, B, Cannon, TD, McGlashan, TH, Woods, SW, Perkins, DO, Seidman, LJ, Heinssen, R, Walker, EF (2013). Sexual dimorphisms and prediction of conversion in the NAPLS psychosis prodrome. Schizophrenia Research 144, 43–50.Google Scholar
Wessler, BS, Lai Yh, L, Kramer, W, Cangelosi, M, Raman, G, Lutz, JS, Kent, DM (2015). Clinical prediction models for cardiovascular disease: tufts predictive analytics and comparative effectiveness clinical prediction model database. Circulation. Cardiovascular Quality and Outcomes 8, 368–375.CrossRefGoogle ScholarPubMed
Wynants, L, Collins, GS, Van Calster, B (2016). Key steps and common pitfalls in developing and validating risk models. BJOG: An International Journal of Obstetrics and Gynaecology. Published online 30 June 2016. doi:10.1111/1471-0528.14170.Google ScholarPubMed
Xu, L, Zhang, T, Zheng, L, Li, H, Tang, Y, Luo, X, Sheng, J, Wang, J (2016). Psychometric Properties of Prodromal Questionnaire-Brief Version among Chinese help-seeking individuals. PLOS ONE 11, e0148935.Google Scholar
Yung, AR, Phillips, LJ, McGorry, PD, McFarlane, CA, Francey, S, Harrigan, S, Patton, GC, Jackson, HJ (1998). Prediction of psychosis. A step towards indicated prevention of schizophrenia. British Journal of Psychiatry. Supplement 172, 14–20.Google Scholar
Yung, AR, Phillips, LJ, Yuen, HP, Francey, SM, McFarlane, CA, Hallgren, M, McGorry, PD (2003). Psychosis prediction: 12-month follow up of a high-risk (“prodromal”) group. Schizophrenia Research 60, 21–32.Google Scholar
Yung, AR, Phillips, LJ, Yuen, HP, McGorry, PD (2004). Risk factors for psychosis in an ultra high-risk group: psychopathology and clinical features. Schizophrenia Research 67, 131–142.CrossRefGoogle Scholar
Ziermans, T, de Wit, S, Schothorst, P, Sprong, M, van Engeland, H, Kahn, R, Durston, S (2014). Neurocognitive and clinical predictors of long-term outcome in adolescents at ultra-high risk for psychosis: a 6-year follow-up. PLOS ONE 9, e93994.Google Scholar
Supplementary material: File

Studerus supplementary material

Table S2

Download Studerus supplementary material(File)
File 35.8 KB
Supplementary material: File

Studerus supplementary material

Table S1

Download Studerus supplementary material(File)
File 373.8 KB