Hostname: page-component-78c5997874-s2hrs Total loading time: 0 Render date: 2024-11-15T16:41:36.751Z Has data issue: false hasContentIssue false
  • English
  • Français

Association between suicidal ideation and suicide: meta-analyses of odds ratios, sensitivity, specificity and positive predictive value

Published online by Cambridge University Press:  31 January 2019

Catherine M. McHugh ,
Amy Corderoy ,
Christopher James Ryan ,
Ian B. Hickie  and
Matthew Michael Large
Catherine M. McHugh
Affiliation:
Brain and Mind Centre, University of Sydney, Australia
Amy Corderoy
Affiliation:
Doctor of Medicine Candidate, School of Medicine, University of Notre Dame Australia, Australia
Christopher James Ryan
Affiliation:
Clinical Associate Professor, Consultation-Liaison Psychiatrist, Westmead Hospital, Discipline of Psychiatry; and Sydney Health Ethics, University of Sydney, Australia
Ian B. Hickie
Affiliation:
Co-Director, Health and Policy, The University of Sydney Central Clinical School Brain and Mind Centre Faculty of Medicine and Health, University of Sydney, Australia
Matthew Michael Large
Affiliation:
Conjoint Professor, School of Psychiatry, University of New South Wales, Australia
Rights & Permissions [Opens in a new window]

Abstract

Background

The expression of suicidal ideation is considered to be an important warning sign for suicide. However, the predictive properties of suicidal ideation as a test of later suicide are unclear.

Aims

To assess the strength of the association between suicidal ideation and later suicide measured by odds ratio (OR), sensitivity, specificity and positive predictive value (PPV).

Method

We located English-language studies indexed in PubMed that reported the expression or non-expression of suicidal ideation among people who later died by suicide or did not. A random effects meta-analysis was used to assess the pooled OR, sensitivity, specificity and PPV of suicidal ideation for later suicide among groups of people from psychiatric and non-psychiatric settings.

Results

There was a moderately strong but highly heterogeneous association between suicidal ideation and later suicide (n = 71, OR = 3.41, 95% CI 2.59–4.49, 95% prediction interval 0.42–28.1, I2 = 89.4, Q-value = 661, d.f.(Q) = 70, P ≤0.001). Studies conducted in primary care and other non-psychiatric settings had similar pooled odds to studies of current and former psychiatric patients (OR = 3.86 v. OR = 3.23, P = 0.7). The pooled sensitivity of suicidal ideation for later suicide was 41% (95% CI 35–48) and the pooled specificity was 86% (95% CI 76–92), with high between-study heterogeneity. Studies of suicidal ideation expressed by current and former psychiatric patients had a significantly higher pooled sensitivity (46% v. 22%) and lower pooled specificity (81% v. 96%) than studies conducted in non-psychiatric settings. The PPV among non-psychiatric cohorts (0.3%, 95% CI 0.1%–0.5%) was significantly lower (Q-value = 35.6, P < 0.001) than among psychiatric samples (3.9%, 95% CI 2.2–6.6).

Conclusions

Estimates of the extent of the association between suicidal ideation and later suicide are limited by unexplained between-study heterogeneity. The utility of suicidal ideation as a test for later suicide is limited by a modest sensitivity and low PPV.

Declaration interest

M.M.L. and C.J.R. have provided expert evidence in civil, criminal and coronial matters. I.B.H. has been a Commissioner in Australia's National Mental Health Commission since 2012. He is the Co-Director, Health and Policy at the Brain and Mind Centre (BMC) University of Sydney. The BMC operates an early-intervention youth services at Camperdown under contract to Headspace. I.B.H. has previously led community-based and pharmaceutical industry-supported (Wyeth, Eli Lily, Servier, Pfizer, AstraZeneca) projects focused on the identification and better management of anxiety and depression. He is a Board Member of Psychosis Australia Trust and a member of Veterans Mental Health Clinical Reference group. He was a member of the Medical Advisory Panel for Medibank Private until October 2017. He is the Chief Scientific Advisor to, and an equity shareholder in, InnoWell. InnoWell has been formed by the University of Sydney and PricewaterhouseCoopers to administer the $30 M Australian Government Funded Project Synergy. Project Synergy is a 3-year programme for the transformation of mental health services through the use of innovative technologies.

Keywords

Suicidemortalityrisk assessment
Type
Papers
Information
BJPsych Open , Volume 5 , Issue 2 , March 2019 , e18
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.
Copyright
Copyright © The Royal College of Psychiatrists 2019

Health professionals are expected to have the skills to assess and manage patients with suicidal thoughts.Reference Weber, Michail, Thompson and Fiedorowicz1 The presence of thoughts of suicide and the expression of suicidal ideation are signs of significant distress, and all patients who express suicidal ideation require a careful clinical assessmentReference Rives2, Reference Hawley, James, Birkett, Baldwin, de Ruiter and Priest3 leading to a comprehensive treatment plan.Reference Hawley, James, Birkett, Baldwin, de Ruiter and Priest3 In addition, suicidal ideation is sometimes considered to be the most important sign of short-term suicide risk,Reference Morgan and Stanton4 and it has been argued that questions about suicidal ideation play a crucial role in suicide screening.Reference Boudreaux, Camargo, Arias, Sullivan, Allen and Goldstein5, Reference Horowitz, Snyder, Ludi, Rosenstein, Kohn-Godbout and Lee6

Despite the undoubted clinical importance of suicidal ideation, meta-analysis has only been used recently to estimate the statistical strength of suicidal ideation as a test for later suicide. Several meta-analyses have highlighted the modest strength of the association between suicidal ideation and later suicideReference Chapman, Mullin, Ryan, Kuffel, Nielssen and Large7–Reference Franklin, Ribeiro, Fox, Bentley, Kleiman and Huang10 and one has quantified the positive predictive value (PPV).Reference Hubers, Moaddine, Peersmann, Stijnen, van Duijn and van der Mast8 To date, no meta-analysis has examined the sensitivity and specificity of suicidal ideation for suicide. Knowledge of the sensitivity of suicidal ideation for later suicide is particularly important because screening tests should be sufficiently sensitive, such that a high proportion of affected people screen positive.Reference Wilson and Jungner11 Moreover, existing meta-analyses of the association between suicide and suicidal ideation have been limited to studies with a cohort design,Reference Ribeiro, Franklin, Fox, Bentley, Kleiman and Chang9 only included patients with particular diagnosesReference Chapman, Mullin, Ryan, Kuffel, Nielssen and Large7 or included studies where suicidal ideation was established post-mortem using psychological autopsy methods.Reference Hubers, Moaddine, Peersmann, Stijnen, van Duijn and van der Mast8

We report a meta-analysis of cohort and case–control studies that examined the presence or absence of expressed suicidal ideation among those who later did or did not die by suicide.

Aims and hypotheses

The first aim of this study was to calculate a comprehensive set of measures of prediction of suicidal ideation for later suicide, including of effect size as measured by the pooled odds ratio (OR), sensitivity, specificity, meta-analytically derived receiver operator curve (ROC) and area under the curve (AUC) and the PPV. The second aim was to explore the extent of, and moderators of, between-study heterogeneity in these predictive measures. We hypothesised that suicidal ideation would be a more sensitive and specific test for suicide in non-psychiatric settings, where patients with suicidal ideation can be expected to have fewer additional and potentially confounding risk factors.

Methods

We conducted a registered meta-analysis (Prospero 42017059236) according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (see supplementary Table 1 available at https://doi.org/10.1192/bjo.2018.88).Reference Moher, Liberati, Tetzlaff and Altman12

Searches

Multiple search strategies were initially explored using the databases Medline, Embase and PsycINFO from inception to January 2017. These searches yielded an excessive number of titles when the term ‘suicide’ was used as a keyword (>200 000) or in the title (>70 000). Moreover, placing limits in the searches using relevant search terms (such as ‘ideation’ or ‘mortality’) missed numerous relevant papers that were known to the authors through earlier hand searching. However, PubMed indexed all but one English-language publication located by searches of multiple databases in earlier studies.Reference Chapman, Mullin, Ryan, Kuffel, Nielssen and Large7, Reference Hubers, Moaddine, Peersmann, Stijnen, van Duijn and van der Mast8 Therefore, and in order to obtain a large and representative sample of relevant studies, one author (M.M.L.) examined the titles of English-language publications with an accompanying abstract that contained variants of the single term ‘suicide’ (suicid*) in their title and were published in PubMed from inception to 14 September 2017 and hand searched the references list of relevant review articles.Reference Chapman, Mullin, Ryan, Kuffel, Nielssen and Large7–Reference Franklin, Ribeiro, Fox, Bentley, Kleiman and Huang10, Reference Large, Smith, Sharma, Nielssen and Singh13, Reference Large, Sharma, Cannon, Ryan and Nielssen14 Two authors (M.M.L. and C.M.M.) winnowed the resulting abstracts and full-text publications (Fig. 1).

Fig. 1 Flow chart of searches.

Included studies

Studies were included if they reported the expression or non-expression of suicidal ideation among people who later died by suicide or did not, such that all individuals could be classified into the following groups: true positive (those with suicidal ideation and suicide), false positive (those with suicidal ideation but without suicide), false negative (those without suicidal ideation but with suicide) or true negative (those without suicidal ideation or suicide). Studies were included if they provided effect size or other data that was sufficient to enable the calculation of the numbers of people in the true positive, false positive, false negative and true negative categories.

We included studies of patients who had received psychiatric care and people recruited from non-psychiatric settings including primary healthcare, general population samples, military populations and prisons. We excluded studies in which the dependent variable was suicide attempts, or when suicidal ideation was assessed by psychological autopsy methods, when all the controls were deceased or all of the individuals had made suicide attempts. We excluded studies of patients with severe medical illnesses such as malignancies or human immunodeficiency virus.

Definitions of suicide and suicidal ideation

Two authors (M.M.L. and C.M.M.) independently extracted the numbers of individuals with suicidal ideation among those who had died by suicide and controls and the moderator variables. A third author (A.C.) re-examined the data points. The discrepant points were re-examined by two authors. We accepted the definition of suicidal ideation used in the primary research, acknowledging that the primary research might not have fully reflected differences in the way suicidal ideation was expressed. Studies were coded into four categories according to their definition of suicidal ideation: studies that reported a composite measure of suicidality (inclusive of both suicidal ideation and behaviour); studies that reported suicide plans (including suicide ‘threats’ as reported in the primary studies); studies that recorded a wish to die; and the majority of studies in which the nature of the suicidal ideation or suicidal thoughts was not specified.

Moderator variables

We collected data about potential moderators of the association between suicidal ideation and suicide; (a) whether the individuals were from psychiatric or non-psychiatric samples; (b) whether the samples consisted of hospital-treated patients; (c) the mean sample age; (d) the duration of follow-up after the assessment for suicidal ideation; (e) whether the study had a cohort or control design; (f) whether patients with suicidal behaviour were regarded as having suicidal ideation; (g) the year of study publication; (h) the proportion of individuals included with suicidal ideation; and (i) the incidence of suicide in the study.

Assessment of strength of reporting

Two researchers (M.M.L. and C.M.M.) independently assessed the reporting strength (and hence the risk of bias) of each primary study's research using a scale with four further moderator variables (scored 0–4) derived from the Newcastle Ottawa Scale.Reference Wells, Shea, O'Connnell, Peterson, Welch and Losos15 One point was allocated according to each of the following criteria: (a) use of a structured method to assess suicidal ideation; (b) collection of data about suicidal ideation in a method that was masked to the patient's suicide (either by a masking method or by electronic recording at the point of assessment); (c) if all the suicides were defined using a mortality database; and (d) inclusion of open verdicts as suicide. Studies counting open verdicts were rated as having stronger reporting because open verdicts are thought to be similar to suicides in some jurisdictionsReference Linsley, Schapira and Kelly16 and because the presence of a history of suicidal ideation might make a coronial verdict of suicide more likely.

Data synthesis

Random effects meta-analysis was chosen a priori for all estimates (OR, sensitivity, specificity, PPV) because of the diversity of study populations and the differences in methods used in the primary research. Between-study heterogeneity in effect sizes was examined using I2, Q-value statistics and prediction intervals. The possibility of publication bias was assessed using a funnel plot and Egger's regression,Reference Egger, Davey Smith, Schneider and Minder17 and was quantified using Duval & Tweedie's trim and fill method.Reference Duval and Tweedie18 Subgroup analysis (random effects within subgroups) and random-effects meta-regression (method of moments) were used to explore the extent to which between-study heterogeneity was explained by categorical and continuous variables (including the year of publication and the proportion of all participants with suicidal ideation).

Moderator variables (including strength of reporting score items) that were significantly associated with between-study heterogeneity in sensitivity and specificity at P≤0.05 were included in multiple meta-regression models. Multiple meta-regression was not used to examine heterogeneity in ORs because of the absence of significant moderator variables. Comprehensive Meta-Analysis (Version 3, Biostat, Englewood NJ) was used for the main analysis and a meta-analytic estimate of the ROC and AUC was calculated using Meta-DiSc.Reference Zamora, Abraira, Muriel, Khan and Coomarasamy19

Results

Searches and data extraction

The initial searches yielded 320 potentially relevant papers that were examined in full text. There were 70 papers,Reference Allebeck, Varla, Kristjansson and Wistedt20–Reference Yim, Yip, Li, Dunn, Yeung and Miao89 reporting 71 studies in which true positive, false positive, false negative and true negative could be ascertained. Eight differences in the effect size in the independent data extraction were resolved by re-examination of the data (Table 1 and see supplementary Table 2 for the data used in meta-analysis).

Table 1 Included studies

The earliest study was published in 1961 and the latest in 2017. The median publication date was 2006. In total, 42 studies had a case–control design and 29 were cohort studies. Of the studies, 58 studies were of current or former psychiatric patients including 42 studies of current or former psychiatric in-patients.

In 52 studies suicidal ideation was defined clinically and 19 studies used a structured method. Twelve studies reported a composite measure of suicidality (inclusive of suicide attempts). Four studies reported on suicide plans and two studies recorded a wish to die rather than active suicidal ideation.

The 71 studies included in total of 4 669 303 individuals who had been assessed for suicidal ideation (mean 65 765, median 235) of whom 5 811 died by suicide (mean 81.8, median 49). In total there were 2082 people in the true positive, 125 034 in the false positive, 3729 in the false negative and 4 538 458 in the true negative groups.

Meta-analyses

Meta-analysis of odds ratios

The pooled OR of suicide associated with suicidal ideation was 3.41 (95% CI 2.59–4.49; 95% prediction interval (PI) 0.42–28.1). There was very marked between-study heterogeneity, I 2 89.4, Q-value = 661, d.f.(Q) = 70, P<0.001. The range of ORs was 0.28 to 81, the first quartile,  2.03, median, 2.96 and the third quartile  6.45 (Fig. 2).

Fig. 2 Forest plot of the odds of the association between suicidal ideation and suicide.

This was associated with a meta-analytic AUC of 0.676 (s.e. = 0.021) (Fig. 3). The funnel plot appeared symmetrical (Fig. 4) but an Egger's regression was borderline significant (intercept 1.55, t-value = 2.07, d.f. = 69, P = 0.04). Duval & Tweedie's trim and fill did not identify any hypothetically missing studies.

Fig. 3 Meta-analytic area under the curve.

Fig. 4 Funnel plot of the odds of the association between suicidal ideation and suicide.

Structured methods to assess suicidal ideation were associated with a non-significantly lower OR (OR = 2.38, 95% CI 1.14–4.99) than clinically defined suicidal ideation (OR = 3.72, 95% CI 2.96–4.67, Q-value = 1.28, d.f.(Q) = 1, P = 0.26). There was no evidence that studies that reported on a composite measure of suicidality spanning suicidal behaviour and suicidal ideation (OR = 3.09, 95% CI 2.02–4.72) differed from studies that simply reported on suicidal ideation (OR = 3.54, 95% CI 2.57–4.88, Q = 0.26, d.f.(Q) = 1, P = 0.61). Suicide plans were more strongly associated with suicide (four studies, OR = 8.51, 95% CI 5.51–13.06) than studies that reported a wish to die (two studies, OR = 3.01, 95% CI 1.49–6.06) and studies that did not specify the degree of intent or planning (65 studies, OR = 3.2, 95% CI 2.41–4.29, Q-value = 14.6, d.f.(Q) = 2, P = 0.001)

Studies that were considered to have stronger reporting on the basis of a total strength of reporting score of ≥2 had a non-significantly weaker association (Q-value = 3.58, d.f.(Q) = 1, P = 0.06) between suicidal ideation and later suicide (n = 37, OR = 2.70, 95% CI 1.81–4.01) than studies with a strength of reporting score of <2 (n = 34, OR = 4.41, 95% CI 3.20–6.08).

Between-study heterogeneity was not significantly explained by any of our predetermined moderator variables (Table 2). Studies of non-psychiatric samples (n = 13, OR = 3.86, 95% CI 2.67–3.97) had similar ORs to studies of psychiatric patients (n = 58, OR = 3.23, 95% CI 2.64–3.96; Q-value = 0.119, d.f.(Q) = 1, P = 0.7). The main result and the analyses of moderators were not sensitive to the exclusion of two non-psychiatric studies with very large sample sizes that when combined included over 90% of individuals and 3.5% of the study weight in the random-effects meta-analysis.Reference Crandall, Fullerton-Gleason, Aguero and LaValley36, Reference Hyman, Ireland, Frost and Cottrell57

Table 2 Meta-regression of moderators of the odds ratio of suicidal ideation for later suicide

a. 58 studies in the analysis.

b. 68 studies included in the analysis.

Meta-analysis of sensitivity

The pooled sensitivity of suicidal ideation for suicide was 41% (95% CI 35–48). There was very high between-study heterogeneity, I 2 = 93.2, Q-value = 1033, d.f.(Q) = 70, P≤0.001. The sensitivity ranged from 0 to 97%, first quartile  25%, median 44% and the third quartile  67%. An Egger's regression was non-significant (intercept 0.60, t-value = 0.61, d.f. = 69, P = 0.54).

Studies of non-psychiatric populations had a significantly lower sensitivity (n = 13, sensitivity  22%, 95% CI 13–53) than studies among psychiatric patients (n = 58, sensitivity  46%, 95% CI 40–52; Q-value = 10.6, d.f.(Q) = 1, P = 0.001). This result was not sensitive to the exclusion of the two non-psychiatric studies with very large sample sizes.

Use of a mortality database and more recent publication were associated with a lower sensitivity. Sensitivity was increased among studies of patients in hospital and in studies with a higher proportion of individuals with suicidal ideation (Table 3). The proportion of people with suicidal ideation was the only factor that was independently associated with between-study heterogeneity in a multiple meta-regression (supplementary Table 3). The multiple meta-regression model suggested that the proportion of people with suicidal ideation was the only independent moderator and explained 58% of the between-study heterogeneity in sensitivity.

Table 3 Meta-regression of moderators of the sensitivity of suicidal ideation for later suicide

a. 58 studies included in the analysis.

b. 68 studies in the analysis.

Meta-analysis of specificity

The pooled specificity of suicidal ideation for later suicide was 86% (95% CI 76–92). There was very high between-study heterogeneity, I 2 = 99.9, Q-value >10 000, d.f. (Q) = 70, P≤0.001. The specificity ranged from 28 to 100%, first quartile 69%, median  83% and the third quartile  93%. An Egger's regression was non-significant (intercept  −7.27 t-value = 0.74, d.f. = 69, P = 0.46).

Studies of non-psychiatric populations had a higher specificity (n = 13, specificity 96%, 95% CI 84–99) than studies of psychiatric populations (n = 58, specificity  81%, 95% CI 76–85, Q-value = 4.78, d.f.(Q) = 1, P = 0.03). This result was not sensitive to the exclusion of the non-psychiatric studies with very large sample sizes.

Suicidal ideation was a less specific indicator of suicide among studies with a high proportion of people with suicidal ideation and among samples of hospital-treated patients. A multivariate model found that hospital treatment and the proportion of people with suicidal ideation in the study population were significant variables, explaining 79% of the between-study heterogeneity in specificity. (Table 4 and supplementary Table 4).

Table 4 Meta-regression tests of potential moderators of the specificity of suicidal ideation for later suicide

a. 58 studies in the analysis.

b. 68 studies included in the analysis.

Meta-analysis of PPV

The pooled PPV for suicidal ideation as a test for later suicide among the 29 cohort studies was 1.7% (95% CI 0.9–3.2) over an average duration of follow-up of 9.1 years. There was very high between-study heterogeneity, I 2 = 97.9, Q-value = 1186, d.f.(Q) = 28, P≤0.001. An Egger's regression was non-significant (intercept 2.81, t-value = 1.13, d.f. = 27, P = 0.27). The PPV among nine non-psychiatric cohorts (0.3% 95% CI 0.1–0.5) was significantly lower (Q-value = 35.6, d.f.(Q) = 1, P<0.001) than among psychiatric samples (3.9% 95% CI 2.2–6.6).

Discussion

Main findings

Clinicians sometimes rely on suicidal ideation as a crucial test for short-term suicide risk,Reference Morgan and Stanton4 and it has been argued that asking about suicidal ideation could form part of a screening test for later suicide.Reference Boudreaux, Camargo, Arias, Sullivan, Allen and Goldstein5, Reference Horowitz, Snyder, Ludi, Rosenstein, Kohn-Godbout and Lee6 Our finding of a pooled OR of suicide associated with suicidal ideation of 3.41 and the meta-analytic AUC of 0.676 indicate a statistical association between suicidal ideation and later suicide with moderate strength.Reference Rosenthal90 However, this should be interpreted very cautiously because of the very high and largely unexplained heterogeneity between studies. Moreover, the low PPV of suicidal ideation for suicide, which flows from the low incidence of suicide outcomes, even among psychiatric cohorts, highlights the limitations of suicidal ideation as a practical test of future suicide.

We had hypothesised that suicidal ideation would be a more sensitive and specific test for suicide in non-psychiatric settings, where patients with suicidal ideation might have fewer other risk factors. We found no evidence that suicidal ideation is more strongly associated with suicide in non-psychiatric settings than in psychiatric settings. Contrary to our hypothesis we found that suicide ideation was a less sensitive test for later suicide in non-psychiatric settings, and that the prevalence of suicidal ideation is associated with the sensitivity for later suicide.

Our results can be compared with those of recent meta-analyses of the psychometric properties of suicide risk scalesReference Carter, Milner, McGill, Pirkis, Kapur and Spittal91, Reference Chan, Bhatti, Meader, Stockton, Evans and O'Connor92 and multivariate models of suicide risk,Reference Large, Kaneson, Myles, Myles, Gunaratne and Ryan93 which found similar odds of suicide, sensitivity, specificity and PPV among studies conducted in psychiatric settings as well as similar between-study heterogeneity despite examining quite different study sets. Collectively, these studies highlight a high degree of uncertainty about the statistical strength of commonly used approaches to suicide risk assessment.

The current paper advances knowledge of the association between suicidal ideation and suicide by reporting the pooled sensitivity and specificity of suicidal ideation for suicide. The main finding is the limited sensitivity of suicidal ideation for suicide, such that approximately 60% of people who go on to die by suicide have not expressed suicidal ideation at a specified earlier time. In contrast, the non-expression of suicidal ideation is fairly specific to not dying by suicide in the study period, with only 14% of those not dying by suicide expressing suicidal ideation.

Implications

Knowledge of the sensitivity and specificity of suicidal ideation for suicide is important because any given OR can be achieved with differing trade-offs between sensitivity and specificity. In this study we found firm evidence that such trade-offs differ between studies, in part as a result of the proportion of individuals expressing suicidal ideation. The proportion of people with suicidal ideation was strongly associated with sensitivity and was inversely associated with specificity, such that similar odds of suicide were found in populations with higher rates of suicidal ideation (i.e. psychiatric patients) and lower rates of suicidal ideation (i.e. non-psychiatric populations).

In general, screening tests should be sensitive enough to capture most cases.Reference Wilson and Jungner11 Our study suggests that suicidal ideation is not sensitive enough to be very helpful as a stand-alone screening test for suicide in psychiatric or non-psychiatric settings, a limitation that is compounded by the modest specificity. Clinicians should not assume that patients experiencing mental distress without suicidal ideation are not at elevated risk of suicide. It is notable that our subgroup analysis of non-psychiatric settings found that nearly 80% of people who later die by suicide had not expressed suicidal ideation. Conversely suicidal ideation is somewhat specific to suicide, particularly in settings where the prevalence of suicidal ideation is low. In this sense, the presence of suicidal ideation conveys more salient information about later suicide than the absence of suicidal ideation.

The finding that the proportion of individuals with suicidal ideation was strongly associated with sensitivity and inversely associated with specificity requires some consideration. To some extent this result is a logical consequence of the definition of sensitivity as the proportion of all cases detected – hypothetically, an extremely sensitive test that included the slightest degree of suicidal ideation would likely select almost the entire population and would approach a 100% sensitivity. Although variation in the threshold test for suicidal ideation might have contributed to variation in the prevalence of suicidal ideation between studies in non-extreme examples with typical rates of suicidal ideation, a very wide variation in the proportion of people with suicidal ideation can be associated with a wide variation of sensitivities.

In addition to highlighting uncertainty in the strength of the association between suicidal ideation and suicide, this study highlights an important clinical dilemma about the trade-off between sensitivity and specificity. Although detailed questioning about suicidal ideation is often indicated, for example when there is suspicion of a suicide plan, and such questioning might bring about a greater sensitivity, it might be associated with a higher false positive rate than less detailed questioning.

Limitations

In addition to the main limitation posed by the extent of between-study heterogeneity, this meta-analysis has a number of other limitations. In most studies there was a lack of clarity about how suicidal ideation was determined and defined. However, this limitation might not have had very much impact on our results because we found little evidence that structured methods of assessing suicidal ideation predicted suicide more accurately than clinical methods. Although it is logical that there may be important threshold issues on the spectrum of severity of suicidal ideation – a possibility supported by our finding of an association between the rate of suicidal ideation and the sensitivity of suicidal ideation for later suicide – there were too few studies that reported different points on the spectrum of ideation to properly explore threshold issues. Another limitation is that we lacked any data about how factors such as gender, comorbid conditions and wider cultural factors might affect the association between suicidal ideation and suicide.

Our results should also be interpreted in light of the knowledge that suicidal ideation can fluctuate over short periods of time.Reference Williams, Davidson and Montgomery94 The primary studies we included measured suicidal ideation at the beginning of a follow-up period, often a follow-up of years. In cases of individuals who died by suicide without earlier suicidal ideation it is likely that suicidal ideation emerged closer to the time of the suicide. It remains to be seen if real time or more temporally proximal measures of suicidal ideation will be more strongly associated with later suicide.Reference Kleiman, Turner, Fedor, Beale, Huffman and Nock95

In conclusion, enquiring about suicidal ideation will always be a central skill for mental health professionals, not because suicidal ideation is a meaningful forecast of suicide but because patients who express suicidal ideation are making important communications about their inner world and level of psychological distress. However, clinicians should be aware of the statistical limitations of ideation as a screening tool, and not be lured into either a false confidence generated by an absence of ideation, or determinism about the likelihood of suicide if it is present.

Supplementary material

Supplementary material is available online at https://doi.org/10.1192/bjo.2018.88.

References

1Weber, AN, Michail, M, Thompson, A, Fiedorowicz, JG. Psychiatric emergencies: assessing and managing suicidal ideation. Med Clin North Am 2017; 101: 553–71.Google Scholar
2Rives, W. Emergency department assessment of suicidal patients. Psychiatr Clin North Am 1999; 22: 779–87.Google Scholar
3Hawley, CJ, James, DV, Birkett, PL, Baldwin, DS, de Ruiter, MJ, Priest, RG. Suicidal ideation as a presenting complaint. Associated diagnoses and characteristics in a casualty population. Br J Psychiatry 1991; 159: 232–8.Google Scholar
4Morgan, HG, Stanton, R. Suicide among psychiatric in-patients in a changing clinical scene. Suicidal ideation as a paramount index of short-term risk. Br J Psychiatry 1997; 171: 561–3.Google Scholar
5Boudreaux, ED, Camargo, CA Jr, Arias, SA, Sullivan, AF, Allen, MH, Goldstein, AB, et al. Improving suicide risk screening and detection in the emergency department. Am J Prev Med 2016; 50: 445–53.Google Scholar
6Horowitz, LM, Snyder, D, Ludi, E, Rosenstein, DL, Kohn-Godbout, J, Lee, L, et al. Ask suicide-screening questions to everyone in medical settings: the asQ'em Quality Improvement Project. Psychosomatics 2013; 54: 239–47.Google Scholar
7Chapman, CL, Mullin, K, Ryan, CJ, Kuffel, A, Nielssen, O, Large, MM. Meta-analysis of the association between suicidal ideation and later suicide among patients with either a schizophrenia spectrum psychosis or a mood disorder. Acta Psychiatr Scand 2015; 131: 162–73.Google Scholar
8Hubers, AA, Moaddine, S, Peersmann, SH, Stijnen, T, van Duijn, E, van der Mast, RC, et al. Suicidal ideation and subsequent completed suicide in both psychiatric and non-psychiatric populations: a meta-analysis. Epidemiol Psychiatr Sci 2018; 27:186–98.Google Scholar
9Ribeiro, JD, Franklin, JC, Fox, KR, Bentley, KH, Kleiman, EM, Chang, BP, et al. Self-injurious thoughts and behaviors as risk factors for future suicide ideation, attempts, and death: a meta-analysis of longitudinal studies. Psychol Med 2016; 46: 225–36.Google Scholar
10Franklin, JC, Ribeiro, JD, Fox, KR, Bentley, KH, Kleiman, EM, Huang, X, et al. Risk factors for suicidal thoughts and behaviors: a meta-analysis of 50 years of research. Psychologic Bull 2017; 143: 187–232.Google Scholar
11Wilson, J, Jungner, G. Principles and Practice of Screening for Disease. World Health Organization, 1968.Google Scholar
12Moher, D, Liberati, A, Tetzlaff, J, Altman, DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 2009; 6: e1000097.Google Scholar
13Large, M, Smith, G, Sharma, S, Nielssen, O, Singh, SP. Systematic review and meta-analysis of the clinical factors associated with the suicide of psychiatric in-patients. Acta Psychiatr Scand 2011; 124: 18–29.Google Scholar
14Large, M, Sharma, S, Cannon, E, Ryan, C, Nielssen, O. Risk factors for suicide within a year of discharge from psychiatric hospital: a systematic meta-analysis. Aust N Z J Psychiatry 2011; 45: 619–28.Google Scholar
15Wells, GA, Shea, B, O'Connnell, D, Peterson, J, Welch, V, Losos, M, et al. The Newcastle-Ottawa Scale (NOS) for Assessing the Quality of Nonrandomised Studies in Meta-Analysis. The Ottawa Hospital Research Institute, 2013 (http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp).Google Scholar
16Linsley, KR, Schapira, K, Kelly, TP. Open verdict v. suicide - importance to research. Br J Psychiatry 2001; 178: 465–8.Google Scholar
17Egger, M, Davey Smith, G, Schneider, M, Minder, C. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997; 315: 629–34.Google Scholar
18Duval, S, Tweedie, R. Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics 2000; 56: 455–63.Google Scholar
19Zamora, J, Abraira, V, Muriel, A, Khan, K, Coomarasamy, A. Meta-DiSc: a software for meta-analysis of test accuracy data. BMC Med Res Methodol 2006; 6: 31.Google Scholar
20Allebeck, P, Varla, A, Kristjansson, E, Wistedt, B. Risk factors for suicide among patients with schizophrenia. Acta Psychiatr Scand 1987; 76: 414–9.Google Scholar
21Al-Sayegh, H, Lowry, J, Polur, RN, Hines, RB, Liu, F, Zhang, J. Suicide history and mortality: a follow-up of a national cohort in the United States. Arch Suicide Res 2015; 19: 35–47.Google Scholar
22Appleby, L, Dennehy, JA, Thomas, CS, Faragher, EB, Lewis, G. Aftercare and clinical characteristics of people with mental illness who commit suicide: a case-control study. Lancet 1999; 353: 1397–400.Google Scholar
23Baader-Matthei, T, Richter, P, Mundt, C. Suicides of psychiatric hospitalized patients and their risk factors. A Case-control study. Rev Chil Neuro-Psiquiat 2004; 42: 293–316.Google Scholar
24Beck, AT, Brown, GK, Steer, RA, Dahlsgaard, KK, Grisham, JR. Suicide ideation at its worst point: a predictor of eventual suicide in psychiatric outpatients. Suicide Life Threat Behav 1999; 29: 1–9.Google Scholar
25Beisser, AR, Blanchette, JE. A study of suicides in a mental hospital. Dis Nerv Syst 1961; 22: 365–9.Google Scholar
26Berg, JE. Death by suicide long after electroconvulsive therapy. Is the sense of coherence test of Antonovsky a predictor of mortality from depression? Ment Illn 2010; 2: e3.Google Scholar
27Bertelsen, M, Jeppesen, P, Petersen, L, Thorup, A, Ohlenschlaeger, J, le Quach, P, et al. Suicidal behaviour and mortality in first-episode psychosis: the OPUS trial. Br J Psychiatry 2007; 191 (suppl 51): s140–6.Google Scholar
28Bickley, H, Hunt, IM, Windfuhr, K, Shaw, J, Appleby, L, Kapur, N. Suicide within two weeks of discharge from psychiatric inpatient care: a case-control study. Psychiatr Serv 2013; 64: 653–9.Google Scholar
29Blumenthal, S, Bell, V, Neumann, NU, Schuttler, R, Vogel, R. Mortality and rate of suicide of first admission psychiatric patients. A 5–year follow-up of a prospective longitudinal study. Psychopathology 1989; 22: 50–6.Google Scholar
30Borg, SE, Stahl, M. Prediction of suicide. A prospective study of suicides and controls among psychiatric patients. Acta Psychiatr Scand 1982; 65: 221–32.Google Scholar
31Bradvik, L, Berglund, M. Risk factors for suicide in melancholia. A case-record evaluation of 89 suicides and their controls. Acta Psychiatr Scand 1993; 87: 306–11.Google Scholar
32Cheng, KK, Leung, CM, Lo, WH, Lam, TH. Risk factors of suicide among schizophrenics. Acta Psychiatr Scand 1990; 81: 220–4.Google Scholar
33Conlon, L, Garland, M, Prescott, P, Mannion, L, Leonard, M, Fahy, TJ. Psychiatric aftercare and suicide risk: a case-control study using blind rating. Arch Suicide Res 2007; 11: 291–5.Google Scholar
34Coryell, W, Young, EA. Clinical predictors of suicide in primary major depressive disorder. J Clin Psychatry 2005; 66: 412–7.Google Scholar
35Coryell, W, Kriener, A, Butcher, B, Nurnberger, J, McMahon, F, Berrettini, W, et al. Risk factors for suicide in bipolar I disorder in two prospectively studied cohorts. J Affect Disord 2016; 190: 1–5.Google Scholar
36Crandall, C, Fullerton-Gleason, L, Aguero, R, LaValley, J. Subsequent suicide mortality among emergency department patients seen for suicidal behavior. Acad Emerg Med 2006; 13: 435–42.Google Scholar
37De Hert, M, McKenzie, K, Peuskens, J. Risk factors for suicide in young people suffering from schizophrenia: a long-term follow-up study. Schizophr Res 2001; 47: 127–34.Google Scholar
38Didham, R, Dovey, S, Reith, D. Characteristics of general practitioner consultations prior to suicide: a nested case-control study in New Zealand. N Z Med J 2006; 119: U2358.Google Scholar
39Dingman, CW, McGlashan, TH. Discriminating characteristics of suicides. Chestnut Lodge follow-up sample including patients with affective disorder, schizophrenia and schizoaffective disorder. Acta Psychiatr Scand 1986; 74: 91–7.Google Scholar
40Dobscha, SK, Denneson, LM, Kovas, AE, Teo, A, Forsberg, CW, Kaplan, MS, et al. Correlates of suicide among veterans treated in primary care: case-control study of a nationally representative sample. J Gen Intern Med 2014; 29 (suppl 4): 853–60.Google Scholar
41Dong, JY, Ho, TP, Kan, CK. A case-control study of 92 cases of in-patient suicides. J Affect Disord 2005; 87: 91–9.Google Scholar
42Drake, RE, Gates, C, Cotton, PG. Suicide among schizophrenics: a comparison of attempters and completed suicides. Br J Psychiatry 1986; 149: 784–7.Google Scholar
43Dutta, R, Boydell, J, Kennedy, N, Van Os, J, Fearon, P, Murray, RM. Suicide and other causes of mortality in bipolar disorder: a longitudinal study. Psychol Med 2007; 37: 839–47.Google Scholar
44Farberow, NL, Shneidman, ES, Neuringer, C. Case history and hospitalization factors in suicides of neuropsychiatric hospital patients. J Nerv Ment Dis 1966; 142: 32–44.Google Scholar
45Fernando, S, Storm, V. Suicide among psychiatric patients of a district general hospital. Psychol Med 1984; 14: 661–72.Google Scholar
46Flood, RA, Seager, CP. A retrospective examination of psychiatric case records of patients who subsequently committed suicide. Br J Psychiatry 1968; 114: 443–50.Google Scholar
47Fosse, R, Ryberg, W, Carlsson, MK, Hammer, J. Predictors of suicide in the patient population admitted to a locked-door psychiatric acute ward. PLoS One 2017; 12: e0173958.Google Scholar
48Fowler, R, Tsuang, MT, Kronfol, Z. Communication of suicidal intent and suicide in unipolar depression. J Affect Disord 1979; 1: 219–25.Google Scholar
49Fruehwald, S, Matschnig, T, Koenig, F, Bauer, P, Frottier, P. Suicide in custody: case-control study. Br J Psychiatry 2004; 185: 494–8.Google Scholar
50Funahashi, T, Ibuki, Y, Domon, Y, Nishimura, T, Akehashi, D, Sugiura, H. A clinical study on suicide among schizophrenics. Psychiatry Clin Neurosci 2000; 54: 173–9.Google Scholar
51Goldstein, RB, Black, DW, Nasrallah, A, Winokur, G. The prediction of suicide. Sensitivity, Specificity, and predictive value of a multivariate model applied to suicide among 1906 patients with affective disorders. Arch Gen Psychiatry 1991; 48: 418–22.Google Scholar
52Green, KL, Brown, GK, Jager-Hyman, S, Cha, J, Steer, RA, Beck, AT. The predictive validity of the beck depression inventory suicide item. J Clin Psychatry 2015; 76: 1683–6.Google Scholar
53Hoyer, EH, Licht, RW, Mortensen, PB. Risk factors of suicide in inpatients and recently discharged patients with affective disorders. A case-control study. Eur Psychiatry 2009; 24: 317–21.Google Scholar
54Hunt, IM, Kapur, N, Webb, R, Robinson, J, Burns, J, Turnbull, P, et al. Suicide in current psychiatric in-patients: a case-control study. The National Confidential Inquiry into Suicide and Homicide. Psychol Med 2007; 37: 831–7.Google Scholar
55Hunt, IM, Bickley, H, Windfuhr, K, Shaw, J, Appleby, L, Kapur, N. Suicide in recently admitted psychiatric in-patients: a case-control study. J Affect Disord 2013; 144: 123–8.Google Scholar
56Hunt, IM, Kapur, N, Webb, R, Robinson, J, Burns, J, Shaw, J, et al. Suicide in recently discharged psychiatric patients: a case-control study. Psychol Med 2009; 39: 443–9.Google Scholar
57Hyman, J, Ireland, R, Frost, L, Cottrell, L. Suicide incidence and risk factors in an active duty US military population. Am J Public Health 2012; 102 (suppl 1): S138–46.Google Scholar
58Kan, CK, Ho, TP, Dong, JY, Dunn, EL. Risk factors for suicide in the immediate post-discharge period. Soc Psychiatry Psychiatr Epidemiol 2007; 42: 208–14.Google Scholar
59Kessler, RC, Warner, CH, Ivany, C, Petukhova, MV, Rose, S, Bromet, EJ, et al. Predicting suicides after psychiatric hospitalization in US army soldiers: the army study to assess risk and resilience in servicemembers (Army STARRS). JAMA Psychiatry 2015; 72: 49–57.Google Scholar
60Khang, YH, Kim, HR, Cho, SJ. Relationships of suicide ideation with cause-specific mortality in a longitudinal study of South Koreans. Suicide Life Threat Behav 2010; 40: 465–75.Google Scholar
61Khanra, S, Mahintamani, T, Bose, S, Khess, C, Umesh, S, Ram, D. Inpatient suicide in a psychiatric hospital: a nested case-control study. Indian J Psychol Med 2016; 38: 571–6.Google Scholar
62Kim, HM, Smith, EG, Ganoczy, D, Walters, H, Stano, CM, Ilgen, MA, et al. Predictors of suicide in patient charts among patients with depression in the Veterans Health Administration health system: importance of prescription drug and alcohol abuse. J Clin Psychatry 2012; 73: e1269–75.Google Scholar
63Kjelsberg, E, Neegaard, E, Dahl, AA. Suicide in adolescent psychiatric inpatients: incidence and predictive factors. Acta Psychiatr Scand 1994; 89: 235–41.Google Scholar
64Kleiman, EM, Liu, RT. Prospective prediction of suicide in a nationally representative sample: religious service attendance as a protective factor. Br J Psychiatry 2014; 204: 262–6.Google Scholar
65Li, J, Ran, MS, Hao, Y, Zhao, Z, Guo, Y, Su, J, et al. Inpatient suicide in a Chinese psychiatric hospital. Suicide Life Threat Behav 2008; 38: 449–55.Google Scholar
66Lin, SK, Hung, TM, Liao, YT, Lee, WC, Tsai, SY, Chen, CC, et al. Protective and risk factors for inpatient suicides: a nested case-control study. Psychiatry Res 2014; 217: 54–9.Google Scholar
67Lopez-Morinigo, JD, Ayesa-Arriola, R, Torres-Romano, B, Fernandes, AC, Shetty, H, Broadbent, M, et al. Risk assessment and suicide by patients with schizophrenia in secondary mental healthcare: a case-control study. BMJ Open 2016; 6: e011929.Google Scholar
68Louzon, SA, Bossarte, R, McCarthy, JF, Katz, IR. Does suicidal ideation as measured by the PHQ-9 predict suicide among VA patients? Psychiatr Serv 2016; 67: 517–22.Google Scholar
69Lukaschek, K, Baumert, J, Krawitz, M, Erazo, N, Forstl, H, Ladwig, KH. Determinants of completed railway suicides by psychiatric in-patients: case-control study. Br J Psychiatry 2014; 205: 398–406.Google Scholar
70McEwan, T, Mullen, P, MacKenzie, R. Suicide among stalkers. J Forens Psychiatry Psychol 2010; 4: 514–20.Google Scholar
71Mock, CN, Grossman, DC, Mulder, D, Stewart, C, Koepsell, TS. Health care utilization as a marker for suicidal behavior on an American Indian Reservation. J Gen Intern Med 1996; 11: 519–24.Google Scholar
72Motto, JA, Bostrom, A. Empirical indicators of near-term suicide risk. Crisis 1990; 11: 52–9.Google Scholar
73Murphy, GE, Wetzel, RD. The lifetime risk of suicide in alcoholism. Arch Gen Psychiatry 1990; 47: 383–92.Google Scholar
74Park, EH, Hong, N, Jon, DI, Hong, HJ, Jung, MH. Past suicidal ideation as an independent risk factor for suicide behaviours in patients with depression. Int J Psychiatry Clin Pract 2017; 21: 24–8.Google Scholar
75Powell, J, Geddes, J, Deeks, J, Goldacre, M, Hawton, K. Suicide in psychiatric hospital in-patients. Risk factors and their predictive power. Br J Psychiatry 2000; 176: 266–72.Google Scholar
76Roy, A. Suicide in chronic schizophrenia. Br J Psychiatry 1982; 141: 171–7.Google Scholar
77Salama, AA. Depression and suicide in schizophrenic patients. Suicide Life Threat Behav 1988; 18: 379–84.Google Scholar
78Sani, G, Tondo, L, Koukopoulos, A, Reginaldi, D, Kotzalidis, GD, Koukopoulos, AE, et al. Suicide in a large population of former psychiatric inpatients. Psychiatry Clin Neurosci 2011; 65: 286–95.Google Scholar
79Shah, AK, Ganesvaran, T. Inpatient suicides in an Australian mental hospital. Aust N Z J Psychiatry 1997; 31: 291–8.Google Scholar
80Sharma, V, Persad, E, Kueneman, K. A closer look at inpatient suicide. J Affect Disord 1998; 47: 123–9.Google Scholar
81Simon, GE, Rutter, CM, Peterson, D, Oliver, M, Whiteside, U, Operskalski, B, et al. Does response on the PHQ-9 Depression Questionnaire predict subsequent suicide attempt or suicide death? Psychiatr Serv 2013; 64: 1195–202.Google Scholar
82Sinclair, JM, Harriss, L, Baldwin, DS, King, EA. Suicide in depressive disorders: a retrospective case-control study of 127 suicides. J Affect Disord 2005; 87: 107–13.Google Scholar
83Sinclair, JM, Mullee, MA, King, EA, Baldwin, DS. Suicide in schizophrenia: a retrospective case-control study of 51 suicides. Schizophrenia Bull 2004; 30: 803–11.Google Scholar
84Spiessl, H, Hubner-Liebermann, B, Cording, C. Suicidal behaviour of psychiatric in-patients. Acta Psychiatr Scand 2002; 106: 134–8.Google Scholar
85Stephens, JH, Richard, P, McHugh, PR. Suicide in patients hospitalized for schizophrenia: 1913–1940. J Nerv Ment Dis 1999; 187: 10–4.Google Scholar
86Thong, JY, Su, AH, Chan, YH, Chia, BH. Suicide in psychiatric patients: case-control study in Singapore. Aust N Z J Psychiatry 2008; 42: 509–19.Google Scholar
87Weiser, M, Goldberg, S, Werbeloff, N, Fenchel, D, Reichenberg, A, Shelef, L, et al. Risk of completed suicide in 89,049 young males assessed by a mental health professional. Eur Neuropsychopharmacol 2016; 26: 341–9.Google Scholar
88Werbeloff, N, Dohrenwend, BP, Levav, I, Haklai, Z, Yoffe, R, Large, M, et al. Demographic, behavioral, and psychiatric risk factors for suicide. Crisis 2016; 37: 104–11.Google Scholar
89Yim, PH, Yip, PS, Li, RH, Dunn, EL, Yeung, WS, Miao, YK. Suicide after discharge from psychiatric inpatient care: a case-control study in Hong Kong. Aust N Z J Psychiatry 2004; 38: 65–72.Google Scholar
90Rosenthal, J. Qualitative descriptors of strength of association and effect size. J Soc Serv Res 1996; 21: 37–57.Google Scholar
91Carter, G, Milner, A, McGill, K, Pirkis, J, Kapur, N, Spittal, MJ. Predicting suicidal behaviours using clinical instruments: systematic review and meta-analysis of positive predictive values for risk scales. Br J Psychiatry 2017; 210: 387–395.Google Scholar
92Chan, MK, Bhatti, H, Meader, N, Stockton, S, Evans, J, O'Connor, RC, et al. Predicting suicide following self-harm: systematic review of risk factors and risk scales. Br J Psychiatry 2016; 209: 277–83.Google Scholar
93Large, M, Kaneson, M, Myles, N, Myles, H, Gunaratne, P, Ryan, C. Meta-analysis of longitudinal cohort studies of suicide risk assessment among psychiatric patients: heterogeneity in results and lack of improvement over time. PLoS One 2016; 11: e0156322.Google Scholar
94Williams, CL, Davidson, JA, Montgomery, I. Impulsive suicidal behavior. J Clin Psychol 1980; 36: 90–4.Google Scholar
95Kleiman, EM, Turner, BJ, Fedor, S, Beale, EE, Huffman, JC, Nock, MK. Examination of real-time fluctuations in suicidal ideation and its risk factors: results from two ecological momentary assessment studies. J Abnorm Psychol 2017; 126: 726–38.Google Scholar
Figure 0

Fig. 1 Flow chart of searches.

Figure 1

Table 1 Included studies

Figure 2

Fig. 2 Forest plot of the odds of the association between suicidal ideation and suicide.

Figure 3

Fig. 3 Meta-analytic area under the curve.

Figure 4

Fig. 4 Funnel plot of the odds of the association between suicidal ideation and suicide.

Figure 5

Table 2 Meta-regression of moderators of the odds ratio of suicidal ideation for later suicide

Figure 6

Table 3 Meta-regression of moderators of the sensitivity of suicidal ideation for later suicide

Figure 7

Table 4 Meta-regression tests of potential moderators of the specificity of suicidal ideation for later suicide

Supplementary material: File

McHugh et al. supplementary material

McHugh et al. supplementary material 1

Download McHugh et al. supplementary material(File)
File 63.5 KB
Supplementary material: File

McHugh et al. supplementary material

McHugh et al. supplementary material 2

Download McHugh et al. supplementary material(File)
File 19 KB
Supplementary material: File

McHugh et al. supplementary material

McHugh et al. supplementary material 3

Download McHugh et al. supplementary material(File)
File 60.5 KB
Supplementary material: File

McHugh et al. supplementary material

McHugh et al. supplementary material 4

Download McHugh et al. supplementary material(File)
File 51.7 KB
Submit a response

eLetters

No eLetters have been published for this article.