Psychiatrists' appreciation of statistical v. clinical significance: a quick test

Abstract

Aims and Method

Pharmaceutical advertising material can confuse clinical and statistical significance. We used a brief questionnaire (five questions) to evaluate psychiatrists' appreciation of this difference. This approximated to the level of critical appraisal competence of the MRCPsych part 3 examination.

Results

Of the 113 questionnaires distributed 93 were returned complete (response rate 82%). Senior trainees were significantly better than junior trainees at correctly interpreting data (mean score (maximum 5) 2.61v.2.08; P = 0.04). Consultants did less well than senior trainees, although our sample of consultant respondents was too small for significance testing.

Clinical Implications

Learning critical appraisal for the MRCPsych examination may provide psychiatrists with valuable transferable skills and prevent gaps in our knowledge being exploited by misleading study data. Psychiatrists of all grades need to maintain their research appraisal skills and should not regard the MRCPsych examination as the end of their learning.

As psychiatrists, we are presented with research data from various sources, and we need to be able to apply the findings to our clinical practice. We are often given presentations by pharmaceutical representatives, both on a one-to-one basis and in larger groups. Our observations suggest that the mode of data presentation in these settings can be misleading. Data are presented in ways that can appear to show large benefits for a particular medication over existing treatments. This is particularly the case when pharmaceutical companies are trying to differentiate their medication from others in the same class. However, scrutiny of the often colourful charts and tables can reveal less positive results. We decided to assess to what extent learning critical appraisal skills equipped psychiatrists to identify ‘statistical spin’ and measure how confident and competent psychiatrists are in interpreting the results of study data. A questionnaire was devised to test critical appraisal skills and, it was hoped, to stimulate psychiatrists of all grades to maintain these skills. We aimed to make the evaluation educational, and issued doctors with worked answer sheets once they had completed the questionnaire.

Method

We designed a questionnaire to assess psychiatrists’ ability to critically analyse clinical trial data. This was designed to reflect the skills required for the MRCPsych examination (J.W. is Chair of the MRCPsych critical appraisal question panel). After piloting to ten doctors, the questionnaire was distributed at three educational meetings within St Mary's and Charing Cross psychiatry training rotations. At each meeting, a short presentation was given to introduce the study aims, and time was scheduled for doctors to complete the questionnaire and hand it in at the end of the afternoon. Most doctors completed the questionnaire on the day; those who took the questionnaire away were asked to return it the following week. The questionnaire was anonymous, but respondent psychiatrists were asked their grade and year of qualification, whether they qualified and trained in the UK, and to rate how confident they are (on a scale of 0 to 10) in interpreting results of trial data presented by pharmaceutical companies. The questionnaire consisted of five questions and a ‘best fit’ answer from two or three statements (see Appendix 1 in the online supplement to this paper). Those doctors wishing to see the answers were asked to give their email address. We waited until we had collected around 100 completed questionnaires before distributing the worked answers (see Appendix 2 in the online supplement to this paper). Analysis was undertaken using SPSS version 15 for Windows with chi-squared, Mann–Whitney and Kruskal–Wallis tests as appropriate.

Results

In total, 93 questionnaires were returned from 113 distributed (82% response rate); 87 respondents returned the completed questionnaire on the day of distribution and the remaining 6 returned the completed questionnaire within 1 week. All grades of psychiatrist were represented in the study. The year of qualification ranged from 1977 to 2005, with 2000 as the most common year (20 of the 93 doctors qualified that year). Table 1 lists the number of correct answers (out of 5) and the average score per grade. Table 2 lists the number of correct replies (out of 93) and the percentage giving a correct answer for each question.

Table 1. Respondents’ scores

	Respondents n	Correct answers out of 5, n						Score Mean (s.d.)
Grade		0/5	1/5	2/5	3/5	4/5	5/5
SHO (ST1-3)	51	2	19	11	13	4	2	2.08 (1.21)
Staff	6	0	2	0	2	2	0	2.67 (1.37)
SpR (ST4-6)	28	0	4	9	10	4	1	2.61 (1.03)
Consultant	8	0	2	2	4	0	0	2.25 (0.89)
Total	93	2	27	22	29	10	3	2.29 (1.16)

Table 2. Respondents’ answers analysed by question (n = 93)

Respondents answering correctly	Question
	1	2	3	4	5
n	61	34	21	60	37
%	66	37	23	65	40

The modal score was 3 out of 5, with an average of 2.29. In the senior house officer/specialty training (ST) 1–3 year group, 2 of the 51 doctors got all five questions wrong, but 2 in this group got all five correct. One specialist registrar/ST4–6 got all the questions correct. No staff-grade respondent or consultant achieved full marks, but these groups were small compared with the other grades. The average score by grade increased with experience from senior house officer/ST1–3, through staff and specialist registrar/ST4–6 grades. Junior trainees (senior house officers and ST1–3 trainees) were significantly less confident than specialist registrars in their critical appraisal skills (z = –2.45, P = 0.01) and also significantly less competent (z = –2.05, P = 0.04). The average score appeared to fall again at consultant level, although in our small consultant sample this was not significant (χ² = 4.9, d.f. = 3, P = 0.18). We split the year of qualification results into two groups around the median (pre-2001, and 2001 and beyond) but found no significant difference in scores (χ² = 5.7, d.f. = 5, P = 0.34). No significant difference was found whether the respondents were UK-qualified or not (z = –0.78, P = 0.44), or whether they had the bulk of their postgraduate psychiatric training in the UK or not (z = –1.81, P = 0.07). There did appear to be a slight positive correlation between higher ratings of ‘confidence’ and score achieved (r = 0.24, P = 0.02). However, all of the 13 doctors who achieved four or five correct answers rated themselves only ‘quite confident’ rather than ‘very confident’. Most respondents did best on questions 1 and 4. Question 3 appeared to be the most difficult, with only 23% of doctors achieving the correct answer.

Discussion

Our good response rate was largely due to our approaching psychiatrists at local training scheme educational meetings and achieving a ‘captive’ audience. Of the 93 respondents, 83 completed and returned the questionnaire on the day of distribution. Five of the remaining six respondents were consultants who wanted more time to answer the questions, and returned the answer sheet within 1 week. (We do not feel that this altered the overall study results, and assume no conferring took place.)

The results show an increase in critical appraisal confidence as psychiatrists progress from junior to senior posts, which is matched by an increase in competence up to consultant level. It is noteworthy that skills in the consultant group do appear to tail off from those achieved by the specialist registrar/staff grades. This probably reflects the need to develop good appraisal skills for the MRCPsych examination, which can attenuate following membership attainment. Another explanation is that a few consultants had qualified before the critical appraisal examination was introduced, so never acquired the skills. One concern was that the questionnaire was too difficult. However, there were 13 doctors who scored four or more questions correctly. Question 3 was answered least well, and on reflection could be somewhat of a ‘trick’ requiring respondents to spot that the data are non-parametric and t-tests are not applicable. In case this question skewed the findings we looked at the effect of removing it from the results. No significant difference was seen in the results by grade, year qualified, UK-qualified/trained or confidence.

Psychiatrists need to be confident and able to critically appraise the wide variety of research evidence presented to them. An obvious example of this need is our evaluation of pharmaceutical research, although critical appraisal skills are required for many aspects of clinical practice. Many doctors regard their time with pharmaceutical company representatives as an important source of clinical and prescribing information. However, data can be presented in misleading ways and we need to maintain our critical appraisal skills. Recent studies suggest that the colourful charts and tables presented to us can be biased, or even false. Casares et al examined promotional material given by pharmaceutical representatives to family doctors in Spain. ¹ The promotional material was cross-referenced and scrutinised alongside the original study on which the information was supposedly based. In 45% of cases the advertising material was not supported by the research, highlighting the need to view such material with the original study data. Another study investigated whether research funding by pharmaceutical companies had any bearing on the outcomes: Lexchin et al concluded that studies sponsored by pharmaceutical companies were more likely (by an odds ratio greater than 4) to have outcomes favouring the sponsor. ² Research trials showing negative or insignificant effects of new medications may not be published at all.

Although we have concentrated on doctors’ ability to interpret data correctly in the context of pharmaceutical company advertising literature, the skills examined here (and in the MRCPsych critical appraisal paper) are fundamental to many aspects of clinical practice. Despite there being many sources of ‘ready-made’ appraisal such as the Cochrane Collaboration and BMJ Clinical Evidence, clinicians may still need to appraise individual studies prior to making treatment decisions, undertaking audits, writing reports and business cases. We are pleased such skills appear to develop with training but are concerned that the skills may attenuate if not used frequently.

Implications of the study

In order to draw valid and clinically relevant conclusions from research data, we must retain our critical ‘eye’. Psychiatrists need to be able to critically appraise various types of evidence, including that presented by drug companies. This study suggests that the MRCPsych critical appraisal paper may help trainees develop these skills; but we need to maintain them beyond the MRCPsych examination.

Declaration of interest

None.