Evaluation of a mental health website for teenagers

Abstract

Aim and Method

Ru-ok.com is a recently developed website that includes a self-assessment questionnaire. The aim of this study was to evaluate the website and compare the self-assessment questionnaire with established screening questionnaires. A total of 105 teenagers from schools completed three paper-based questionnaires and the online ru-ok.com questionnaire.

Results

The website receives 730 visits a week. Visits to the advice section and stories about mental health and relationships account for 35% of activity. Of the returned questionnaires, 80% were positive about the website. There were modest and expected correlations between the website questionnaire (RU–OK) and the Mood and Feelings (MFQ) and Strength and Difficulties (SDQ) questionnaires.

Clinical Implications

Internet-based self-assessment is feasible and acceptable to teenagers. Self-assessment of perceived need by teenagers may be a useful tool for tier one professionals, including teachers, general practitioners, school nurses, social workers and learning mentors.

A systematic review of health promotion in schools demonstrated the varying success of self-help programmes designed to change risky behaviour, such as smoking or substance misuse (Lister-Sharp et al, 1999). Internet and computer-based assessment and interventions offer an alternative method to be used alongside more traditional approaches. CD–ROMs have shown promise for changing behaviour and promoting better psychological health in the USA (Bosworth et al, 1994, 2000).

The internet is an accessible resource (McAlindon et al, 2003) and media, such as television, can provide information in a non-stigmatising way (Sanders et al, 2000). The internet also facilitates individual autonomy (Theodosiou & Green, 2003), which is an important part of adolescent development. The internet can also provide self-help assessment and interventions to people who do not seek help for their mental health problems (Marks et al, 2003; Christensen et al, 2004; Kenwright et al, 2004).

There is hardly any knowledge of how self-assessment questionnaires completed online compare with paper questionnaires (Ritter et al, 2004). Frequent concerns are that online data may not be secure or reliable because the rater may not understand the instructions, may not take as much care when completing the forms online, or may be influenced by peers to give higher or lower ratings if the online assessment takes place in too public an area, such as an open library, classroom or drop-in centre.

With these issues in mind, the website ru-ok.com was developed in 2003 with the aid of focus groups of teenagers. The research project was funded by the Health Foundation. Focus groups were positive about the final result and the site went live in 2003. The site allows teenagers to assess their strengths and weaknesses and obtain self-help advice. The site includes, among other psychological self-help material, a widely used educational resource, ‘Retracking’ (Bates, 1997) and a self-assessment questionnaire based on the Salford Needs Assessment Schedule for Adolescents (Kroll et al, 1999). There are a range of other features, including interactive stories, cartoons about depression, eating problems, teenage pregnancy and school problems, and some games.

The aims of this study were: (a) to examine whether teenagers find the website acceptable and useful; (b) to validate the RU–OK self-assessment questionnaire by comparing it with standard questionnaires.

Method

Participants

The sample was drawn from a Salford comprehensive school and a special hospital school for children with depression and anxiety.

Measures

The RU–OK self-assessment questionnaire examines problems and strengths of teenagers. The teenager, using a secure method, enters their chosen user name and password. There are 26 questions about problems, each question rated on a 5-point scale from 1 to 5, zero equating to ‘not rated’. The questions are taken from the Salford Needs Assessment Schedule for Adolescents (Kroll et al, 1999). A score of 3 or more is judged to indicate a significant problem and is based on validity data from the original research study. Data can be saved at different time points and reviewed on revisiting the site. The online questionnaire takes about 5–10 min to complete and advice is available for each problem area depending on the score. Low scores would usually lead to advice about self-help, such as using downloadable material from the site or links to other recommended sites. Higher scores would stress the importance of discussing problems with a mentor, parent or professional.

A qualitative questionnaire asking about acceptability, ease of use and usefulness of parts of the website was administered. Each of the questions is rated on a 5-point scale.

The adolescent self-completion version of the Strength and Difficulties Questionnaire (SDQ; Goodman, 1997) consists of 25 items scored on a 3-point scale. In addition to a total score, there are sub-scales for difficulties in the areas of hyperactivity, conduct, emotions and social interactions. There is also one prosocial scale. For epidemiological research studies, cut-off points on each scale can be set in order to classify teenagers into positive and negative groups depending on the research requirements.

The adolescent Mood and Feelings Questionnaire (MFQ; Costello & Angold, 1988) is designed to screen for depression. It consists of 34 questions scored on a 3-point scale. A cut-off point of 27 is often used, giving a sensitivity and specificity of about 80% compared with a gold standard research interview.

Sub-scales from the RU–OK questionnaire were generated so that Pearson correlation could be assessed with the MFQ and SDQ. To do this, scores from certain areas were added together. The sub-scales were:

1. • violence: consisting of two areas, violence to property and persons

2. • emotional: consisting of seven areas, depression, anxiety, trauma, self-harm, eating problems, self-esteem and paranoid thinking

3. • relationships: consisting of two areas, peer problems and family problems

4. • drugs and alcohol: consisting of two areas, drugs and alcohol

5. • total score: consisting of all 26 areas.

The sub-scales and individual scores were checked to ensure normal statistical distribution.

Data collection and loss

The study received local ethical approval and was then explained to teenagres by two research psychiatrists, S.E. and A.K., in a personal, social and health education (PSHE) lesson. The parent or guardian was then informed and teenagers completed consent forms. The following week, teenagers supplied demographic details and completed the SDQ and MFQ. They were given a postcard with a research identification number and instruction about the website. The next PHSE lesson was held in the information technology room and teenagers completed (in confidence) the RU–OK questionnaire and evaluation questionnaire. The same method of data collection was used with the teenagers who attended the hospital school.

Of the teenagers approached, only four did not agree to participate in the study. About 40% of evaluation questionnaires were not completed owing to lack of time in the lesson; the final numbers of questionnaires ranged from 64 to 67. A total of 105 teenagers completed the SDQ and 101 the MFQ questionnaires. A total of 103 completed the RU–OK self-assessment questionnaire but about 30% of the data were lost to analysis because of failure to track pupils’ private identification numbers on the questionnaires. The complete set of questionnaires for analysis of correlation between RU–OK and the SDQ and MFQ was thus 69.

Results

Participants

A total of 105 teenagers were included in the final analysis, 46 boys and 59 girls; 91 teenagers were aged 13–14 years and came from year 9 in the comprehensive school; 14 teenagers came from the hospital school of which 13 were 14- to 15-year-olds (year 10 and 11), the other being from year 9.

Responses

Table 1 shows the responses to the three questionnaires. The sub-scale scores for the SDQ are not significantly different from the national SDQ norms (http://www.sdqinfo.com). The numbers of teenagers classified as having a significant problem on the MFQ and the RU–OK questionnaire are also shown.

Table 1. Responses of participants to the three questionnaires

Questionnaire	Mean	(s.d.)	Below cut-off, n	Above cut-off, n
SDQ, n=105
Total score	11.8	(5.2)
Emotional sub-scale	2.6	(2.0)
Conduct	2.7	(1.7)
Hyperactivity	4.3	(2.3)
Peer problems	2.1	(1.6)
Prosocial score	6.0	(1.7)
MFQ, n=101
Total score	91	(10)
RU-OK, n=103
Looking after self			91	10
Cooking			85	16
Physical illness			95	6
School attendance			93	8
School work			89	12
Employment			4	7
Peer			91	10
Family			88	13
Cultural/racial			94	3
Sexual activity			94	3
Alcohol			100	1
Drugs			95	4
Depression			87	14
Self-harm			96	5
Anxiety			91	11
Self-esteem			87	14
Bullying			96	5
Anorexia/bulimia			96	5
Psychological trauma			97	4
Sexual health/knowledge			97	4
Paranoid thinking			99	2
Money			91	10
Neighbourhood			91	10
Leisure activities			91	10
Violence to others			87	14
Defiance, making trouble			94	7

Correlation between the RU–OK questionnaire subscales, the MFQ and SDQ total and sub-scale scores are shown in Table 2. This shows modest correlations between scales. The MFQ, SDQ and RU–OK questionnaire emotional sub-scales and conduct sub-scales had better correlations as expected. We thus performed a standard statistical 2 × 2 table screening procedure of two areas (Warner, 2004), despite the small sample size. A cut-off point of 4 on the SDQ for emotional and conduct subscales was used. Using the SDQ emotional sub-scale as the gold standard, the RU–OK ‘depression’ sub-scale gave a sensitivity of 17.4% and specificity of 97%, with a positive predictive value of 80% and a negative predictive value of 70%. Confidence intervals were wide. Comparison with the MFQ showed much lower sensitivity and specificity. When the RU–OK ‘violence to others’ problem area was compared with the conduct sub-scale of the SDQ, the sensitivity was 48%, specificity 94%, positive predictive value 77% and negative predictive value 80%.

Table 2. Correlations between MFQ, SDQ and RU–OK sub-scales

	SDQ sub-scales and total score n=105						MFQ n=101	RU—OK sub-scales and total score n=69
	Hyperactivity	Conduct	Emotional	Prosocial	Peer problems	SDQ total score	MFQ total score	RU-OK total score	Emotional sub-scale	Violence sub-scale	Relationships sub-scale	Drugs/alcohol
SDQ
Hyperactivity	1
Conduct	0.565**	1
Emotional	0.283**	0.170	1
Prosocial	-0.293**	-0.337**	0.090	1
Peer problems	0.102	0.074	0.415**	0.034	1
Total score	0.770**	0.672**	0.694**	-0.197*	0.543**	1
MFQ total score	0.337**	0.231*	0.584**	0.110	0.347**	0.557**	1
RU—OK
Total score	0.196	0.148	0.275*	-0.043	0.442**	0.387**	0.373**	1
Emotional	0.131	0.015	0.420**	0.016	0.534**	0.403**	0.537**	0.814**	1
Violence	0.329**	0.337**	0.051	-0.310**	0.157	0.326**	0.188	0.690**	0.422**	1
Relationship	0.107	0.057	0.173	0.058	0.451**	0.281*	0.278*	0.822**	0.575**	0.469**	1
Drug and alcohol	0.313**	-0.280*	0.017	-0.134	-0.067	0.215	0.180	0.260**	0.012	0.222*	0.218*	1

The summary results of the questionnaire are shown in Table 3. The 5-point scales were reduced to two categories as shown. Excellent, good and OK were combined into ‘positive’, and poor and bad into ‘negative’. Table 3 shows that 70–80% of respondents were positive about the site. There were no differences between the main-stream school and the specialist hospital school.

Table 3. Summary results of the questionnaire evaluating the ru-ok website

	Positive view		Negative view		Total n
Evaluation of site	n	%	n	%
Navigation of site	59	88.1	8	11.9	67
Self-assessment area	52	78.8	14	21.2	66
Your overall opinion of site	58	86.6	9	13.4	67
Graphics/pictures on site	56	87.5	8	12.5	64
Customising the site	57	85.1	10	14.9	67
Games	50	76.9	15	23.1	65
Interactive stories	54	83.1	11	16.9	65

We analysed website traffic; in the 20 months since the site went live there were 18 382 visits: 2956 people had logged on to the site and 640 completed the RU-OK questionnaire. Of activity on the site, 23% was dedicated to downloading advice and information files and 12% to looking at the cartoons about depression, eating disorders, school problems and pregnancy. On average, there are about 900 visitors per month, of which 150 logged on to the secure section of the site containing the online questionnaire. There continues to be a steady increase in activity; the number of visitors to the site is now about 730 per week.

Discussion

This study shows that the ru-ok.com website was positively evaluated by a sample of representative teenagers aged 13–14 years. Website traffic indicates a steady rise in use, as more teenagers become aware of the existence of the site. The self-assessment questionnaire shows acceptable correlations with established paper-and-pencil screening questionnaires.

Adolescent mental health problems in school are increasingly recognised and questionnaires can be used to screen for these (Stansfeld et al, 2004). The advantage of the RU–OK questionnaire is that it is based on a needs assessment methodology; the paper version (Kroll et al, 1999) has good face validity and acceptability with teenagers. The RU–OK questionnaire does not require any summary analysis to interpret the results or to obtain downloadable advice and information about interventions. Some questionnaires, however, can be misused and misunderstood (Clark & Harrington 1999; Warner, 2004). Many also require scoring and interpretation.

Shortcomings of this study include the limited age range of teenagers studied and the small sample size. Thus, formal screening analysis and receiver operator curve analysis was not possible. Ideally, the RU–OK questionnaire should be compared with a gold standard diagnostic interview and other observer-rated methods using a larger sample of teenagers.

Schools face an increasing challenge of first identifying pupils with problems and then delivering acceptable interventions at an appropriate tiered level. Stepped care models are being proposed for certain conditions (National Institute for Clinical Excellence, 2004), and it would be useful to study further the role of screening questionnaires and self-assessment in schools who use a stepped care approach.

Declaration of interest

L.K. contributes to the management and development of the ru-ok.com website.