Criteria of validity in experimental psychopathology: application to models of anxiety and depression

Abstract

The modeling of abnormal behavior in ‘normal’ subjects (often animals) has a long history in pharmacological research for the screening of novel drug compounds. Systematic criteria have been outlined in that literature to estimate the external validity of a model, that is to estimate how closely the model is linked to the disorder of interest. Experimental psychopathology (EPP) also uses behavioral models to study the psychological processes that underlie abnormal behavior. Although EPP researchers may occasionally feel uneasy about the validity of the model that they use, the issue has not received direct attention in this literature. Here, we review the criteria of validity as set out in pharmacology research (face, predictive and construct validity) and discuss their relevance for EPP research. Furthermore, we propose diagnostic validity as an additional criterion of external validity that is relevant to EPP research. We evaluate two models for the study of anxiety and depression, and show that they have good face, diagnostic and construct validity. However, EPP research generally lacks direct tests of predictive validity. We conclude that combined evaluations of predictive, diagnostic and construct validity provide a sound basis to infer the external validity of behavioral models in EPP research.

Introduction

Researchers in the domain of experimental psychopathology (EPP) use the experimental methodology to study psychological processes that underlie abnormal behavior. One important line of research uses experimental models aimed at mimicking abnormal processes in healthy individuals, with the aim of studying these processes in a highly controlled way. This is ideal from an experimental perspective, but has the disadvantage of a translational gap with the clinical case. The success of this translation depends on the external validity of the model. The crucial question is: ‘To what extent do the results from the model inform us about the disorder of interest?’

External validity has received a lot of attention in the pharmacological literature because of the long history of using animal subjects to model human disorders (for the screening of new drugs)1†. Most authors identify three criteria (face, predictive and construct validity), although there is still debate about their relative importance. We discuss these criteria and add one that is particularly relevant to EPP research (diagnostic validity).

Criteria of external validity

Face validity

Face validity represents the most straightforward criterion, as it refers to the degree of phenomenological similarity between the behavior in the model and the symptoms of the disorder (Sarter & Bruno, 2002). However, face validity is probably the weakest criterion. Similar behaviors can have different functions across species/groups whereas different behaviors may have similar functions. In addition, disorders often cannot be modeled in their entirety, and some symptoms are extremely difficult to model directly (e.g. hallucinations in schizophrenia). Face validity is neither a necessary nor a sufficient criterion of external validity.

Predictive validity

Predictive validity means that performance in the model predicts performance in the disorder (Joel, 2006). In pharmacological research, the aim is often to predict how symptoms of a disorder will react to a novel drug compound. Predictive validity is based on testing the behavior in the model with drugs that have been shown to effectively treat the disorder. The behavior in the model should react to the drug in a similar way. This criterion is more stringent than face validity. For example, a rat model of hallucinations may lack face validity but react strongly to drugs that are known to reduce hallucinations in schizophrenia (D₂ antagonists; Nestler & Hyman, 2010). However, the model may also react to known treatments through a mechanism that is not related to the disorder. In such cases, the model will fail to promote further developments in the field and the validity will remain restricted to the specific drugs that originally supported the usefulness of the model (Sarter & Bruno, 2002). Predictive validity is therefore a necessary but not a sufficient criterion.

It is important to note that applying a treatment to a model often requires adaptation of the treatment to the boundaries of the model. For example, drugs are often administered chronically to patients but acutely in the rat model (Nestler & Hyman, 2010). Similarly, an EPP researcher will have to select the working ingredients from a psychotherapeutic treatment and adapt these to the model. Testing the predictive validity of a model requires a theory of the applied treatment.

Construct validity

Construct validity refers to the disease relevance of the methods by which the model is constructed, with a focus on recreating the etiological process in the model (Nestler & Hyman, 2010). This requires an elaborated (etiological) theory of the disorder and of the model, and theoretical reasons to assume that the process in the model parallels the clinical process of interest. For example, prepulse inhibition tests of attention are not directly related to hallucinations (weak face validity), but they respond well to known drug treatments (D₂ antagonists; strong predictive validity) and there is an elaborated theory that links attentional dysfunctions to the production of hallucinations (Lubow, 2005; strong construct validity).

Diagnostic validity

The use of human participants in EPP research provides the unique possibility of testing patients in the model as well. To the extent that the behaviors differ from healthy individuals (in intensity or frequency), the model has diagnostic validity and can be used as a diagnostic marker. Diagnostic validity adds to external validity because it shows that the model taps into processes that are unique to patients. However, behavioral differences are not necessarily crucial to the disorder (but, for example, reflect sociodemographic differences). Diagnostic validity supports external validity, but it is not a sufficient criterion in itself.

In conclusion, the validity of an experimental model is determined primarily by its predictive, diagnostic and construct validity. Testing the model with established treatments and with patients can evaluate the first two empirically. Construct validity has to be evaluated on theoretical grounds. Face validity is generally not important. Pharmacological models are often well tested for predictive validity, but construct validity receives much less attention (Sarter & Bruno, 2002). By contrast, EPP models are often built on elaborated psychological theories (strong construct validity), but predictive validity has hardly been tested. In the following sections we illustrate the validation process further by applying the criteria set to two EPP models of anxiety and depression.

Application to the fear-conditioning model of anxiety

An experience of panic inside an elevator may produce fear of elevators, based on the formation of an association between elevators and panic attacks. This is modeled in fear conditioning, where a neutral stimulus (e.g. a tone) is followed by a threatening stimulus (e.g. an electrical shock). This typically produces fear of the tone (increased heartbeat, sweat gland activity, startle ability, avoidance), akin to clinical anxiety symptoms. These observations add to the face validity of the model.

The predictive validity is relatively well supported. Exposure treatments reduce clinical anxiety by exposing a patient to their phobic stimulus. Similarly, exposures to a fear-conditioned stimulus reduce fear reactions (extinction; Bouton, 2002). In addition, the human fear-conditioning model is sensitive to anxiolytic drugs, but the effects vary across procedures and drug compounds (Grillon, 2008). More systematic studies are needed.

The model has strong construct validity. The theory of the disorder and the model converge (the formation of an association between a neutral and a threatening stimulus). Furthermore, fear conditioning recruits a similar neurocircuitry as in phobic patients. Most importantly, fear-conditioned stimuli increase amygdala activity in a similar way as phobic stimuli (Sehlmeyer et al. 2009).

Finally, the fear-conditioning model has good diagnostic validity. A meta-analysis indicated that anxiety patients display a deviant pattern of fear reactions in the model, especially during periods of safety (Lissek et al. 2005). This suggests specific deficits in safety learning. Hence, testing diagnostic validity not only evaluates the model but also informs the theory of the disorder.

Application to the cognitive reactivity (CR) model of depression

CR refers to the degree to which an (experimentally induced) sad mood triggers negative thinking patterns as observed in clinical depression (Scher et al. 2005). This constitutes the face validity of the model.

The predictive validity of the CR model has not been tested systematically. Kuehner et al. (2009) found that student participants induced to ruminate (modeling depressotypic thinking) showed a significant increase in CR, whereas participants experimentally induced into a mindful self-focus (modeling an evidence-based intervention technique) showed a marginally significant decrease in CR. Further testing of depression treatments in the CR model is warranted.

The CR model has strong construct validity. First, the reinstatement of negative thinking patterns is a conceptual analog to the (cognitive) theory of depression, which precisely states that a depressive episode is triggered when a mild sad mood state reactivates negative thinking in vulnerable individuals. Second, CR is associated with biological markers of depression vulnerability (Booij & van der Does, 2007) and genes that are implicated in the pathophysiology of depression (Antypa & van der Does, 2010). Third, Raes et al. (2009) found that an evidence-based treatment to reduce depression relapse also reduced self-reported CR in a mixed sample including recovered depressed individuals and non-clinically depressed individuals (compared to a matched control group). This supports the construct validity of the CR model by showing that CR is indeed an important aspect of depression2.

Finally, the CR model has strong diagnostic validity; formerly depressed patients show elevated CR levels in this model compared to never depressed controls (Miranda et al. 1998). High CR is also predictive of later relapse (Segal et al. 2006).

Conclusions

The relevance of an EPP model depends on its external validity. We have shown that this is based primarily on two empirical criteria (predictive and diagnostic validity) and a theoretical criterion (construct validity). Face validity is generally not important. Whereas most EPP models have strong construct validity (and often diagnostic validity), tests of predictive validity are generally lacking. Systematic validation of EPP models will structure the array of models in this field, stimulate further developments in models and theories of psychopathology, and facilitate screenings of novel intervention techniques. The validation focus may catalyze the transformation of theoretical innovations into practical innovations.