It is important to preface this piece by advising the reader that the
author is not writing from the point of view of a statistician, but rather
that of a user of reliable change. The author was invited to comment
following the publication of an original inquiry concerning Reliable Change
Index (RCI) formulae (Hinton-Bayre, 2000) and
after acting as a reviewer for the current Maassen paper (this issue, pp. 888–893). Having been a bystander
in the development of various RCI methods, this comment serves to represent
the struggle of a non-statistician to understand the relevant statistical
issues and apply them to clinical decisions. When I first stumbled across
the ‘classical’ RCI attributed to Jacobson and Truax (1991) (Maassen, this issue,
Equation 4), I was quite excited and immediately applied the formula to my
own data (Hinton-Bayre et al., 1999). Later, upon
reading the Temkin et al. (1999) paper I commented
on what seemed to be an inconsistency in their calculation of the error term
(Hinton-Bayre, 2000). My “confusion”
as Maassen suggests was derived from the fact that I noted the error term
used was based on the standard deviation of the difference scores
(Maassen, Expression 5*) rather than the Jacobson and Truax formula
(Maassen, Expression 4). This apparent anomaly was subsequently
addressed when Temkin et al. (2000) explained
they had employed the error term proposed by Christensen and Mendoza
(1986) (Maassen, Expression 5). My concern
with the Maassen manuscript was that it initially appeared two separate
values could be derived through using expressions 5 and 5* using the
Temkin et al. (1999) data. This suggested
there might be four (expressions 4, 5, 5*, and 6), rather than three,
ways to calculate the reliable change error term based on a null
hypothesis model. Once again I was confused. Only very recently did I
discover that expressions 5 and 5* yield identical results when applied
to the same data set (N.R. Temkin, personal communication) and when
estimated variances are used (G. Maassen, personal communication). The
reason for expressions 5 and 5* yielding slightly different error term
values using the Temkin et al. (1999) data
was due to use of nonidentical samples for parameter estimation. The
use of non-identical samples came to light in the review process of the
present Maassen paper—which Maassen now indicates in an
author's note. Thus there were indeed only three approaches to
consider (Expressions 4, 5, & 6). Nonetheless, Maassen maintains
(personal communication) that Expression 5, as elaborated by
Christensen and Mendoza (1986), represents
random errors comprising the error distribution of a given person,
whereas Expression 5* refers to the error distribution of a given
sample. While it seems clear on the surface that the expressions
represent separate statistical entities, it remains unclear to the
present author how these expressions can then yield identical values
when applied to test–retest data derived from a single normative
group. Unfortunately however, my confusion does not stop there.