Adjusted Residuals for Evaluating Conditional Independence in IRT Models for Multistage Adaptive Testing

Abstract

The key assumption of conditional independence of item responses given latent ability in item response theory (IRT) models is addressed for multistage adaptive testing (MST) designs. Routing decisions in MST designs can cause patterns in the data that are not accounted for by the IRT model. This phenomenon relates to quasi-independence in log-linear models for incomplete contingency tables and impacts certain types of statistical inference based on assumptions on observed and missing data. We demonstrate that generalized residuals for item pair frequencies under IRT models as discussed by Haberman and Sinharay (J Am Stat Assoc 108:1435–1444, 2013. https://doi.org/10.1080/01621459.2013.835660) are inappropriate for MST data without adjustments. The adjustments are dependent on the MST design, and can quickly become nontrivial as the complexity of the routing increases. However, the adjusted residuals are found to have satisfactory Type I errors in a simulation and illustrated by an application to real MST data from the Programme for International Student Assessment (PISA). Implications and suggestions for statistical inference with MST designs are discussed.

Multistage adaptive testing (MST) is becoming increasingly popular with major educational assessment programs currently operating in this fashion (Robin et al., 2014; Yamamoto et al., 2019). MST consists of a modular approach to adapt the difficulty level of the test to the ability level of test takers in which the possible test paths are limited in number and can be reviewed in advance. Item response theory (IRT) models are often behind MST applications. One of the key pillars of IRT is the assumption of independence of item responses conditional on the latent ability variable, often referred to as local independence (Lord & Novick, 1968, Section 16.3). Evaluating this conditional independence (CI) assumption for MST data is more complicated than with data from a linear test due to dependencies in the incomplete data that arise from MST designs. However, it remains important to evaluate the CI assumption in the context of MST, because violations can lead to inflated measurement precision (Wainer & Thissen, 1996), confounds in subscores (Yen, 1993), and issues with the adaptive algorithm (Zenisky et al., 2001). In this paper, we demonstrate that existing methods for evaluating CI can fail for MST data and present appropriate adjustments for one particular method based on generalized residuals (Haberman & Sinharay, 2013).

Incomplete designs in which test takers only see a subset of all available items play an important role in large-scale educational assessments for various reasons such as test security in the case of continuous testing programs (Kolen & Brennan, 2004) and domain coverage in the case of educational surveys with limited testing time (Johnson, 1992). Assuming dichotomous responses to J items, a $2^J$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$2^J$$\end{document} -contingency table is obtained (Tjur, 1982). Incomplete designs lead to what is called an incomplete contingency table in log-linear modeling (Bishop, Fienberg, & Holland, 2007, Chapter 5). That is, the table has structural zeros. However, for MST designs, the routing decisions can cause specific deviations for certain cells of the contingency table. In addition to structural zeros, not all item-response patterns are equally likely due to the specifics of the MST design (e.g., a set of difficult items is presented only when the sum score on a set of earlier items exceeds a specific value). The extent to which these deviations arise depends on the interplay between the routing decisions, the difficulty level of the MST modules, and the ability level of test takers. In terms of log-linear modeling, this issue is referred to as quasi-independence (Goodman, 1968), which means that an independence model can only be specified for a subset of the outcome space. In case of MST data, as we shall see later, this subset is not always straightforward, which complicates the evaluation of CI for IRT models.

When it comes to statistical inference using observed and missing data from MST designs with respect to model parameters and model fit, it is important to distinguish between likelihood and sampling-distribution inference (Rubin, 1976).1 Likelihood inference is based on (ratios of) the likelihood function for different parameter values and sampling-distribution inference on comparing observed values of a statistic with its sampling distribution under the null hypothesis. With missing data caused by incomplete designs, inference can proceed by making use of the ignorability principle (Rubin, 1976). For sampling-distribution inference, missing data is ignorable if missing data are missing at random (MAR) and observed data are observed at random (OAR), which together is referred to as missing completely at random (MCAR; Rubin, 1976). For likelihood inference, missing data is ignorable, if MAR holds and the missing-data parameter is distinct from the observed-data parameter.

As an example of likelihood inference with MST data, Glas (1988) showed that solvable equations can be obtained for maximum marginal likelihood estimation of item parameters for the Rasch and two-parameter logistic (2PL) models in MST designs. Furthermore, Mislevy and Wu (1996, Theorem 5.2) showed that the missing data caused by adaptive testing designs is MAR, but not MCAR. However, in contrast to Glas (1988), they assume that item parameters are known and consider inference on ability parameters only. Either way, MST and other adaptive testing designs can lead to issues with evaluating the assumption of CI (Mislevy & Chang, 2000) and dimensionality assessment (Zhang, 2013) when sampling-distribution inference is used. Therefore, it is critical to evaluate the impact of MST data on statistical methods and properly adjust such methods to account for the adaptive design to provide accurate results (Ali et al., in press).

Notwithstanding these issues, common methods for evaluating CI in IRT models such as residual correlations (Yen, 1993) and Pearson’s $X^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X^2$$\end{document} statistic for item pairs (Chen & Thissen, 1997) have been used in the context of adaptive testing (Pommerich & Segall, 2008). It is not the purpose of this paper to compare several different conditional dependence diagnostics. Liu & Maydeu-Olivares (2013) provide an overview of methods for evaluating CI, including limited-information methods based on item pairs. However, it is generally unknown what the impact of MST is on such methods and what needs to be done to take the MST design into account. The purpose of the present paper is to address these issues focusing on residual analysis (Haberman & Sinharay, 2013; Reiser, 1996) as this has worked well in terms of Type I error (van Rijn et al., 2016). In the context of evaluating CI, this analysis can be performed by evaluating residuals for frequencies of item pairs or triplets. We demonstrate that not accounting for the MST design generally leads to incorrect residuals and provide appropriate adjustments that take into account the MST design.

In general, item parameters are needed to create MST designs and report test results. In this context, a distinction can be made between pre-calibrated and post-calibrated MSTs (Jewsbury & van Rijn, 2020). In a pre-calibrated MST, item parameters are estimated from an earlier data collection and treated as fixed in the given MST data. This earlier data collection can be a pilot study specifically designed to obtain item parameters (e.g., a linear pre-test) or an earlier administration of the studied MST (e.g., with seeded new items). In a post-calibrated MST, item parameters are (re-)estimated from the given data set. We mostly focus on the latter situation in which it becomes important to discuss MST design aspects that drive the tradeoff between information in the data for estimating item parameters and that for estimating ability parameters (Zwitser & Maris, 2015). That is, MST designs that work well for ability estimation, may not work well for item parameter estimation.

The paper is outlined as follows. In the Method section, we discuss CI, IRT models, estimation, and adjusted residuals. We illustrate the proposed adjustments by means of a simulation study in which parameter recovery is also briefly addressed. Finally, the methods are applied to real MST data from the Programme for International Student Assessment (PISA). The paper ends with a discussion.

1. Method

We denote the observed item response variable by $x_{\mathrm{ij}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_{ij}$$\end{document} for test takers $i=1,\dots ,N$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i=1,\ldots ,N$$\end{document} and items $j=1,\dots ,J$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$j=1,\ldots ,J$$\end{document} . The test length is denoted by L with $L\le J$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L\le J$$\end{document} . Let S be the set of possible item response vectors $x_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\textbf{x}}_i$$\end{document} , i.e., the sample space.

In this paper, we make comparisons between two types of linear designs and two types of MST designs. The first linear design is a complete design in which test takers see all items (i.e., $L=J$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L=J$$\end{document} ). The second linear design is a random design in which test takers see a random subset of all items (i.e., $L<J$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L<J$$\end{document} ).

The first MST design is a basic two-stage design in which test takers start with a routing module, which is followed by an adaptive module of either low or high difficulty, depending on a test-taker’s performance on the routing module. Since test takers are routed to only one of the two adaptive modules, we have $L<J$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L<J$$\end{document} . In the basic MST design, the routing and adaptive modules are fixed. The following example of a basic MST design consisting of three modules A, B, and C will be used throughout to illustrate our methods. Figure 1 illustrates this design. We assume that module A is of medium difficulty, module B is of low difficulty, and module C is of high difficulty. Test takers start with module A. If their sum score on module A ( $x_A^{+}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_A^+$$\end{document} ) is below the cutoff $c_A$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c_A$$\end{document} , they are routed to module B. If their sum score on module A is at least $c_A$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c_A$$\end{document} , they are routed to module C. The basic MST design can be generalized to (1) more than two stages, (2) more than two difficulty levels beyond the first stage, and (3) more complicated routing rules.

Figure. 1 Basic MST design with two stages and two levels of difficulty (module B is of lower difficulty, module C is of higher difficulty).

The second MST design is a balanced two-stage design. In this design, again a routing module is followed by an adaptive module, but we let each item appear in both stages by creating multiple routing modules and multiple adaptive modules. This design is intended to mimic balanced incomplete block designs used in linear testing (Messick et al., 1983). If all modules are to have the same length, then three routing modules, three low-difficulty modules, and three high-difficulty modules can be assembled to create a balanced MST design from the above basic MST design. Essentially, the basic MST design is replicated three times using different subsets from the item pool to create a balanced MST design. The design is illustrated in Fig. 2. Here, all items are used once across modules A, D, and G, but also once across A, B, C, and D, E, F, and G, H, I. Note that there are different ways to create a balanced two-stage design. For example, by splitting the item pool randomly in parts and creating modules from each part.

Figure. 2 Balanced MST design with two stages, two levels of difficulty, and all items used in both stages.

Different designs can lead to different sample spaces, which is important to realize because this can lead to different sampling distributions as well. Consider the special case where each module consists of a single dichotomous item and the cutoff $c_A=1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c_A=1$$\end{document} . For the basic MST design, the sample space then only has four elements: $S_{\text{basic}}=(00\text{M},01\text{M},1\text{M}0,1\text{M}1)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S_\text {basic}=(00\text {M},01\text {M},1\text {M}0,1\text {M}1)$$\end{document} , where 0 indicates an incorrect response, 1 indicates a correct response, and M indicates missing by design. So, the pattern 00M indicates that items A and B were answered incorrectly and, by design, the response to item C is missing. It follows that if item B is observed, then item A must be incorrect, and if item C is observed, then item A must be correct. Alternatively, if we employ the balanced MST design by allowing each item to be the first for 1/3 of the test takers and, depending on the response, one of the remaining items to be second, the sample space has nine elements: $S_{\text{balanced}}=(00\text{M},01\text{M},1\text{M}0,1\text{M}1,10\text{M},\text{M}10,\text{M}11,\text{M00},0\text{M}1)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S_\text {balanced}=(00\text {M},01\text {M},1\text {M}0,1\text {M}1,10\text {M},\text {M}10,\text {M}11,\text {M00},0\text {M}1)$$\end{document} . Finally, if two out of the three items are administered at random, the sample space $S_{\text{random}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S_\text {random}$$\end{document} consists of all twelve possible response patterns. For these three examples, the sample spaces are related as follows: $S_{\text{basic}}\subset S_{\text{balanced}}\subset S_{\text{random}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S_\text {basic} \subset S_\text {balanced} \subset S_\text {random}$$\end{document} . If each module consists of multiple items instead of a single item, the dependencies between items across modules become more complex. For example, in the basic MST with multiple items per module, if module B is observed, then it follows that $x_A^{+}<c_A$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_A^+ < c_A$$\end{document} , and if module C is observed, $x_A^{+}\ge c_A$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_A^+ \ge c_A$$\end{document} .

We next discuss how MST designs affect the assumption of CI. This is followed by a brief exposition of IRT models and their estimation. We then discuss residuals for evaluation of CI and present the proper adjustments for MST designs.

1.1. Conditional Independence

In general, for test taker i, a vector of J item responses $x_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\textbf{x}}_i$$\end{document} and a latent ability variable ${\theta }_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{{\theta }}}_i$$\end{document} , the assumption of CI amounts to being able to write the joint probability as

(1) $\begin{array}{c}p\left(x_i\right|{\theta }_i)=\prod _{j=1}^{J}p\left(x_{\mathrm{ij}}\right|{\theta }_i),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} p({\textbf{x}}_i|{\varvec{{\theta }}}_i)=\prod _{j=1}^J p(x_{ij}|{\varvec{{\theta }}}_i), \end{aligned}$$\end{document}

with ${\prod }_{j=1}^J(m_j+1)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\prod _{j=1}^J (m_j + 1)$$\end{document} possible item response vectors in S if item j is scored $0,1,\dots ,m_j$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0,1,\ldots ,m_j$$\end{document} , where $m_j$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$m_j$$\end{document} is the maximum score for item j. We assume here that there are no missing values and no design restrictions on the sample space. The above notion is typically referred to as strong CI, whereas weak CI only states that item pairs are conditionally independent (McDonald, 1999).

In the context of MST, the conditional probability in Eq. 1 needs to be rewritten to take into account routing rules (Haberman & von Davier, 2014, Section 15.2.2). To this end, we let $F_{\mathrm{iH}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F_{iH}$$\end{document} be the set of items administered to test taker i up to stage H and $G_H$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$G_H$$\end{document} be the set of all available items in these stages with $F_{\mathrm{iH}}\subset G_H$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F_{iH} \subset G_H$$\end{document} . According to Haberman & von Davier (2014), CI then boils down to

(2) $\begin{array}{cc}p\left(x_{i{G}_H}\right|{\theta }_i)& =\prod _{j\in F_{\mathrm{iH}}}p\left(x_{\mathrm{ij}}\right|{\theta }_i),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} p({\textbf{x}}_{iG_H}|{\varvec{{\theta }}}_i)&=\prod _{j \in F_{iH}} p(x_{ij}|{\varvec{{\theta }}}_i), \end{aligned}$$\end{document}

which essentially means that CI can only be assumed to hold for items that were observed.

As an illustration consider the basic MST design example described above where $G_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$G_2$$\end{document} consists of the items for modules A, B, and C and $F_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F_2$$\end{document} consists of either modules A and B or A and C. Let’s assume further that the latent variable is unidimensional. In this design, CI conditional on $\theta$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta $$\end{document} across modules B and C cannot be determined. In this case, CI can only be assessed for the sample space, which relates to quasi-independence in log-linear models with incomplete data (Goodman, 1968). However, taking into account the missing data does not fully capture the basic MST design because the responses on module A and the cutoff $c_A$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c_A$$\end{document} have an impact on either observing module B or C. This needs to be taken into account as well when one wants to evaluate CI. To this end, we need to find the probabilities for the item responses of module A conditional on the routing decisions (i.e., the sum score for module A), which are

(3) $\begin{array}{cc}p\left(x_A\right|\theta ,x_{A+}<c_A)& =\frac{p\left(x_A\right|\theta )}{p(x_{A+}<c_A|\theta )},\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} p({\textbf{x}}_{A}|\theta ,x_{A+}< c_A)&= \frac{p({\textbf{x}}_{A}|\theta )}{p(x_{A+} < c_A|\theta )}, \end{aligned}$$\end{document}

(4) $\begin{array}{cc}p\left(x_A\right|\theta ,x_{A+}\ge c_A)& =\frac{p\left(x_A\right|\theta )}{p(x_{A+}\ge c_A|\theta )},\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} p({\textbf{x}}_{A}|\theta ,x_{A+}\ge c_A)&= \frac{p({\textbf{x}}_{A}|\theta )}{p(x_{A+}\ge c_A|\theta )}, \end{aligned}$$\end{document}

where $x_A$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\textbf{x}}_A$$\end{document} denotes the vector of item responses for module A. These probabilities are needed in evaluating CI for item pairs that do not appear within the same module (i.e., they are separated by a routing decision). In MST designs, CI in the sense of a simple product without conditioning on the routing decision only holds for item pairs within the same module. Furthermore, it can be shown that for the basic MST design for items j and k

$\begin{array}{cc}p(x_{A_j},x_{A_k}|\theta )& =p\left(x_{A_j}\right|\theta )p\left(x_{A_k}\right|\theta ),\\ p(x_{B_j},x_{B_k}|\theta )& =p\left(x_{B_j}\right|\theta )p\left(x_{B_k}\right|\theta ),\\ p(x_{C_j},x_{C_k}|\theta )& =p\left(x_{C_j}\right|\theta )p\left(x_{C_k}\right|\theta ),\\ p(x_{A_j},x_{B_k}|\theta )& \ne p\left(x_{A_j}\right|\theta )p\left(x_{B_k}\right|\theta ),\\ p(x_{A_j},x_{C_k}|\theta )& \ne p\left(x_{A_j}\right|\theta )p\left(x_{C_k}\right|\theta ),\\ p(x_{B_j},x_{C_k}|\theta )& =0.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} p(x_{A_j},x_{A_k}|\theta )&= p(x_{A_j}|\theta )p(x_{A_k}|\theta ),\\ p(x_{B_j},x_{B_k}|\theta )&= p(x_{B_j}|\theta )p(x_{B_k}|\theta ),\\ p(x_{C_j},x_{C_k}|\theta )&= p(x_{C_j}|\theta )p(x_{C_k}|\theta ),\\ p(x_{A_j},x_{B_k}|\theta )&\ne p(x_{A_j}|\theta )p(x_{B_k}|\theta ),\\ p(x_{A_j},x_{C_k}|\theta )&\ne p(x_{A_j}|\theta )p(x_{C_k}|\theta ),\\ p(x_{B_j},x_{C_k}|\theta )&= 0. \end{aligned}$$\end{document}

1.2. Model

We now turn to discussing specific IRT models. A general logistic IRT model can be specified by

(5) $\begin{array}{c}p(X_{\mathrm{ij}}=x|\theta )=\frac{exp({\alpha }_{\mathrm{jx}}^{\prime }D{\theta }_i+{\beta }_{\mathrm{jx}})}{{\sum }_{h=0}^{m_j}exp({\alpha }_{\mathrm{jh}}^{\prime }D{\theta }_i+{\beta }_{\mathrm{jh}})},x=0,\dots ,m_j,\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} p(X_{ij}=x|{\varvec{{\theta }}})=\frac{\exp ({\varvec{{\alpha }}}_{jx}'{\textbf{D}}{\varvec{{\theta }}}_i+\beta _{jx})}{\sum _{h=0}^{m_j}\exp ({\varvec{{\alpha }}}_{jh}'{\textbf{D}}{\varvec{{\theta }}}_i+\beta _{jh})},\qquad x=0,\ldots ,m_j, \end{aligned}$$\end{document}

where $D$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\textbf{D}}$$\end{document} is a design matrix, ${\theta }_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{{\theta }}}_i$$\end{document} is a latent ability variable, ${\alpha }_{\mathrm{jx}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{{\alpha }}}_{jx}$$\end{document} contains item slopes, and ${\beta }_{\mathrm{jx}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{jx}$$\end{document} is an item intercept. The general form in Eq. 5 can accommodate polytomous IRT models such as the nominal response model (Bock, 1972), but also quite general multidimensional IRT models (Reckase, 2009). Further generalizations, such as the three-parameter logistic model, are possible when an additional layer of latent item response variables is introduced (Haberman, 2013). For a unidimensional two-parameter logistic (2PL) model, the assumption of CI leads to the following convenient form

(6) $\begin{array}{cc}logp\left(x_i\right|{\theta }_i)& =P_0({\theta }_i,\xi )+\sum _{j=1}^J{x}_j({\alpha }_j\theta +{\beta }_j),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \log p({\textbf{x}}_i|\theta _i)&= {\mathcal {P}}_0(\theta _i,{\varvec{{\xi }}})+\sum _{j=1}^{J}x_j(\alpha _j\theta +\beta _j), \end{aligned}$$\end{document}

where $P_0({\theta }_i,\xi )=log\left({\prod }_{j=1}^J,{[1+exp({\alpha }_j\theta +{\beta }_j)]}^{-1}\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {P}}_0(\theta _i,{\varvec{{\xi }}})=\log \left\{ \prod _{j=1}^{J} [1+\exp (\alpha _j\theta +\beta _j)]^{-1}\right\} $$\end{document} is a normalizing constant (Glas, 1989, Eq. 2.2.1) and $\xi$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{{\xi }}}$$\end{document} is the vector containing all item parameters.

Although our focus is on evaluating CI and not on modeling conditional dependence, we briefly discuss some approaches for the latter case. Random effects for conditional dependence can be expressed through, for example, testlet models (Wainer et al., 2007) or bifactor models (Gibbons & Hedeker, 1992). In general, a MIRT model for dichotomous items can be specified as

(7) $\begin{array}{cc}logp\left(x_i\right|{\theta }_i)& =P_0({\theta }_i,\xi )+\sum _{j=1}^J{x}_j({\alpha }_j^{\prime }D{\theta }_i+{\beta }_j).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \log p({\textbf{x}}_i|{\varvec{{\theta }}}_i)&= {\mathcal {P}}_0({\varvec{{\theta }}}_i,{\varvec{{\xi }}})+\sum _{j=1}^{J}x_j({\varvec{{\alpha }}}_j'{\textbf{D}}{\varvec{{\theta }}}_i+\beta _j). \end{aligned}$$\end{document}

Fixed effects for conditional dependence can be expressed through a log-linear approach. Kelderman & Rijkes (1994) and Verhelst & Verstralen (2008) discuss several models related to the Rasch family, including polytomous and multidimensional extensions. Another model that explicitly represents dependencies among items is the interaction model (Haberman, 2007), in which the probability of an item response vector is given by

(8) $\begin{array}{cc}logp\left(x_i\right|{\theta }_i)& =P_0({\theta }_i,\xi )+\sum _{j=1}^J{x}_j({\theta }_i+{\beta }_j)+\sum _{j=2}^J\sum _{k=1}^{j-1}{x}_j{x}_k({\gamma }_j+{\gamma }_k).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \log p({\textbf{x}}_i|\theta _i)&= {\mathcal {P}}_0(\theta _i,{\varvec{{\xi }}})+\sum _{j=1}^{J}x_j(\theta _i+\beta _j)+\sum _{j=2}^{J}\sum _{k=1}^{j-1}x_jx_k(\gamma _j + \gamma _k). \end{aligned}$$\end{document}

A salient feature of the interaction model is that CI does not hold, but the sum score is still a sufficient statistic for $\theta$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta $$\end{document} . Some other alternatives are discussed by Ip (2002) and Nikoloulopoulos & Joe (2015).

1.3. Estimation

As noted, item parameters are needed to create MST designs and generally such designs lead to incomplete data. Missing data in MST designs is MAR, but not OAR and, hence, not MCAR (Mislevy & Wu, 1996). This can be demonstrated in the basic MST design. Let $u$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\textbf{u}}$$\end{document} be the missing-data indicator. If module B is missing, then the sum score for module A must be at least $c_A$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c_A$$\end{document} , so that the joint probability of missing data on module B, $p\left(u_B\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p({\textbf{u}}_B)$$\end{document} , is independent of $x_B$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\textbf{x}}_B$$\end{document} and $x_C$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\textbf{x}}_C$$\end{document} , but not of $x_A$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\textbf{x}}_A$$\end{document} :

(9) $\begin{array}{cc}p\left(u_B\right|x_A,x_B,x_C)& =p\left(u_B\right|x_A)\ne p\left(u_B\right).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} p({\textbf{u}}_B|{\textbf{x}}_A,{\textbf{x}}_B,{\textbf{x}}_C)&=p({\textbf{u}}_B|{\textbf{x}}_A) \ne p({\textbf{u}}_B). \end{aligned}$$\end{document}

The first step in the above equation indicates that the data in module B is missing at random, whereas the second step indicates the data in module B is not observed at random.

Similar reasoning applies for module C in the basic MST design and for other MST designs. As noted, this result has implications for sampling-distribution inference, but not for parameter estimation if the marginal likelihood is used. That is, in this case, the ignorability principle applies for estimation of item and population parameters of unidimensional IRT models (Rubin, 1976; Eggen & Verhelst, 2011). The situation for multidimensional IRT models and MST data is somewhat more complex (see Jewsbury & van Rijn, 2020).

For marginal maximum likelihood (MML) estimation of item parameters with MST data, Glas (1988) showed that with sequentially administered items a, b, and c, their joint marginal probability can be written as:

(10) $\begin{array}{cc}p(x_a,x_b,x_c|\xi )& =p\left(x_a\right|{\xi }_a)p(x_b,x_c|x_a,\xi )\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} p(x_a,x_b,x_c|{\varvec{{\xi }}})&=p(x_a|{\varvec{{\xi }}}_a)p(x_b,x_c|x_a,{\varvec{{\xi }}}) \end{aligned}$$\end{document}

(11) $\begin{array}{cc}& =p\left(x_a\right|{\xi }_a)\int p(x_b,x_c|\theta ,\xi )f\left(\theta \right|x_a,{\xi }_a)d\theta \end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&=p(x_a|{\varvec{{\xi }}}_a)\int p(x_b,x_c|\theta ,{\varvec{{\xi }}})f(\theta |x_a,{\varvec{{\xi }}}_a)d\theta \end{aligned}$$\end{document}

(12) $\begin{array}{cc}& =\int p(x_a,x_b,x_c|\theta ,\xi )f\left(\theta \right)d\theta ,\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&=\int p(x_a,x_b,x_c|\theta ,{\varvec{{\xi }}})f(\theta )d\theta , \end{aligned}$$\end{document}

where $f\left(\theta \right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f(\theta )$$\end{document} is typically a normal distribution (Bock & Aitkin, 1981), although there are many other possibilities (Woods, 2015). This result shows that MST data poses no problems for MML estimation. For conditional maximum likelihood (CML) estimation of item parameters in the Rasch model with MST data, the situation is different because the sum score is used to obtain the conditional likelihood. Although we do not discuss CML further here, it is interesting to note that in order to apply CML to MST data, one essentially has to take into account the routing decisions (Zwitser & Maris, 2015).

We let q be a positive integer and $\Omega$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Omega $$\end{document} be the parameter space consisting of a nonempty open set of q-dimensional vectors. Then, MML estimation of the q-dimensional parameter vector $\xi$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{{\xi }}}$$\end{document} proceeds by defining the marginal log-likelihood function

(13) $\begin{array}{c}\ell \left(\xi \right)=\sum _{i=1}^N{\ell }_i\left(\xi \right),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \ell ({\varvec{{\xi }}}) = \sum _{i=1}^N \ell _i({\varvec{{\xi }}}), \end{aligned}$$\end{document}

where the individual contributions are given by

(14) $\begin{array}{c}{\ell }_i\left(\xi \right)=log\int p\left(x_i\right|\theta )f\left(\theta \right)d\theta .\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \ell _i({\varvec{{\xi }}}) = \log \int p({\textbf{x}}_i|\theta )f(\theta ) d\theta . \end{aligned}$$\end{document}

Estimates can be found by solving the likelihood equations $\partial \ell \left(\xi \right)/\partial \xi =0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\partial \ell ({\varvec{{\xi }}}) / \partial {\varvec{{\xi }}}=0$$\end{document} . The integrals can be solved by numerical integration methods (e.g., adaptive quadrature; Naylor & Smith, 1982).

1.4. Adjusted Residuals

We now discuss our method for evaluating CI and present adjustments to the original methods to account for MST design. The approach is based on generalized residuals for the frequencies of a pair of scores on items j and k as developed by Haberman & Sinharay (2013):

(15) $\begin{array}{c}{Z}_{x_j,x_k}=\frac{O_{x_j,x_k}-E_{x_j,x_k}}{S_{x_j,x_k}}.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Z_{x_j,x_k}=\frac{O_{x_j,x_k}-E_{x_j,x_k}}{S_{x_j,x_k}}. \end{aligned}$$\end{document}

where $O_{x_j,x_k}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$O_{x_j,x_k}$$\end{document} is the observed frequency of $x_j$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_j$$\end{document} and $x_k$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_k$$\end{document} , $E_{x_j,x_k}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_{x_j,x_k}$$\end{document} is the expected frequency, and $S_{x_j,x_k}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S_{x_j,x_k}$$\end{document} is the associated standard deviation. For example, $O_{1_j,1_k}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$O_{1_j,1_k}$$\end{document} denotes the observed frequency of a score of 1 on both items j and k. The expected frequency is typically given by

(16) $\begin{array}{c}{E}_{x_j,x_k}=\sum _{i=1}^N\int p(x_j,x_k|\theta )f\left(\theta \right|x_i)d\theta .\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} E_{x_j,x_k} = \sum _{i=1}^N \int p(x_j,x_k|\theta )f(\theta |{\textbf{x}}_i)d\theta . \end{aligned}$$\end{document}

However, for MST data, an adjustment is needed in order to account for patterns that either are unobserved or are more or less likely due to the MST design.

Consider again the basic MST design with modules A (medium difficulty), B (low difficulty), and C (high difficulty). If $A_j$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A_j$$\end{document} denotes the j-th item in module A and $c_A$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c_A$$\end{document} is the sum score cutoff on module A, then the following item probabilities given that the sum score takes on certain values are relevant

(17) $\begin{array}{cc}p\left(x_{A_j}\right|\theta ,x_A^{+}<c_A)& =\frac{p(x_{A_j},x_A^{+}<c_A|\theta )}{p(x_A^{+}<c_A|\theta )},\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} p(x_{A_j}|\theta ,x_A^+< c_A)&= \frac{p(x_{A_j},x_A^+< c_A|\theta )}{p(x_A^+ < c_A|\theta )},\end{aligned}$$\end{document}

(18) $\begin{array}{cc}p\left(x_{A_j}\right|\theta ,x_A^{+}\ge c_A)& =\frac{p(x_{A_j},x_A^{+}\ge c_A|\theta )}{p(x_A^{+}\ge c_A|\theta )},\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} p(x_{A_j}|\theta ,x_A^+ \ge c_A)&= \frac{p(x_{A_j},x_A^+ \ge c_A|\theta )}{p(x_A^+ \ge c_A|\theta )}, \end{aligned}$$\end{document}

where $p(x_A^{+}<c_A|\theta )={\sum }_{k=0}^{c_A-1}p(x_A^{+}=k|\theta )$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p(x_A^+ < c_A|\theta ) = \sum _{k=0} ^{c_A - 1} p(x_A^+ = k |\theta )$$\end{document} and $p(x_A^{+}\ge c_A|\theta )={\sum }_{k=c_A}^{J_A}p(x_A^{+}=k|\theta )$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p(x_A^+ \ge c_A|\theta ) = \sum _{k=c_A} ^{J_A} p(x_A^+ = k |\theta )$$\end{document} . The probabilities for individual sum scores are readily found by making use of the Lord-Wingersky recursions for computing the distribution of the sum score conditional on $\theta$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta $$\end{document} (Lord &Wingersky, 1984). The equations can be illustrated with a basic example. Consider the case of three dichotomous items and a cut score of two: The probability that the first item is incorrect given that the total score is smaller than two is: $(QQQ+QPQ+QQP)/(QQQ+PQQ+QPQ+QQP)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(QQQ + QPQ + QQP) / (QQQ+PQQ+QPQ+QQP)$$\end{document} with P and Q indicating probabilities for correct and incorrect item responses. The probability that the first item is correct given $x^{+}<2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x^+ < 2$$\end{document} is then $PQQ/(QQQ+PQQ+QPQ+QQP)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$PQQ / (QQQ+PQQ+QPQ+QQP)$$\end{document} . In addition, the probability that the first item is incorrect given $x^{+}\ge 2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x^+ \ge 2$$\end{document} is $\left(QPP\right)/(PPQ+PQP+QPP+PPP)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(QPP) / (PPQ+PQP+QPP+PPP)$$\end{document} and the probability that the first item is correct given $x^{+}\ge 2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x^+ \ge 2$$\end{document} is $(PPQ+PQP+PPP)/(PPQ+PQP+QPP+PPP)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(PPQ+PQP+PPP) / (PPQ+PQP+QPP+PPP)$$\end{document} . Although the equations are straightforward, some carefulness in their application is required since certain combinations of item and sum scores cannot occur (e.g., correct item response and sum score of zero).

For a pair with item j in module A and item k in module B, it then follows that the expected frequency is

(19) $\begin{array}{c}{E}_{x_{A_j},x_{B_k}}=\sum _{i\in N_{\mathrm{jk}}}{E}_{x_{i{A}_j},x_{i{B}_k}}=\sum _{i\in N_{\mathrm{jk}}}\int p\left(x_{A_j}\right|\theta ,x_{A+}<c_A)p\left(x_{B_k}\right|\theta )f\left(\theta \right|x_i)d\theta ,\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} E_{x_{A_j},x_{B_k}} = \sum _{i \in {\mathcal {N}}_{jk}} E_{x_{iA_j},x_{iB_k}}=\sum _{i \in {\mathcal {N}}_{jk}} \int p(x_{A_j}|\theta ,x_{A+}<c_A)p(x_{B_k}|\theta )f(\theta |{\textbf{x}}_i)d\theta , \end{aligned}$$\end{document}

For a pair with item j in module A and item k in module C, the expected frequency is

(20) $\begin{array}{c}{E}_{x_{A_j},x_{C_k}}=\sum _{i\in N_{\mathrm{jk}}}{E}_{x_{i{A}_j},x_{i{C}_k}}=\sum _{i\in N_{\mathrm{jk}}}\int p\left(x_{A_j}\right|\theta ,x_{A+}\ge c_A)p\left(x_{C_k}\right|\theta )f\left(\theta \right|x_i)d\theta ,\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} E_{x_{A_j},x_{C_k}} = \sum _{i \in {\mathcal {N}}_{jk}} E_{x_{iA_j},x_{iC_k}}= \sum _{i \in {\mathcal {N}}_{jk}} \int p(x_{A_j}|\theta ,x_{A+}\ge c_A)p(x_{C_k}|\theta )f(\theta |{\textbf{x}}_i)d\theta , \end{aligned}$$\end{document}

If routing is based on a different criterion than the sum score (e.g., ability estimate, maximum information, minimum expected posterior variance), similar conditioning principles would apply for taking into account the MST design. However, computations become more involved compared to conditioning on the sum score, which enables the use of the straightforward Lord-Wingersky recursions. For example, using ability estimates may require summarizing over all response patterns that lead to an ability estimate below or above the cut.

To ease the following presentation, we simplify notation by suppressing the particular item scores and modules, so that the residual for individual test taker i on item pair (j, k) can be written as

(21) $\begin{array}{cc}{r}_{\mathrm{ijk}}& =O_{x_{\mathrm{ij}},x_{\mathrm{ik}}}-E_{x_{\mathrm{ij}},x_{\mathrm{ik}}},\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} r_{ijk}&=O_{x_{ij},x_{ik}}-E_{x_{ij},x_{ik}}, \end{aligned}$$\end{document}

where $O_{x_{\mathrm{ij}},x_{\mathrm{ik}}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$O_{x_{ij},x_{ik}}$$\end{document} is the indicator for observing $x_{\mathrm{ij}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_{ij}$$\end{document} and $x_{\mathrm{ik}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_{ik}$$\end{document} (e.g., if both items are answered correctly, its value is 1). Then, the variance of the residual $r_{\mathrm{jk}}={\sum }_{i\in N_{\mathrm{jk}}}{r}_{\mathrm{ijk}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$r_{jk}=\sum _{i \in {\mathcal {N}}_{jk}} r_{ijk}$$\end{document} is found to be

(22) $\begin{array}{cc}V\left(r_{\mathrm{jk}}\right)& =\sum _{i\in N_{\mathrm{jk}}}{\left(r_{\mathrm{ijk}},-,d_{\mathrm{jk}}^{\prime },\nabla ,{\ell }_i,\left(\hat{\xi }\right)\right)}^2,\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} V(r_{jk})&= \sum _{i \in {\mathcal {N}}_{jk}} \left[ r_{ijk} - {\textbf{d}}_{jk}'\nabla \ell _i(\hat{{\varvec{{\xi }}}})\right] ^2, \end{aligned}$$\end{document}

where $\nabla {\ell }_i\left(\hat{\xi }\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\nabla \ell _i(\hat{{\varvec{{\xi }}}})$$\end{document} is the gradient contribution of test taker i and

(23) $\begin{array}{cc}{d}_{\mathrm{jk}}& =J{\left(\hat{\xi }\right)}^{-1}\sum _{i\in N_{\mathrm{jk}}}{r}_{\mathrm{ijk}}\nabla {\ell }_i\left(\hat{\xi }\right),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {\textbf{d}}_{jk}&= J(\hat{{\varvec{{\xi }}}})^{-1}\sum _{i \in {\mathcal {N}}_{jk}} r_{ijk}\nabla \ell _i(\hat{{\varvec{{\xi }}}}), \end{aligned}$$\end{document}

where $J\left(\hat{\xi }\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$J(\hat{{\varvec{{\xi }}}})$$\end{document} is the Fisher information matrix for model parameters (Louis, 1982). A general formulation of Eq. 22 is found in Equation 9 of Haberman & Sinharay (2013), but our version is more similar to Equation 46 of Haberman et al. (2013), although in a slightly different context. Except for centering the gradients, no further adjustments are needed for using the gradient and the Fisher information matrix, because these were obtained by correct application of the ignorability principle (Eggen & Verhelst, 2011). The generalized residual in Eq. 15 is then found by computing, using the simplified notation, $S_{\mathrm{jk}}=\sqrt{V\left(r_{\mathrm{jk}}\right)/N_{\mathrm{jk}}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S_{jk} = \sqrt{V(r_{jk})/N_{jk}}$$\end{document} , where $N_{\mathrm{jk}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{jk}$$\end{document} indicates the sample size for the item pair. Haberman & Sinharay (2013) showed that, under very general conditions, the generalized residual has an asymptotic standard normal distribution. In the context of large-scale assessments with balanced incomplete block designs, van Rijn et al. (2016) found that Type-I-error rates of these residuals for item pairs were close to the nominal level. Note that these residuals can be extended without too much difficulty to more than two items (e.g., item triples) and other IRT models.

With polytomous items, there can be sparse data for item pairs, even for relatively large samples (Joe & Maydeu-Olivares, 2010). In this case, a reduced residual for item pair j and k can be obtained as a product moment following Cai & Hansen (2013) by defining:

(24) $\begin{array}{c}{\tilde{r}}_{\mathrm{ijk}}=\sum _{h_j=1}^{m_j}\sum _{h_k=1}^{m_k}{h}_j{h}_k\left(O_{x_{i{h}_j},x_{i{h}_k}},-,E_{x_{i{h}_j},x_{i{h}_k}}\right).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {\tilde{r}}_{ijk}=\sum _{h_j=1}^{m_j}\sum _{h_k=1}^{m_k} h_jh_k\left(O_{x_{ih_j},x_{ih_k}}-E_{x_{ih_j},x_{ih_k}}\right) . \end{aligned}$$\end{document}

Note that the reduced residual is equal to $r_{\mathrm{ijk}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$r_{ijk}$$\end{document} if $m_j=m_k=1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$m_j=m_k=1$$\end{document} . The reduced generalized residual is then found by entering the above reduced residual into Eqs. 22 and 23. The variance of the reduced residual ${\tilde{r}}_{\mathrm{jk}}={\sum }_{i\in N_{\mathrm{jk}}}{\tilde{r}}_{\mathrm{ijk}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\tilde{r}}_{jk}=\sum _{i \in {\mathcal {N}}_{jk}} {\tilde{r}}_{ijk}$$\end{document} is then

(25) $\begin{array}{cc}V\left({\tilde{r}}_{\mathrm{jk}}\right)& =\sum _{i\in N_{\mathrm{jk}}}{\left({\tilde{r}}_{\mathrm{ijk}},-,{\tilde{d}}_{\mathrm{jk}}^{\prime },\nabla ,{\ell }_i,\left(\hat{\xi }\right)\right)}^2,\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} V({\tilde{r}}_{jk})&= \sum _{i \in {\mathcal {N}}_{jk}} \left[ {\tilde{r}}_{ijk} - \tilde{{\textbf{d}}}_{jk}'\nabla \ell _i(\hat{{\varvec{{\xi }}}})\right] ^2, \end{aligned}$$\end{document}

where

(26) $\begin{array}{cc}{\tilde{d}}_{\mathrm{jk}}& =J{\left(\hat{\xi }\right)}^{-1}\sum _{i\in N_{\mathrm{jk}}}{\tilde{r}}_{\mathrm{ijk}}\nabla {\ell }_i\left(\hat{\xi }\right).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \tilde{{\textbf{d}}}_{jk}&= J(\hat{{\varvec{{\xi }}}})^{-1}\sum _{i \in {\mathcal {N}}_{jk}} {\tilde{r}}_{ijk}\nabla \ell _i(\hat{{\varvec{{\xi }}}}). \end{aligned}$$\end{document}

Before illustrating the adjusted residuals, we briefly mention two other popular methods for evaluating conditional dependence that are affected when dealing with MST data. The first method is the check on the correlation between residual item scores for item pairs (j, k), commonly referred to as the $Q_3$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q_3$$\end{document} statistic (Yen, 1984). Although this method is known to have problems due to lack of a known null distribution (Chen & Thissen, 1997), it has been used succesfully in combination with a parametric bootstrap (Christensen et al., 2017). To compute the statistic, for each item, we need the following residual item score

(27) $\begin{array}{c}{r}_{\mathrm{ij}}=x_{\mathrm{ij}}-P\left(x_{\mathrm{ij}}\right),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} r_{ij} = x_{ij} - P(x_{ij}), \end{aligned}$$\end{document}

with

(28) $\begin{array}{c}P\left(x_{\mathrm{ij}}\right)=\int P\left(x_j\right|\theta )f\left(\theta \right|x_i)d\theta ,\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} P(x_{ij}) = \int P(x_{j}|{\varvec{{\theta }}})f({\varvec{{\theta }}}|{\textbf{x}}_i)d{\varvec{{\theta }}}, \end{aligned}$$\end{document}

as the expected item score. Then, the $Q_3$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q_3$$\end{document} statistic is simply the correlation between the vectors $r_j$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\textbf{r}}_j$$\end{document} and $r_k$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\textbf{r}}_k$$\end{document} . This version is slightly different from the original statistic because the posterior $f\left(\theta \right|x_i)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f({\varvec{{\theta }}}|{\textbf{x}}_i)$$\end{document} is used instead of a point estimate of $\theta$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{{\theta }}}$$\end{document} (e.g., maximum-likelihood estimate (MLE) or expected a posteriori (EAP)). For MST data, subvectors of the residual vector $r_j$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\textbf{r}}_j$$\end{document} or $r_k$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\textbf{r}}_k$$\end{document} may need to be computed in different ways depending on the associated routing rules and the particular item pair j and k.

The second method we mention is the limited information statistic $M_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M_2$$\end{document} developed by Maydeu-Olivares & Joe (2005), which is, for example, produced by commonly used IRT software such as flexMIRT (Houts & Cai, 2016) and the R package mirt (Chalmers, 2012). The statistic assesses overall model fit and is given by

(29) $\begin{array}{c}{M}_2=N{\left(p_2,-,{\pi }_2,\left(\hat{\xi }\right)\right)}^{\prime }{\hat{C}}_2\left(p_2,-,{\pi }_2,\left(\hat{\xi }\right)\right),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} M_2 = N\left[ {\textbf{p}}_2-{\varvec{{\pi }}}_2(\hat{{\varvec{{\xi }}}})\right] '\hat{{\textbf{C}}}_2\left[ {\textbf{p}}_2-{\varvec{{\pi }}}_2(\hat{{\varvec{{\xi }}}})\right] , \end{aligned}$$\end{document}

where $p_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\textbf{p}}_2$$\end{document} is the vector of observed proportions of first- and second-order marginals, ${\pi }_2\left(\hat{\xi }\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{{\pi }}}_2(\hat{{\varvec{{\xi }}}})$$\end{document} is the vector of expected proportions of first- and second-order marginals, and ${\hat{C}}_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{{\textbf{C}}}_2$$\end{document} is the associated covariance matrix (Maydeu-Olivares & Joe, 2005, Eqs. 12 and 13). Under the model, $M_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M_2$$\end{document} converges in distribution to a ${\chi }^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\chi ^2$$\end{document} -distribution with $s-q$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s-q$$\end{document} degrees of freedom, where s is the length of $p_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\textbf{p}}_2$$\end{document} and q is the number of model parameters. For MST data, the second-order marginals would need to be adjusted in a similar fashion as for the generalized residuals. One complication that may arise is that, even though $M_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M_2$$\end{document} focuses on first- and second-order marginals only, observed and expected proportions of full response patterns are required.

2. Illustrations

We now illustrate the adjusted residuals for item pairs to evaluate CI through a small simulation study and an application to PISA data.

2.1. Simulation

In the simulation study, we varied the sample size, the number of items, and the design. Sample sizes were 600 and 6000. The number of items were 9 (3 in each stage) and 45 (15 in each stage). Four different designs were used: (1) A complete design in which all test takers receive all items, (2) a random design in which test takers take 2/3 of all items (i.e., either 6 out of 9 items or 30 out of 45 items), (3) a basic two-stage design with two difficulty levels in the second stage (see Fig. 1), and 4) a balanced two-stage design with two levels in the second stage (see Fig. 2). The test length in the latter two MST designs is the same as in the random design. Only dichotomously scored items that followed a unidimensional 2PL model were used with ${\beta }_j\sim \text{N}(0.13,1.29)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _j \sim \text {N}(0.13,1.29)$$\end{document} and $log{\alpha }_j\sim \text{N}(0.05,0.20)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\log \alpha _j \sim \text {N}(0.05,0.20)$$\end{document} . These values are based on the estimated item parameters from the PISA 2018 reading MST data discussed in the next section. Ability parameters $\theta$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta $$\end{document} were drawn from a standard normal distribution. We used 200 replications in each of the four conditions (number of items x sample size). For each replication, a single complete data set was simulated to which the three designs (random, basic MST, and balanced MST) were applied to create incomplete data.

The MST modules were assembled as follows. After the item parameters were randomly drawn, the items were sorted by difficulty and classified into three difficulty groups (low, medium, and high) with 1/3 of the items in each group (3 or 15). For the routing module in the first stage, an equal number of items was used from each difficulty group (so 1 or 5 easy, 1 or 5 medium, and 1 or 5 high, for 3 or 15 items, respectively). For the low- and high-difficulty modules in the second stage, the majority of the items would come from the associated difficulty group (2 easy, 1 medium for low difficulty module, and 2 high, 1 medium for high difficulty for 3 items; 10 easy, 5 medium for low difficulty module and 10 high, 5 medium for high difficulty for 15 items). In the basic MST design, items were fixed in the three modules, whereas in the balanced, items were rotated across the three modules such that each item occurred in each stage. That is, in the balanced MST, three groups of test takers were created: I, II, and III with each either 200 or 2000 test takers depending on the total sample size used. For the condition with 15 items in each module, in group I, module A consists of 5 easy, 5 medium, and 5 hard items. In group II, module A consists of 5 different easy items, 5 different medium items, and 5 different hard items. In group III, the remaining items from each difficulty group are used. Similar reasoning is applied to create the low-difficulty module B and the high-difficulty module C for groups II and III. The routing decision was based on the sum score on the first module with a cutoff of either 2 out 3 correct or 8 out of 15 items.

We evaluate the simulated data with the different parameter settings on two types of outcomes. The first type consists of outcomes related to the estimation of item and ability parameters. This provides a check on the ignorability principle for the MST designs. We make use of the averaged bias and averaged root mean squared error (RMSE) for both types of parameters, the IRT reliability of $\theta$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta $$\end{document} , and the log determinant of the information matrix for item parameters.2 This latter outcome relates to the D-optimality criterion often used in optimal design research (Berger, 1992). The second type of outcomes concerns CI. We compare the performance of the unadjusted and adjusted generalized residuals for item pair frequencies under the model (Type I error). Since the data and the model are the same across all conditions (only the missing data is different), it is to be expected that the distributions of these residuals are asymptotically similar. We made use of the open-source MIRT software developed by Haberman (2013) to estimate the model parameters.3 The sortware uses marginal maximum likelihood method for model parameter estimation. We developed additional R code (R Core Team, 2019) to compute the required adjustments for the residuals using Equations 19 to 23, which can be found in appendix A.

To illustrate that the computations are nontrivial, consider the cells in the simulation design with 15 items in each stage. With a cutoff score of 8 for the first module, there are 16,384 possible response patterns that lead to a sum score lower than 8 and 16,384 patterns that lead to a sum score 8 or higher. As noted, the Lord-Wingersky algorithm can be put to use in case routing is based on the sum score, which is relatively straightforward. However, if routing is based on a different criterion (e.g., an ability estimate), it may be necessary to repeatedly go over all possible response patterns in order to compute the adjusted residuals.

The results related to item parameter recovery are presented in Appendix B. Table 1 shows the means and standard deviations of Type-I-error rates for nominal $\alpha$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha $$\end{document} levels of .01, .05, and .10 over 200 replications. It can be seen that the results for the complete and random designs are very close to the nominal levels for $N=6000$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N=6000$$\end{document} and significance levels of .05 and .10. The results for the basic MST and balanced MST without adjustments are off by a large amount. However, when the adjustments are applied to take into account the MST design, the results are very similar to the complete and random design. It should be noted that the different designs have different sample spaces when it comes to item pairs.

Table 1 Mean (SD) type I error of generalized residuals for item pair frequencies.

Figure 3 shows plots of the smoothed densities of the residuals for each of the four conditions. The densities for the complete and random designs are close to each other and close to normal. The same holds for the densities under the basic and balanced MST designs with adjustments. QQ plots can be found in appendix B and show a similar result. Appendix B also contains some results on the $Q_3$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q_3$$\end{document} and $M_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M_2$$\end{document} statistics with MST data.

Figure. 3 Density plots of residuals (top left: $N=600$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N=600$$\end{document} , $J=9$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$J=9$$\end{document} ; top right: $N=6000$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N=6000$$\end{document} , $J=9$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$J=9$$\end{document} ; bottom left: $N=600$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N=600$$\end{document} , $J=45$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$J=45$$\end{document} ; bottom right: $N=6000$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N=6000$$\end{document} , $J=45$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$J=45$$\end{document} )

2.2. Real Data Application

In this section, we illustrate the complexity of evaluating CI assumptions with MST data by an application to a large-scale educational assessment, namely, the PISA 2018 Reading MST. In the context of PISA, issues with the CI assumption have been raised before. In PISA’s terminology, items are grouped into units that share stimuli or content. Monseur et al. (2011) found conditional dependencies within units, but also between units. More importantly, this impacted the estimated country-level variance of student proficiencies.

To illustrate the adjusted residuals with real data, we make use of a subset of data from the PISA 2018 Reading MST. For more details on the design, see Yamamoto et al. (2019). The PISA 2018 Reading MST consisted of three stages and was administered to $N=562,051$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N=562,\!\!051$$\end{document} students. The routing is based on (1) random assignment to one of the modules in the first stage, (2) performance in the first and second stage measured by the sum score on automatically scorable items, and (3) a probability layer matrix.

In PISA, the units are combined into testlets that form the modules of the MST. Students take two units in the first stage, three units in the second stage, and two units in the third stage. After the first stage, three performance groups are created based on the total score for items that can be automatically scored: low, medium, and high. The low-performing group is routed with .9 probability to a low-difficulty module in the second stage and with .1 probability to a high-difficulty module.4 The high-performing group is routed in the opposite way as the low-performing group. The medium-performing group is routed with .5 probability to a low-difficulty module in the second stage and with .5 probability to a high-difficulty module. After the second stage, routing to the third stage is performed in a similar fashion. To reduce the impact of item position effects, stages two and three were reversed for 25% of students, which is referred to as design B (while the regular design is referred to as A). The total number of paths in the MST was 192. Note that an MST path can be taken as a test form, with the difference being that an MST path is dependent on the item responses during the test administration whereas a test form can be assigned before the test is administered.

To illustrate conditional dependencies resulting from the unit structure, we computed the generalized residuals for item pairs within each of the five units that were used to create the eight testlets in the first stage. Since students were randomly assigned to take one of these eight testlets, adjustments as described in the previous sections are not necessary. Figure 4 shows the heatmap of the estimated generalized residuals for all 22 items in the first stage. Two pairs of units did not have observations by design (R220–R424 and R559–R560). Within-unit CDs are seen about twice as often as between-unit conditional dependencies: 88% of within-unit residuals are significant at the.01 level vs. 43% of between-unit residuals.

Figure. 4 Heatmap of generalized residuals for item pairs in first stage of PISA 2018 reading MST.

Since substantial conditional dependencies due to the unit structure were found for the items in the first stage and to simplify our analysis, we treated the 45 units (245 items) as 45 polytomously scored items. A multi-group generalized partial credit model (GPCM) was fitted to these 45 items ( $N=562,051$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N=562,\!\!051$$\end{document} ) using the MIRT software with 102 country-by-language groups. Although it would be of interest to investigate within-unit conditional dependence, the size of such an analysis is beyond the scope of our illustration. To simplify things further, we refrain from using sampling weights and the complex sampling design.

We further analyzed one path that can follow from one of the eight modules in the first stage into the second stage, ignoring the third stage. This module consists of two units (R560 and R424) with in total seven automatically scorable items. Students taking this module can be routed to four modules in the second stage, two low-difficulty and two high-difficulty modules. However, we focus our analysis on one of these four paths and compute the residuals for one pair of units with one unit in the first stage (R560) and one unit in one of the low-difficulty modules in the second stage (R542). How the routing works is illustrated in Fig. 5.

Figure. 5 Highlight of MST design for PISA 2018 reading assessment.

After the first stage, six groups of students can be distinguished. The first group is the low-performance group (i.e., $x_A^{+}<3$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_A^+ < 3$$\end{document} ) that is routed to the low-difficulty module, the second group is the low-performance group that is routed to the high-difficulty module, the third group is the medium-performance group (i.e., $3\le x_A^{+}\le 5$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$3 \le x_A^+ \le 5$$\end{document} ) that is routed to the low-difficulty module, up to the sixth group which is the high-performance group (i.e., $x_A^{+}>5$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ x_A^+ > 5$$\end{document} ) that is routed to the high-difficulty module. The adjustments are different for each of the six groups, so that even for this simple subset, the analysis requires careful bookkeeping. The sum score cutoffs for the first module are 3 and 5 with a maximum score of 7. We only focus on the low-difficulty modules in stage two, so the expected frequencies for the three associated groups are given by

(30) $\begin{array}{cc}E(x_{A_j},x_{B_k})& =\sum _{i\in N_{\mathrm{jkLL}}}\int p\left(x_{A_j}\right|\theta ,x_{A+}<3)p\left(x_{B_k}\right|\theta )f\left(\theta \right|x_i)d\theta ,\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} E(x_{A_j},x_{B_k})&= \sum _{i \in {\mathcal {N}}_{jkLL}} \int p(x_{A_j}|\theta ,x_{A+} < 3)p(x_{B_k}|\theta )f(\theta |{\textbf{x}}_i)d\theta ,\end{aligned}$$\end{document}

(31) $\begin{array}{cc}E(x_{A_j},x_{B_k})& =\sum _{i\in N_{\mathrm{jkML}}}\int p\left(x_{A_j}\right|\theta ,3\le x_{A+}\le 5)p\left(x_{B_k}\right|\theta )f\left(\theta \right|x_i)d\theta ,\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} E(x_{A_j},x_{B_k})&= \sum _{i \in {\mathcal {N}}_{jkML}} \int p(x_{A_j}|\theta ,3 \le x_{A+}\le 5)p(x_{B_k}|\theta )f(\theta |{\textbf{x}}_i)d\theta , \end{aligned}$$\end{document}

(32) $\begin{array}{cc}E(x_{A_j},x_{B_k})& =\sum _{i\in N_{\mathrm{jkHL}}}\int p\left(x_{A_j}\right|\theta ,x_{A+}>5)p\left(x_{B_k}\right|\theta )f\left(\theta \right|x_i)d\theta .\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} E(x_{A_j},x_{B_k})&= \sum _{i \in {\mathcal {N}}_{jkHL}} \int p(x_{A_j}|\theta ,x_{A+} > 5)p(x_{B_k}|\theta )f(\theta |{\textbf{x}}_i)d\theta . \end{aligned}$$\end{document}

The other expected frequencies can be obtained in a similar fashion.

Table 2 displays the observed, expected, and residual frequencies, standard deviations and generalized residuals with and without adjustments for the MST design for all patterns on this pair of units for the given subset of the PISA 2018 reading MST data. It is clearly seen that the adjusted expected frequencies are closer to the observed frequencies than the unadjusted frequencies. For example, for the unadjusted residuals, 22 out of 30 residual frequencies are significant at the  .05 level (i.e., Z values are larger than 1.96 in absolute value), whereas for the adjusted residuals, 12 out of 30 are significant. Furthermore, Pearson’s $X^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X^2$$\end{document} Chen & Thissen, 1997) with 20 degrees of freedom is 1043.9 without adjustments and 117.9 with adjustments. To visualize the differences and patterns, we created heatmaps for the unadjusted and adjusted residuals for pairs of scores with an observed frequency of at least 10, which are shown in Fig. 6. It is clear that the adjusted residuals are less extreme than the unadjusted residuals. However, even after adjusting, there seems to be a pattern of underestimating frequencies of low scores on the first unit R560 and high scores on the second unit R542. Although there seem to be some clear deviations from CI, it is not as bad as one would think based on the unadjusted residuals. Furthermore, Cramér’s V coefficient, which can be taken as an effect size measure, is .05, indicating a small effect. After reviewing the units, we could not find substantive reasons for dependence.5

Table 2 Observed, expected, and residual frequencies, standard deviations and generalized residuals with and without MST adjustments for all patterns on one pair of units for subset of PISA 2018 reading MST data ( $N=14,226$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N=14,\!226$$\end{document} ).

Even with the relatively large number of observations for this pair of units ( $N=14,226$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N=14,\!226$$\end{document} ), the two-way contingency table has some sparse elements (e.g., the observed frequencies of $x_j=3$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_j=3$$\end{document} or 4 and $x_k=0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_k=0$$\end{document} are both 1). We can use the reduced residual of Eq. 24 and the associated generalized residual. For the PISA unit pair, the reduced generalized residual is $-$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-$$\end{document} 1.40 without adjustments for the MST design and 1.96 with adjustments for the MST design, which is not entirely in line with the full results. Although this result warrants further study, we leave that to a future investigation.

In closing this section, we want to emphasize that evaluating CI for a large-scale assessment such as PISA is quite the undertaking. With 192 paths in the MST and each student taking 7 units, there are 4032 pairs of units to be inspected and each pair would need a different set of adjustments to obtain the proper residual. If item pairs were analyzed instead of unit pairs, this number would be substantially larger. Furthermore, there is the potential of item-by-country interactions that can affect the scaling of the PISA data (von Davier et al., 2019) as well as the evaluation of CI.

3. Discussion

In this paper, we discussed the key assumption of CI in IRT models in the context of MST data and argued that it is related to quasi-independence in log-linear models for incomplete contingency tables. We demonstrated that generalized residuals for item pair frequencies are inappropriate for MST data without adjustments. The performance of the MST-adjusted residuals was demonstrated through simulations. With respect to parameter recovery in case of MST data, it was found that the balanced MST design seems to have the best of both worlds in the sense that it produced low RMSEs for both item and ability parameters. In the conditions of our simulation study, the adjusted residuals for item pair frequencies seemed to have satisfactory Type-I-error rates for both the basic and balanced MST design when the sample size is sufficiently large.

Figure. 6 Heatmap of unadjusted (left) and adjusted (right) generalized residuals for PISA example.

The adjusted residuals were also applied to a subset of real MST data from PISA, which demonstrated the complexities of evaluating the assumption of CI in a large-scale MST application and indicated a small violation. In the PISA 2018 Reading MST, there were many different paths and our illustration demonstrates what needs to be done to compute the adjusted residuals. In this illustration, CI was assessed for a single path, but the sample size per path may be small and this affects the power to detect violations. If possible, it may help to aggregate results if the same item pair appears in multiple paths. However, there is a difference in the type of CI violation that may warrant assessing CI separately for each path. For example, with a balanced MST and each item appearing in each stage, it may be that CI for the item pair A,B is violated if item A is in the routing stage and item B is in the adaptive stage, but not if item B is in the routing stage and item A is in the adaptive stage. This could indicate a sequential dependence.

As noted, violations to CI can lead to inflated measurement precision, confounds in scores and subscores, and issues with the adaptive algorithm. If many large residuals are found, a number of strategies can be applied to minimize negative impact on the measurement (Yen, 1993). Before applying any strategy, it is important to track down what is causing the conditional dependence. If an explanation can be found for a particular item pair, one can, for example, resort to simply combining scores of item pairs. Alternatively, if the pattern of conditional dependencies is more elaborate, a more complex measurement model may be required and some examples were discussed in the Model section. More importantly, when a more complex measurement model is required, such as a testlet or bifactor MIRT model, but a more basic measurement model, such as a unidimensional IRT model, was used to develop the MST, the adaptivity of the MST may not be as effective and, in extreme cases, may lead to biased outcomes (Jewsbury & van Rijn, 2020).

More research can be conducted to address the impact of MST data on the adjusted residuals for item pair frequencies under more conditions. Different models may be used as well as model violations (dimensionality, conditional dependence). In addition, although we present some results in Appendix B, more work is needed to evaluate the impact of MST data on statistical methods and what adjustments are needed to provide correct results (Ali et al., in press). Such statistical methods include classical item statistics (e.g., percent correct), other statistics for CI (e.g., $Q_3$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q_3$$\end{document} ), and model fit statistics (e.g., $M_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M_2$$\end{document} ). However, the current paper provides some pointers in the right direction.

We end the paper by noting that, in practice, MST designs and routing decisions can be quite complex, as shown in the PISA application. This can make the computation of the adjusted residuals for the complete test quite involved and specific to the situation at hand. That is, it is not straightforward to create software or code that is sufficiently generic to deal with the large variety of MST designs seen in practice.