On the Consistency of Q-Matrix Estimation: A Rejoinder

Abstract

This rejoinder responds to the commentary by Liu (Psychometrika, 2015) entitled “On the consistency of Q-matrix estimation: A commentary” on the paper “A general method of empirical Q-matrix validation” by de la Torre and Chiu (Psychometrika, 2015). It discusses and addresses three concerns raised in the commentary, namely the estimation accuracy when a provisional Q-matrix is used, the consistency of the Q-matrix estimator, and the computational efficiency of the proposed method.

1. Overview

Liu (2015), in his commentary on the Q-matrix validation method proposed by de la Torre and Chiu (2015), raised several concerns that include (a) the accuracy of the posterior distribution w and the item response probabilities $p\left(\alpha \right)=P(Y=1|\alpha )$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p({\alpha })=P(Y=1 | {\alpha })$$\end{document} , as they depend on the provisional Q-matrix $Q_0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q_0$$\end{document} , which may or may not be correctly specified, (b) the consistency of the estimated $\hat{Q}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\hat{Q}}$$\end{document} , and (c) the computational efficiency of the algorithm. We appreciate the opportunity to further explain in detail the proposed method. This rejoinder responds to these concerns by providing some insights and discussion of the three issues.

2. The Accuracy of the Posterior Distribution w and the Item Response Probabilities $p\left(\alpha \right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p({\alpha })$$\end{document}

It can be shown that, when the number of misspecifications in the Q-matrix is small to moderate, as assumed in de la Torre and Chiu’s (2015) study and as is the case with most real assessments, the accuracy of the estimated w would not be significantly affected by using the provisional $Q_0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q_0$$\end{document} . In the maximum likelihood estimation procedure, the likelihood is used in a relative, not absolute sense; thus, w can be well recovered by the item parameter estimates of other, correctly specified items. In other words, it might be the case that the estimates of $p\left(\alpha \right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p({\alpha })$$\end{document} for some misspecified items deviate from the true probabilities, but when the number of misspecified q-entries is small, the whole likelihood is sufficiently robust to produce a posterior w that is very similar to, if not the same as, that arrived at using the correct Q-matrix $Q^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q^{*}$$\end{document} . Of course, this does not guarantee that some unexpected peculiar combinations of the misspecifications would not result in a larger-than-expected distortion of w.

3. The Consistency of $\hat{Q}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\hat{Q}}$$\end{document}

Given the above notion (i.e., that the estimation of w will not be significantly affected by a slightly misspecified Q-matrix compared with the correct Q-matrix), it is questionable whether the hypothesis $P(Q_{\infty }=Q^{\ast })\to 1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P(Q_{\infty } = Q^*) \rightarrow 1$$\end{document} is always true, because from some point on along the iterations, $\hat{Q}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\hat{Q}}$$\end{document} may result in an estimated w, and therefore $\hat{Q}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\hat{Q}}$$\end{document} , that can no longer be updated despite the fact that $\hat{Q}\ne Q^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\hat{Q}} \ne Q^*$$\end{document} . In the commentary, the author introduced the concept of “equilibrium point” to rule out the inconsistent cases discussed above. It was mentioned that it is not difficult to construct examples where $Q_0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q_0$$\end{document} converges to the equilibrium points. It would have been helpful if such examples had been included so that readers could learn more about the asymptotic behavior of $\hat{Q}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\hat{Q}}$$\end{document} , and the relation between $\hat{Q}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\hat{Q}}$$\end{document} and the equilibrium points in the context of cognitive diagnosis. Given the possibility of converging to different stationary points, not surprisingly, additional conditions are needed to make the consistency hold. However, it is unclear from the commentary what these additional conditions might be, and whether it is even possible to establish such conditions. The conjecture that “one conceivable key condition would be the closeness of $Q_0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q_0$$\end{document} to $Q^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q^{*}$$\end{document} ” in the commentary was established without additional elaboration. It seemed to point to the direction of the convergence of $Q_0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q_0$$\end{document} to $Q^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q^*$$\end{document} rather than to the fact that the equilibrium points can be established if $Q_0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q_0$$\end{document} is within a small neighborhood of $Q^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q^{*}$$\end{document} , which may or may not be true. At present, there exists no systematic research indicating that, when the true q-vector is (11100), recovering the true specification from (11110) is easier compared with (10000), although (11110) is “closer” to (11100) than (10000) in terms of distance. This issue is further exacerbated when there exist other misspecifications in the Q-matrix.

4. Computational Efficiency of the Algorithm

In the commentary, it was stated that, when the effects of attributes $1:(K^{\prime }-1)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$1:(K'-1)$$\end{document} are small or ${\alpha }_{1:(K^{\prime }-1)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\alpha }_{1:(K'-1)}$$\end{document} is well predicted by ${\alpha }_{K^{\prime }:K^{\prime \prime }}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\alpha }_{K':K''}$$\end{document} , ${\varsigma }_{K^{\prime }:K^{\prime \prime }}^2>ϵ{\varsigma }_{1:K}^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varsigma ^2_{K':K''}>\epsilon \varsigma ^2_{1:K}$$\end{document} can be true for a fixed $ϵ$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\epsilon $$\end{document} . This raises the question of whether the procedure that de la Torre and Chiu (2015) proposed can only produce one type of errors—underestimations of the true q-vectors. Also, if in fact the effects of attributes $1:(K^{\prime }-1)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$1:(K'-1)$$\end{document} are small or a more compact subset of $\alpha$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\alpha }$$\end{document} can be found, should one deem ${\alpha }_{1:(K^{\prime }-1)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\alpha }_{1:(K'-1)}$$\end{document} necessary? It was suggested in the commentary that $ϵ=1-{(logn)}^{-1}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\epsilon =1-(\log n)^{-1}$$\end{document} should be chosen instead of a prefixed $ϵ$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\epsilon $$\end{document} to overcome possible underestimation of the q-vector. Although we agree that the use of a prefixed $ϵ$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\epsilon $$\end{document} may not necessarily be ideal across various sample sizes, it is not clear that setting $ϵ=1-{(logn)}^{-1}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\epsilon =1-(\log n)^{-1}$$\end{document} would be the solution. De la Torre and Chiu (2015) found the current cutoff of $ϵ=0.95$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\epsilon =0.95$$\end{document} to be optimal across the conditions examined in their study. To match this cutoff using $ϵ=1-{(logn)}^{-1}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\epsilon =1-(\log n)^{-1}$$\end{document} , the sample size needs to be in the vicinity of $n=5\times {10}^8$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n=5\times 10^8$$\end{document} . However, in practical testing situations, it would be more realistic to find n between 200 and 2000, which translates to $ϵ$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\epsilon $$\end{document} between 0.81 and 0.87. In the context of the de la Torre and Chiu (2015) study, it is a concern that these cutoffs may be suboptimal.

Finally, although not related to the three main issues above, we would like to address the issue of whether or not the estimation of $p\left(\alpha \right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p({\alpha })$$\end{document} is independent of the fitted cognitive diagnosis model (CDM). The claim that “[the] $p\left(\alpha \right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p({\alpha })$$\end{document} for a particular item is not based on the estimated G-DINA model” may need some elaboration. This is because the posterior distribution $p\left(\alpha \right|Y_i)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p({\alpha }|{Y}_i)$$\end{document} would definitely change depending on which CDM is used, except when a more general model is used in place of a true reduced model. Given that $p\left(\alpha \right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p({\alpha })$$\end{document} is a function of $p\left(\alpha \right|Y_i)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p({\alpha }|{Y}_i)$$\end{document} , using different CDMs, some of which may be incorrect, can be expected to have an impact on the estimation of $p\left(\alpha \right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p({\alpha })$$\end{document} .