In 1984 Jacobson and colleagues introduced the concept of reliable change, viz the amount of change on a measure that an individual needed to show to determine that it exceeded the extent of change likely due to measurement error alone. Establishing reliable change was a pre-requisite for determining clinical significance. This paper summarizes the rationale for determining reliable change as providing an individual-focused, idiographic alternative to the dominant nomothetic approach to clinical outcome research based on group mean data and statistical significance. The conventional computational steps for calculating an individual’s standardized difference (reliable change) score and the minimum raw change score on the measure (a reliable change index) required to classify individuals as reliably positively changed, indeterminate, or reliably deteriorated are described. Two methods for graphically representing reliable change are presented, and a range of possible uses in both research and practice settings are summarized. A number of issues and debates concerning the calculation of reliable change are reviewed. It is concluded that the concept of reliable change remains useful for both cognitive behavioural researchers and practitioners, but that there are options regarding methods of computation. In any use of reliable change, the rationale for selecting among method options and the exact computations used need clear and careful description so that we can continuously judge the utility and appropriateness of the use of reliable change and enhance its value to the field.

We contribute to the inverse farm size-productivity puzzle (IR) literature by examining the relationship using a unique data set from southern Ghana that covers farms between 5 and 70 ha. The study uses an instrumental variable (IV) for land size to mitigate some effects of measurement error in land size. The inverse relationship between farm size and farm productivity is upheld when ordinary least squares estimators (OLS) are applied but becomes insignificant although still negative in the IV estimation. The results show that measurement error in land size attenuates the IR. While some studies found the IR to flatten and then become positive, this study finds that in Ghana, the IR only flattens.

We offer methods to analyze the “differentially private” Facebook URLs Dataset which, at over 40 trillion cell values, is one of the largest social science research datasets ever constructed. The version of differential privacy used in the URLs dataset has specially calibrated random noise added, which provides mathematical guarantees for the privacy of individual research subjects while still making it possible to learn about aggregate patterns of interest to social scientists. Unfortunately, random noise creates measurement error which induces statistical bias—including attenuation, exaggeration, switched signs, or incorrect uncertainty estimates. We adapt methods developed to correct for naturally occurring measurement error, with special attention to computational efficiency for large datasets. The result is statistically valid linear regression estimates and descriptive statistics that can be interpreted as ordinary analyses of nonconfidential data but with appropriately larger standard errors.

Polls asking respondents about their beliefs in conspiracy theories have become increasingly commonplace. However, researchers have expressed concern about the willingness of respondents to divulge beliefs in conspiracy theories due to the stigmatization of those ideas. We use an experimental design similar to a list experiment to decipher the effect of social desirability bias on survey responses to eight conspiratorial statements. Our study includes 8290 respondents across seven countries, allowing for the examination of social desirability bias across various political and cultural contexts. While the proportion of individuals expressing belief in each statement varies across countries, we observe identical treatment effects: respondents systematically underreport conspiracy beliefs. These findings suggest that conspiracy beliefs may be more prominent than current estimates suggest.

To reduce strategic misreporting on sensitive topics, survey researchers increasingly use list experiments rather than direct questions. However, the complexity of list experiments may increase nonstrategic misreporting. We provide the first empirical assessment of this trade-off between strategic and nonstrategic misreporting. We field list experiments on election turnout in two different countries, collecting measures of respondents’ true turnout. We detail and apply a partition validation method which uses true scores to distinguish true and false positives and negatives for list experiments, thus allowing detection of nonstrategic reporting errors. For both list experiments, partition validation reveals nonstrategic misreporting that is: undetected by standard diagnostics or validation; greater than assumed in extant simulation studies; and severe enough that direct turnout questions subject to strategic misreporting exhibit lower overall reporting error. We discuss how our results can inform the choice between list experiment and direct question for other topics and survey contexts.

While classical measurement error in the dependent variable in a linear regression framework results only in a loss of precision, nonclassical measurement error can lead to estimates, which are biased and inference which lacks power. Here, we consider a particular type of nonclassical measurement error: skewed errors. Unfortunately, skewed measurement error is likely to be a relatively common feature of many outcomes of interest in political science research. This study highlights the bias that can result even from relatively “small” amounts of skewed measurement error, particularly, if the measurement error is heteroskedastic. We also assess potential solutions to this problem, focusing on the stochastic frontier model and Nonlinear Least Squares. Simulations and three replications highlight the importance of thinking carefully about skewed measurement error as well as appropriate solutions.

The prespecification of the network is one of the biggest hurdles for applied researchers in undertaking spatial analysis. In this letter, we demonstrate two results. First, we derive bounds for the bias in nonspatial models with omitted spatially-lagged predictors or outcomes. These bias expressions can be obtained without prior knowledge of the network, and are more informative than familiar omitted variable bias formulas. Second, we derive bounds for the bias in spatial econometric models with nondifferential error in the specification of the weights matrix. Under these conditions, we demonstrate that an omitted spatial input is the limit condition of including a misspecificed spatial weights matrix. Simulated experiments further demonstrate that spatial models with a misspecified weights matrix weakly dominate nonspatial models. Our results imply that, where cross-sectional dependence is presumed, researchers should pursue spatial analysis even with limited information on network ties.

A key challenge facing many large, in-person public opinion surveys is ensuring that enumerators follow fieldwork protocols. Implementing “quality control” processes can improve data quality and help ensure the representativeness of the final sample. Yet while public opinion researchers have demonstrated the utility of quality control procedures such as audio capture and geo-tracking, there is little research assessing the relative merits of such tools. In this paper, we present new evidence on this question using data from the 2016/17 wave of the AmericasBarometer study. Results from a large classification task demonstrate that a small set of automated and human-coded variables, available across popular survey platforms, can recover the final sample of interviews that results when a full suite of quality control procedures is implemented. Taken as a whole, our results indicate that implementing and automating just a few of the many quality control procedures available can streamline survey researchers’ quality control processes while substantially improving the quality of their data.

We show that for any n divisible by 3, almost all order-n Steiner triple systems admit a decomposition of almost all their triples into disjoint perfect matchings (that is, almost all Steiner triple systems are almost resolvable).

Measurement errors are omnipresent in network data. Most studies observe an erroneous network instead of the desired error-free network. It is well known that such errors can have a severe impact on network metrics, especially on centrality measures: a central node in the observed network might be less central in the underlying, error-free network. The robustness is a common concept to measure these effects. Studies have shown that the robustness primarily depends on the centrality measure, the type of error (e.g., missing edges or missing nodes), and the network topology (e.g., tree-like, core-periphery). Previous findings regarding the influence of network size on the robustness are, however, inconclusive. We present empirical evidence and analytical arguments indicating that there exist arbitrary large robust and non-robust networks and that the average degree is well suited to explain the robustness. We demonstrate that networks with a higher average degree are often more robust. For the degree centrality and Erdős–Rényi (ER) graphs, we present explicit formulas for the computation of the robustness, mainly based on the joint distribution of node degrees and degree changes which allow us to analyze the robustness for ER graphs with a constant average degree or increasing average degree.

Experiments should be designed to facilitate the detection of experimental measurement error. To this end, we advocate the implementation of identical experimental protocols employing diverse experimental modes. We suggest iterative nonparametric estimation techniques for assessing the magnitude of heterogeneous treatment effects across these modes. And we propose two diagnostic strategies—measurement metrics embedded in experiments, and measurement experiments—that help assess whether any observed heterogeneity reflects experimental measurement error. To illustrate our argument, first we conduct and analyze results from four identical interactive experiments: in the lab; online with subjects from the CESS lab subject pool; online with an online subject pool; and online with MTurk workers. Second, we implement a measurement experiment in India with CESS Online subjects and MTurk workers.

While the spatial weights matrix $\boldsymbol{W}$ is at the core of spatial regression models, there is a scarcity of techniques for validating a given specification of $\boldsymbol{W}$. I approach this problem from a measurement error perspective. When $\boldsymbol{W}$ is inflated by a constant, a predictable form of endogeneity occurs that is not problematic in other regression contexts. I use this insight to construct a theoretically appealing test and control for the validity of $\boldsymbol{W}$ that is tractable in panel data, which I call the K test. I demonstrate the utility of the test using Monte Carlo simulations.

Most network studies rely on a measured network that differs from the underlying network which is obfuscated by measurement errors. It is well known that such errors can have a severe impact on the reliability of network metrics, especially on centrality measures: a more central node in the observed network might be less central in the underlying network. Previous studies have dealt either with the general effects of measurement errors on centrality measures or with the treatment of erroneous network data. In this paper, we propose a method for estimating the impact of measurement errors on the reliability of a centrality measure, given the measured network and assumptions about the type and intensity of the measurement error. This method allows researchers to estimate the robustness of a centrality measure in a specific network and can, therefore, be used as a basis for decision-making. In our experiments, we apply this method to random graphs and real-world networks. We observe that our estimation is, in the vast majority of cases, a good approximation for the robustness of centrality measures. Beyond this, we propose a heuristic to decide whether the estimation procedure should be used. We analyze, for certain networks, why the eigenvector centrality is less robust than, among others, the pagerank. Finally, we give recommendations on how our findings can be applied to future network studies.