Diabetes and depression have a bidirectional relationship, but some antidepressants (such as the tricyclics) may have detrimental effects in diabetes that are exacerbated by behavioural changes associated with depression. This month's Cochrane Review evaluated the efficacy of psychological and pharmacological treatments of comorbid depression in diabetes and found that such interventions have a moderate and clinically significant effect on depression outcomes in people with diabetes. However, conclusions were limited by significant heterogeneity within examined populations and interventions, and significant risk of bias within trials. This commentary critically appraises the review and aims to contextualise its findings.

When researchers design an experiment, they usually hold potentially relevant features of the experiment constant. We call these details the “topic” of the experiment. For example, researchers studying the impact of party cues on attitudes must inform respondents of the parties’ positions on a particular policy. In doing so, researchers implement just one of many possible designs . Clifford, Leeper, and Rainey (2023. “Generalizing Survey Experiments Using Topic Sampling: An Application to Party Cues.” Forthcoming in Political Behavior. https://doi.org/10.1007/s11109-023-09870-1) argue that researchers should implement many of the possible designs in parallel—what they call “topic sampling”—to generalize to a larger population of topics. We describe two estimators for topic-sampling designs: First, we describe a nonparametric estimator of the typical effect that is unbiased under the assumptions of the design; and second, we describe a hierarchical model that researchers can use to describe the heterogeneity. We suggest describing the heterogeneity across topics in three ways: (1) the standard deviation in treatment effects across topics, (2) the treatment effects for particular topics, and (3) how the treatment effects for particular topics vary with topic-level predictors. We evaluate the performance of the hierarchical model using the Strengthening Democracy Challenge megastudy and show that the hierarchical model works well.

Researchers are often interested in whether discrimination on the basis of racial cues persists above and beyond discrimination on the basis of nonracial attributes that decision makers—e.g., employers and legislators—infer from such cues. We show that existing audit experiments may be unable to parse these mechanisms because of an asymmetry in when decision makers are exposed to cues of race and additional signals intended to rule out discrimination due to other attributes. For example, email audit experiments typically cue race via the name in the email address, at which point legislators can choose to open the email, but cue other attributes in the body of the email, which decision makers can be exposed to only after opening the email. We derive the bias resulting from this asymmetry and then propose two distinct solutions for email audit experiments. The first exposes decision makers to all cues before the decision to open. The second crafts the email to ensure no discrimination in opening and then exposes decision makers to all cues in the body of the email after opening. This second solution works without measures of opening, but can be improved when researchers do measure opening, even if with error.

The bumblebee gut parasite, Crithidia bombi, is widespread and prevalent in the field. Its interaction with Bombus spp. is a well-established epidemiological model. It is spread faecal-orally between colonies via the shared use of flowers when foraging. Accurately measuring the level of infection in bumblebees is important for assessing its distribution in the field, and also when conducting epidemiological experiments. Studies generally use 1 of 2 methods for measuring infection. One approach measures infection in faeces whereas the other method measures infection in guts. We tested whether the method of measuring infection affected the estimation of infection. Bumblebees were inoculated with a standardized inoculum and infection was measured 1 week later using either the faecal or gut method. We found that when the gut method was used to measure infection intensity estimates were significantly different to and approximately double those from the faecal method. These results have implications for the interpretation of previous study results and for the planning of future studies. Given the importance of bumblebees as pollinators, the impact of C. bombi on bumblebee health, and its use as an epidemiological model, we call on researchers to move towards consistent quantification of infections to enable future comparisons and meta-analyses of studies.

Even in the best-designed experiment, noncompliance can complicate analysis. While the intent-to-treat effect remains identified, randomization alone no longer identifies the complier average causal effect (CACE). Instrumental variables approaches, which rely on the exclusion restriction, can suffer from high variance, particularly when the experiment has a low compliance rate. We provide a framework which broadens the set of design and analysis techniques political science researchers can use when addressing noncompliance. Building on the growing literature about the advantages of ex-ante design decisions to improve precision, we show blocking on variables related to both compliance and the outcome can greatly improve all the estimators we propose. Drawing on work in statistics, we introduce the principal ignorability assumption and a class of principal score weighting estimators, which can exhibit large gains in precision in low compliance settings. We then combine principal ignorability and blocking with a simple estimation strategy to derive a more efficient estimation strategy for the CACE. In a re-evaluation of a study on the effect of GOTV on turnout, we find that the principal ignorability approaches result in confidence intervals roughly half the size of traditional instrumental variable approaches.

The majority of research papers in computer-assisted language learning (CALL) report on primarily quantitative studies measuring the effectiveness of pedagogical interventions in relation to language learning outcomes. These studies are frequently referred to in the literature as experiments, although this designation is often incorrect because of the approach to sampling that has been used. This methodological discussion paper provides a broad overview of the current CALL literature, examining reported trends in the field that relate to experimental research and the recommendations made for improving practice. It finds that little attention is given to sampling, even in review articles. This indicates that sampling problems are widespread and that there may be limited awareness of the role of formal sampling procedures in experimental reasoning. The paper then reviews the roles of two key aspects of sampling in experiments: random selection of participants and random assignation of participants to control and experimental conditions. The corresponding differences between experimental and quasi-experimental studies are discussed, along with the implications for interpreting a study’s results. Acknowledging that genuine experimental sampling procedures will not be possible for many CALL researchers, the final section of the paper presents practical recommendations for improved design, reporting, review, and interpretation of quasi-experimental studies in the field.

Many animal preference experiments involve test stimuli that have been chosen by the experimenter to represent different strengths of a single attribute. It is assumed that the animals also scale the test stimuli along a single dimension. This paper shows how it is possible to use the ‘Unfolding Technique' developed by Coombs (1964) to check the validity of this assumption. A simple experiment is described which used Coombs' technique to verify that three visual test stimuli were ranked by laboratory rats along a single dimension. These stimuli were subsequently used in an experiment to see how different housing conditions changed rats' preferences for visual complexity.

Although the physiological and behavioural changes that can indicate poor welfareare generally agreed upon, using these measures in practice sometimes yieldsresults that are hard to interpret. For example, different types of measure maysuggest quite different things about an animal's welfare. Suchcontradictions are often due to the differing properties of the variables beingmeasured. How each variable responds to a stressor can be affected by severalfactors - the type of unpleasant stimulus to which the animal is exposed; whenand for how long exposure occurs; the animal's psychological state, egdoes it feel that it is in control?; and the time at which the measurement ismade, relative to the stressor. Typical responses also often differ betweenspecies and between individuals, and may even change in a single individual overtime. Furthermore, some responses used to assess welfare lack specificity: theycan be elicited by neutral or even pleasant events as well as by aversive ones.Appreciating these factors is vital when designing experiments, when choosingwhat to measure along with each welfare variable, and when interpreting results.Even after taking these factors into consideration, interpreting a result canstill be difficult. One approach then is to consider the effects on welfare ofthe changes measured, eg if there is immunosuppression, does the animal succumbto disease? Another is to use the animal's behaviour to indicate itspreference for, or aversion to, particular environments. Ultimately, however,interpreting welfare measures involves subjective judgements which will beinfluenced by the nature of our concern for the animal under consideration. Byraising these problems, we hope that this review will highlight and clarify theapparent contradictions that sometimes emerge in scientific studies of animalwelfare, and help researchers improve the designs of their experiments for thebenefit of the animals concerned.

Prospect Theory (PT: Kahneman & Tversky, 1979) of risky decision making is based on psychological phenomena (paradoxes) that motivate assumptions about how people react to gains and losses, and how they weight outcomes with probabilities. Recent studies suggest that people’s numeracy affect their decision making. We therefore conducted a large-scale conceptual replication of the seminal study by Kahneman and Tversky (1979), where we targeted participants with larger variability in numeracy. Because people low in numeracy may be more dependent on anchors in the form of other judgments we also manipulated design type (within-subject design, vs. single-stimuli design, where participants assess only one problem). The results from about 1,800 participants showed that design type had no effect on the modal choices. The rate of replication of the paradoxes in Kahneman and Tversky was poor and positively related to the participants’ numeracy. The Probabilistic Insurance Effect was observed at all levels of numeracy. The Reflection Effects were not fully replicated at any numeracy level. The Certainty and Isolation Effects explained by nonlinear probability weighting were replicated only at high numeracy. No participant exhibited all 9 paradoxes and more than 50% of the participants exhibited at most three of the 9 paradoxes. The choices by the participants with low numeracy were consistent with a shift towards a cautionary non-compensatory strategy of minimizing the risk of receiving the worst possible outcome. We discuss the implications for the psychological assumptions of PT.

Determination of sample size (the number of replications) is a key step in the design of an observational study or randomized experiment. Statistical procedures for this purpose are readily available. Their treatment in textbooks is often somewhat marginal, however, and frequently the focus is on just one particular method of inference (significance test, confidence interval). Here, we provide a unified review of approaches and explain their close interrelationships, emphasizing that all approaches rely on the standard error of the quantity of interest, most often a pairwise difference of two means. The focus is on methods that are easy to compute, even without a computer. Our main recommendation based on standard errors is summarized as what we call the 1-2-3 rule for a difference of two treatment means.