Hostname: page-component-cd9895bd7-p9bg8 Total loading time: 0 Render date: 2025-01-04T04:47:37.067Z Has data issue: false hasContentIssue false
  • English
  • Français

Laboratory experiments can pre-design to address power and selection issues

Published online by Cambridge University Press:  01 January 2025

Weili Ding Open the ORCID record for Weili Ding [Opens in a new window]
Weili Ding*
Affiliation:
Queen’s University, Kingston, Canada
*
e-mail: dingw@queensu.ca
Get access Rights & Permissions [Opens in a new window]

Abstract

In this paper, motivated by aspects of preregistration plans we discuss issues that we believe have important implications for how experiments are designed. To make possible valid inferences about the effects of a treatment in question, we first illustrate how economic theories can help allocate subjects across treatments in a manner that boosts statistical power. Using data from two laboratory experiments where subject behavior deviated sharply from theory, we show that the ex-post subject allocation to maximize statistical power is closer to these ex-ante calculations relative to traditional designs that balances the number of subjects across treatments. Finally, we call for increased attention to (i) the appropriate levels of the type I and type II errors for power calculations, and (ii) how experimenters consider balance in part by properly handling over-subscription to sessions.

Type
Original Paper
Information
Journal of the Economic Science Association , Volume 6 , Issue 2: Special Issue: Statistical Issues for Experimental Economists , December 2020 , pp. 125 - 138
Copyright
Copyright © Economic Science Association 2020

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

I would like to thank one anonymous reviewer, the guest editor John Ham and Steven Lehrer for many helpful comments and suggestions that have substantially improved the manuscript. Steven Lehrer also generously provided the experimental data analyzed in the study. I wish to thank SSHRC for research support. I am responsible for all errors.

References

Abreu, D., Gul, F. (2000). Bargaining and reputation. Econometrica, 68(1), 85–117. 10.1111/1468-0262.00094CrossRefGoogle Scholar
Baron, D. P., Ferejohn, J. A. (1989). Bargaining in legislatures. The American Political Science Review, 83(4), 1181–1206. 10.2307/1961664CrossRefGoogle Scholar
Bush, S. A. (2015). Sample size determination for logistic regression: a simulation study. Communications in Statistics Simulation and Computation, 44(2), 360–373.CrossRefGoogle Scholar
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T. et al., (2016). Evaluating replicability of laboratory experiments in economics. Science, 351(6280), 1433–1436. 10.1126/science.aaf0918CrossRefGoogle ScholarPubMed
Casari, M., Ham, J. C., Kagel, J. H. (2007). Selection bias, demographic effects, and ability effects in common value auction experiments. American Economic Review, 97(4), 1278–1304. 10.1257/aer.97.4.1278CrossRefGoogle Scholar
Charness, G., Gneezy, U., Kuhn, M. A. (2012). Experimental methods: Between-subject and within subject design. Journal of Economic Behavior & Organization, 81(1), 1–8. 10.1016/j.jebo.2011.08.009CrossRefGoogle Scholar
Cochran, W. G. (1977). Sampling Techniques, 3New York: Wiley.Google Scholar
Coffman, L. C., Niederle, M. (2015). Pre-analysis plans have limited upside, especially where replications are feasible. Journal of Economic Perspectives, 29(3), 81–98. 10.1257/jep.29.3.81CrossRefGoogle Scholar
Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences, 2Hillsdale: Lawrence Erlbaum Associates.Google Scholar
Czibor, E., Jimenez‐Gomez, D., & List, J. A. (2019). The dozen things experimental economists should do (more of). Southern Economic Journal, 86(2), 371–432.CrossRefGoogle Scholar
Ding, W., Lehrer, S. F. (2011). Experimental estimates of the impacts of class size on test scores: robustness and heterogeneity. Education Economics, 19(3), 229–252. 10.1080/09645292.2011.589142CrossRefGoogle Scholar
Ding, W., Lehrer, S. F. (2010). Estimating treatment effects from contaminated multiperiod education experiments: the dynamic impacts of class size reductions. Review of Economics and Statistics, 92(1), 31–42. 10.1162/rest.2009.11453CrossRefGoogle Scholar
Duflo, E., Glennerster, R., Kremer, M., & Schultz, T., Strauss, J. (2007). Using randomization in development economics research: A toolkit Handbook of Development Economics, Amsterdam: Elsevier 3895–3962.Google Scholar
Embrey, M., Fréchette, G. R., Lehrer, S. F. (2015). Bargaining and reputation: An experiment on bargaining in the presence of behavioural types. Review of Economic Studies, 82(2), 608–631. 10.1093/restud/rdu029CrossRefGoogle Scholar
Engle, R. F., & Griliches, Z., Intriligator, M. D. (1984). Wald, likelihood ratio, and Lagrange multiplier statistics in econometrics Handbook of Econometrics, Amsterdam: North Holland 776–828.Google Scholar
Fisher, R. A. (1925). Statistical Methods for Research Workers, 1Oliver and Boyd: Edinburgh.Google Scholar
Ford, I., Norrie, J., Ahmadi, S. (1995). Model inconsistency, illustrated by the Cox proportional hazards model. Statistics in Medicine, 14(8), 735–746. 10.1002/sim.4780140804CrossRefGoogle ScholarPubMed
Fréchette, G. R., Kagel, J. H., Lehrer, S. F. (2003). Bargaining in legislatures: an experimental investigation of open versus closed amendment rules. American Political Science Review, 97(2), 221–232. 10.1017/S0003055403000637CrossRefGoogle Scholar
Ham, J. C., Kagel, J. H., Lehrer, S. F. (2005). Randomization, endogeneity and laboratory experiments: The role of cash balances in private value auctions. Journal of Econometrics, 125(1–2), 175–205. 10.1016/j.jeconom.2004.04.008CrossRefGoogle Scholar
Ham, J. C., LaLonde, R. (1996). The effect of sample selection and initial conditions in duration models: Evidence from experimental data on training. Econometrica, 64(1), 175–205. 10.2307/2171928CrossRefGoogle Scholar
Heonig, J. M., Heisey, D. M. (2001). The abuse of power: The pervasive fallacy of power calculations for data analysis. American Statistician, 55(1), 19–24. 10.1198/000313001300339897CrossRefGoogle Scholar
Hernandez, A. V., Steyerberg, E. W., Habbema, D. F. (2004). Covariate adjustment in randomized controlled trials with dichotomous outcomes increases statistical power and reduces sample size requirements. Journal of Clinical Epidemiology, 57(5), 454–460. 10.1016/j.jclinepi.2003.09.014CrossRefGoogle ScholarPubMed
Kagel, J. H., Harstad, R. M., Levin, D. (1987). Information impact and allocation rules in auctions with affiliated private values: A laboratory study. Econometrica, 55(4), 1275–1304. 10.2307/1913557CrossRefGoogle Scholar
List, J. A., Sadoff, S., Wagner, M. (2011). So you want to run an experiment, now what? Some simple rules of thumb for optimal experimental design. Experimental Economics, 14(4), 439–457. 10.1007/s10683-011-9275-7CrossRefGoogle Scholar
List, J. A., Shaikh, A. M., Xu, Y. (2019). Multiple hypothesis testing in experimental economics. Experimental Economics, 22(4), 773–793. 10.1007/s10683-018-09597-5CrossRefGoogle Scholar
Maniadis, Z., Tufano, F., List, J. A. (2017). To replicate or not to replicate? Exploring reproducibility in economics through the lens of a model and a pilot study. The Economic Journal, 127(605), F209–F235. 10.1111/ecoj.12527CrossRefGoogle Scholar
Manski, C. (2019). Treatment choice with trial data: Statistical decision theory should supplant hypothesis testing. The American Statistician, 73(s1), 296–304. 10.1080/00031305.2018.1513377CrossRefGoogle Scholar
Manski, C., Tetenov, A. (2016). Sufficient trial size to inform clinical practice. Proceedings of the National Academy of Sciences, 113(38), 10518–10523. 10.1073/pnas.1612174113CrossRefGoogle ScholarPubMed
Nikiforakis, N., Slonim, R. (2015). Editors preface: Statistics, replications and null results. Journal of the Economic Science Association, 1(2), 127–131. 10.1007/s40881-015-0018-yCrossRefGoogle Scholar
Palmer, M. W. (1993). Potential biases in site and species selection for ecological monitoring. Environmental Monitoring and Assessment, 26, 277–282. 10.1007/BF00547504CrossRefGoogle ScholarPubMed
Robinson, L. D., Jewell, N. P. (1991). Some surprising results about covariate adjustment in logistic regression models. International Statistical Review, 59(2), 227–240. 10.2307/1403444CrossRefGoogle Scholar
Rodrik, D. (2015). Economics Rules: The Rights and Wrongs of The Dismal Science, New York: W.W. Norton.Google Scholar
Roth, A. E. (1986). Laboratory experimentation in economics. Economics and Philosophy, 2, 245–273. 10.1017/S1478061500002656CrossRefGoogle Scholar
Shieh, G. (2000). On power and sample size calculations for likelihood ratio tests in generalized linear models. Biometrics, 56(4), 1192–1196. 10.1111/j.0006-341X.2000.01192.xCrossRefGoogle ScholarPubMed
Slonim, R., Wang, C., Garbarino, E., Merrett, D. (2013). Opting-in: Participation bias in economic experiments. Journal of Economic Behavior and Organization, 90(1), 43–70. 10.1016/j.jebo.2013.03.013CrossRefGoogle Scholar