Hostname: page-component-5f745c7db-tvc9f Total loading time: 0 Render date: 2025-01-06T06:57:33.585Z Has data issue: true hasContentIssue false
  • English
  • Français

Generalized Sample Size Determination Formulas for Investigating Contextual Effects by a Three-Level Random Intercept Model

Published online by Cambridge University Press:  01 January 2025

Satoshi Usami
Satoshi Usami*
Affiliation:
University of Tsukuba
*
Correspondence should be made to Satoshi Usami, University of Tsukuba, Tsukuba, Japan. Email: usamis@human.tsukuba.ac.jp
Get access Rights & Permissions [Opens in a new window]

Abstract

Behavioral and psychological researchers have shown strong interests in investigating contextual effects (i.e., the influences of combinations of individual- and group-level predictors on individual-level outcomes). The present research provides generalized formulas for determining the sample size needed in investigating contextual effects according to the desired level of statistical power as well as width of confidence interval. These formulas are derived within a three-level random intercept model that includes one predictor/contextual variable at each level to simultaneously cover various kinds of contextual effects that researchers can show interest. The relative influences of indices included in the formulas on the standard errors of contextual effects estimates are investigated with the aim of further simplifying sample size determination procedures. In addition, simulation studies are performed to investigate finite sample behavior of calculated statistical power, showing that estimated sample sizes based on derived formulas can be both positively and negatively biased due to complex effects of unreliability of contextual variables, multicollinearity, and violation of assumption regarding the known variances. Thus, it is advisable to compare estimated sample sizes under various specifications of indices and to evaluate its potential bias, as illustrated in the example.

Keywords

sample size determinationmultilevel modelstatistical powerconfidence intervalcontextual effects
Type
Original Paper
Information
Psychometrika , Volume 82 , Issue 1 , March 2017 , pp. 133 - 157
Copyright
Copyright © 2016 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Electronic supplementary material The online version of this article (doi:10.1007/s11336-016-9532-y) contains supplementary material, which is available to authorized users.

References

Borenstein, M., Hedges, L. V., & Rothstein, H. (2012). CRT power, 2Teaneck, NJ: Biostat.Google Scholar
Christ, O. (2014). et al. Contextual effect of positive intergroup contact on outgroup prejudice. Proceedings of the National Academy of Sciences of the United States of America, 111, 3996–4000.CrossRefGoogle ScholarPubMed
Chow, S. C., Shao, J., & Wang, H. (2003). Sample size calculation in clinical research, 2New York: Chapman & Hall.CrossRefGoogle Scholar
Cohen, J. (1962). The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology, 65, 145–153.CrossRefGoogle ScholarPubMed
Cohen, J. (1988). Statistical power analysis for the behavioral sciences, Mahwah, NJ: Erlbaum.Google Scholar
Cohen, M. P. (2005). Sample size considerations for multilevel surveys. International Statistical Review, 73, 279–287.CrossRefGoogle Scholar
Cook, J. A., Bruckner, T., MacLennan, G. S., & Seiler, C. M. (2012). Clustering in surgical trials -database of intracluster correlations. Trials, 13, 1–8.CrossRefGoogle ScholarPubMed
Dai, D. Y., & Rinn, A. N. (2008). The big-fish-little-pond effect: What do we know and where do we go from here?. Educational Psychological Review, 20, (3), 283–317.CrossRefGoogle Scholar
De Jong, K., Moerbeek, M., & der Leeden, Van (2010). A priori power analysis in longitudinal three-level multilevel models: An example with therapist effects. Psychotherapy Research, 20, 273–284.CrossRefGoogle ScholarPubMed
Donner, A., & Klar, N. (2000). Design and analysis of cluster randomization trials in health research, London: Arnold.Google Scholar
Fisher, R. A. (1958). Statistical methods for research workers, 13London: Hafner Press.Google Scholar
Goldstein, H. (2003). Multilevel statistical models, 3New York: Oxford University Press.Google Scholar
Grilli, L., & Rampichini, C. (2011). The role of sample cluster means in multilevel models: A view on endogeneity and measurement error issues. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 7, 121–133.CrossRefGoogle Scholar
Hayes, R. J., & Moulton, L. H. (2009). Cluster randomised trials, Boca Raton: CRC Press.CrossRefGoogle Scholar
Hedeker, D., Gibbons, R. D., & Waternaux, C. (1999). Sample size estimation for longitudinal designs with attrition: Comparing time-related contrasts between two groups. Journal of Educational and Behavioral Statistics, 24, 70–93.CrossRefGoogle Scholar
Hedges, L. V., & Hedberg, E. (2007). Intraclass correlation values for planning group randomized trials in education. Educational Evaluation and Policy Analysis, 29, 60–87.CrossRefGoogle Scholar
Heo, M., & Leon, A. C. (2008). Statistical power and sample size requirements for three-level hierarchical cluster randomized trials. Biometrics, 64, 1256–1262.CrossRefGoogle ScholarPubMed
Heo, M., & Leon, A. C. (2009). Sample size requirements to detect an intervention by time interaction in longitudinal cluster randomized trials. Statistical Medicine, 28, 1017–1027.CrossRefGoogle Scholar
Heo, M., Xue, X., & Kim, M. Y. (2013). Sample size requirements to detect an intervention by time interaction in longitudinal cluster randomized clinical trials with random slopes. Computational Statistics and Data Analysis, 60, 169–178.CrossRefGoogle ScholarPubMed
Heo, M., Xue, X., & Kim, M. Y. (2014). Sample size requirements to detect a two- or three-way interaction in longitudinal cluster randomized clinical trials with second-level randomization. Clinical Trials, 11, 503–507.CrossRefGoogle ScholarPubMed
Hox, J. (2010). Multilevel analysis: Techniques and applications, 2Mahwah, NJ: Erlbaum.CrossRefGoogle Scholar
Huguet, P., Dumas, F., Marsh, H., Régner, I., Wheeler, L., Suls, J., Seaton, M., & Nezlek, J. (2009). Clarifying the role of social comparison in the Big-Fish-Little-Pond Effect (BFLPE): An integrative study. Journal of Personality and Social Psychology, 96, 156–170.CrossRefGoogle Scholar
Ito, A., & Matsui, H. (2001). Construction of the classroom climate inventory. The Japanese Journal of Educational Psychology, 49, 449–457.CrossRefGoogle Scholar
Kline, R. B. (2004). Beyond significance testing, Washington, DC: American Psychological Association.Google Scholar
Little, T. D., Bovaird, J. A., & Card, N. A. (Eds). (2007). Modeling contextual effects in longitudinal studies. Mahwah, NJ: Erlbaum.CrossRefGoogle Scholar
Little, T. D. (2013). Longitudinal structural equation modeling. New York: Guilford Press.Google Scholar
Lüdtke, O., Marsh, H., Robitzsch, A., Trautwein, U., Asparouhov, T., & Muthén, B. (2008). The multilevel latent covariate model: A new, more reliable approach to group-level effects in contextual studies. Psychological Methods, 13, 203–229.CrossRefGoogle Scholar
Maas, CJM, & Hox, J. (2005). Sufficient sample sizes for multilevel modeling. Methodology, 1, (3), 86–92.CrossRefGoogle Scholar
Marsh, H. W. (1974). Judgmental anchoring: Stimulus and response variables. Unpublished doctoral dissertation, University of California, Los AngelesGoogle Scholar
Marsh, H. W., & Parker, J. (1984). Determinants of student self-concept: Is it better to be a relatively large fish in a small pond even if you don’t learn to swim as well?. Journal of Personality and Social Psychology, 47, 213–231.CrossRefGoogle Scholar
Marsh, H. W., Kong, C. K., & Hau, K. (2000). Longitudinal multilevel models of the big-fish-little-pond effect on academic self-concept: Counterbalancing contrast and reflected glory effects in Hong Kong schools. Journal of Personality and Social Psychology, 78, 337–349.CrossRefGoogle ScholarPubMed
Marsh, H. W., & Hau, K. T. (2003). Big fish little pond effect on academic self-concept: A crosscultural (26 country) test of the negative effects of academically selective schools. American Psychologist, 58, 364–376.CrossRefGoogle ScholarPubMed
Marsh, H. W., Seaton, M., Trautwein, U., Lüdtke, O., Hau, K. T., O’Mara, A. J., & Craven, R. G. (2008). The big-fish-little-pond-effect stands up to critical scrutiny: Implications for theory, methodology, and future research. Educational Psychology Review, 20, 319–350.CrossRefGoogle Scholar
Marsh, H. W., Trautwein, U., Lüdtke, O., & Köller, O. (2008). Social comparison and big-fish-little-pond effects on self-concept and other self-belief constructs: Role of generalized and specific others. Journal of Educational Psychology, 100, 510–524.CrossRefGoogle Scholar
Marsh, H. W., Lüdtke, O., Robitzsch, A., Trautwein, U., Asparouhov, T., Muthén, B., & Nagengast, B. (2009). Doubly-latent models of school contextual effects: Integrating multilevel and structural equation approaches to control measurement and sampling error. Multivariate Behavioral Research, 44, 764–802.CrossRefGoogle ScholarPubMed
Maxwell, S. E. (2004). The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods, 9, 147–163.CrossRefGoogle ScholarPubMed
Maxwell, S. E., Kelley, K., & Rausch, J. R. (2008). Sample size planning for statistical power and accuracy in parameter estimation. Annual Review of Psychology, 59, 537–563.CrossRefGoogle ScholarPubMed
McArdle, J. J., & Nesselroade, J. R. (2014). Longitudinal data analysis using structural equation models, Washington DC: American Psychological Association.CrossRefGoogle Scholar
McCulloch, C. E., & Searle, S. R. (2001). Generalized, linear, and mixed models, New York: Wiley.Google Scholar
Murray, D. M., & Blitstein, J. L. (2003). Methods to reduce the impact of intraclass correlation in group-randomized trials. Evaluation Review, 27, 79–103.CrossRefGoogle ScholarPubMed
Muthén, L. K., & Muthén, B. O. (2002). How to use a monte carlo study to decide on sample size and determine power. Structural Equation Modeling, 9, 599–620.CrossRefGoogle Scholar
Preckel, F., Zeidner, M., Goetz, T., & Schleyer, E. (2008). Female ’big fish’ swimming against the tide: The ’big-fish-little-pond effect’ and gender-ratio in special gifted classes. Contemporary Educational Psychology, 33, 78–96.CrossRefGoogle Scholar
R Development Core Team (2011). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, http://www.R-project.org/.Google Scholar
Rabe-Hesketh, S., Skrondal, A., & Pickles, A. (2004). Generalized multilevel structural equation modelling. Psychometrika, 69, 167–190.CrossRefGoogle Scholar
Raudenbush, S. W., & Bryk, A. S. (1986). A hierarchical model for studying school effects. Sociology of Education, 59, 1–17.CrossRefGoogle Scholar
Raudenbush, S. W. (1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2, 173–185.CrossRefGoogle Scholar
Raudenbush, S. W., & Liu, X. (2000). Statistical power and optimal design for multisite randomized trials. Psychological Methods, 5, 199–213.CrossRefGoogle ScholarPubMed
Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods, 2London: Sage.Google Scholar
Raudenbush, S. W., Martinez, A., & Spybrook, J. (2007). Strategies for improving precision in group randomized experiments. Educational Evaluation and Policy Analysis, 29, 5–29.CrossRefGoogle Scholar
Rotondi, M. A., & Donner, A. (2009). Sample size estimation in cluster randomized educational trials: An empirical Bayes approach. Journal of Educational and Behavioral Statistics, 34, 229–237.CrossRefGoogle Scholar
Raudenbush, S. W., Spybrook, J., Congdon, R., Liu, X., Martinez, A., Bloom, H., et al. (2011). Optimal design plus empirical evidence, Version 3.0.Google Scholar
Roy, A., Bhaumik, D. K., Aryal, S., & Gibbons, R. D. (2007). Sample size determination for hierarchical longitudinal designs with differential attrition rates. Biometrics, 63, 699–707.CrossRefGoogle ScholarPubMed
Satterthwaite, F. E. (1941). Synthesis of variance. Psychometrika, 6, 309–316.CrossRefGoogle Scholar
Searle, S. R., Casella, G., & McCulloch, C. E. (1992). Variance components, New York: Wiley.CrossRefGoogle Scholar
Seaton, M., Marsh, H. W., & Craven, R. G. (2009). Earning its place as a pan-human theory: Universality of the big-fish-little-pond effect across 41 culturally and economically diverse countries. Journal of Educational Psychology, 101, 403–419.CrossRefGoogle Scholar
Shin, Y., & Raudenbush, S. W. (2010). A latent cluster mean approach to the contextual effects model with missing data. Journal of Educational and Behavioral Statistics, 35, 26–53.CrossRefGoogle Scholar
Singer, J. D., & Willett, J. B. (2003). Applied longitudinal data analysis, New York: Oxford.CrossRefGoogle Scholar
Skrondal, A., & Rabe-Hesketh, S. (2004). Generalized latent variable modeling. Multilevel, longitudinal, and structural equation models, Boca Raton, FL: Chapman & Hall/CRC.Google Scholar
Smeeth, L., & Ng, ES-W (2002). Intra-class correlation coefficients for cluster randomized trials in primary care: Data from the MRC trial of the assessment and management of older people in the community. Control Clinical Trials, 23, 409–421.CrossRefGoogle Scholar
Snijders, TAB, & Bosker, R. J. (1993). Standard errors and sample sizes for two-level research. Journal of Educational Statistics, 18, 237–260.CrossRefGoogle Scholar
Snijders, TAB, & Bosker, R. J. (2012). Multilevel analysis, 2London: Sage.Google Scholar
Thompson, D. M., Fernald, D. H., & Mold, J. W. (2012). Intraclass correlation coefficients typical of cluster-randomized studies: Estimates from the Robert Wood Johnson Prescription for Health projects. The Annals of Family Medicine, 10, 235–240.CrossRefGoogle Scholar
Tymms, P. (2001). A test of the big fish in a little pond hypothesis: An investigation into the feelings of seven-year-old pupils in school. School Effectiveness and School Improvement, 12, 161–181.CrossRefGoogle Scholar
Tymms, P., Schagen, I., & Elliot, K. (2004). Effect sizes in multilevel models. But what does it mean? The use of effect sizes in educational research, London: National Foundation for Educational Research 55–66.Google Scholar
Usami, S. (2011). Statistical power of experimental research with hierarchical data. Behaviormetrika, 38, 63–84.CrossRefGoogle Scholar
Usami, S. (2011). A unified method for determining sample size needed to evaluate mean difference in hierarchical research design and construction of numerical table. Japanese Journal of Educational Psychology, 59, 385–401.Google Scholar
Usami, S. (2014). Generalized sample size determination formulas for experimental research with hierarchical data. Behavior Research Methods, 46, 346–356.CrossRefGoogle ScholarPubMed
Kenny, J., Kenny, D. A., & Judd, C. M. (2014). Statistical power and optimal design in experiments in which samples of participants respond to samples of stimuli. Journal of Experimental Psychology: General, 143, 2020–2045.Google Scholar
Yang, J. S., & Cai, L. (2012). Estimation of contextual effects through nonlinear multilevel latent variable modeling with a Metropolis-Hastings Robbins-Monro algorithm. Lincoln, NE: Paper presented at International Meeting of the Psychometric Society.Google Scholar
Supplementary material: File

Usami supplementary material

Supplemental Materials S1-S4
Download Usami supplementary material(File)
File 36.8 KB
Supplementary material: File

Usami supplementary material

Supplemental Material S5
Download Usami supplementary material(File)
File 197.4 KB
Supplementary material: File

Usami supplementary material

Supplemental Material S6
Download Usami supplementary material(File)
File 9 KB
Supplementary material: File

Usami supplementary material

Supplemental Material S7
Download Usami supplementary material(File)
File 190 KB
Supplementary material: File

Usami supplementary material

Supplemental Material S8
Download Usami supplementary material(File)
File 168.3 KB
Supplementary material: File

Usami supplementary material

Supplemental Materials S9-S10
Download Usami supplementary material(File)
File 182.9 KB
Supplementary material: File

Usami supplementary material

Usami supplementary material 1
Download Usami supplementary material(File)
File 79.5 KB