Book contents
- Handbook of Research Methods in Social and Personality Psychology
- Cambridge Handbooks in Psychology
- Handbook of Research Methods in Social and Personality Psychology
- Copyright page
- Contents
- Figures
- Tables
- Contributors
- Introduction
- 1 The Romance of Research Methods
- Part I Before You Dive In
- Part II Basic Design Considerations to Know, No Matter What Your Research Is About
- Part III Deep Dives on Methods and Tools for Testing Your Question of Interest
- Part IV Understanding What Your Data Are Telling You About Psychological Processes
- 20 Measurement
- 21 Advanced Psychometrics
- 22 Dealing with Repeated Measures
- 23 The Design and Analysis of Data from Dyads and Groups
- 24 Random Factors and Research Generalization
- 25 Combining Statistical and Causal Mediation Analysis
- 26 Mathematical and Computational Models
- 27 Meta-analysis
- Index
- References
24 - Random Factors and Research Generalization
from Part IV - Understanding What Your Data Are Telling You About Psychological Processes
Published online by Cambridge University Press: 12 December 2024
Book contents
- Handbook of Research Methods in Social and Personality Psychology
- Cambridge Handbooks in Psychology
- Handbook of Research Methods in Social and Personality Psychology
- Copyright page
- Contents
- Figures
- Tables
- Contributors
- Introduction
- 1 The Romance of Research Methods
- Part I Before You Dive In
- Part II Basic Design Considerations to Know, No Matter What Your Research Is About
- Part III Deep Dives on Methods and Tools for Testing Your Question of Interest
- Part IV Understanding What Your Data Are Telling You About Psychological Processes
- 20 Measurement
- 21 Advanced Psychometrics
- 22 Dealing with Repeated Measures
- 23 The Design and Analysis of Data from Dyads and Groups
- 24 Random Factors and Research Generalization
- 25 Combining Statistical and Causal Mediation Analysis
- 26 Mathematical and Computational Models
- 27 Meta-analysis
- Index
- References
Summary
In most social psychological studies, researchers conduct analyses that treat participants as a random effect. This means that inferential statistics about the effects of manipulated variables address the question whether one can generalize effects from the sample of participants included in the research to other participants that might have been used. In many research domains, experiments actually involve multiple random variables (e.g., stimuli or items to which participants respond, experimental accomplices, interacting partners, groups). If analyses in these studies treat participants as the only random factor, then conclusions cannot be generalized to other stimuli, items, accomplices, partners, or groups. What are required are mixed models that allow multiple random factors. For studies with single experimental manipulations, we consider alternative designs with multiple random factors, analytic models, and power considerations. Additionally, we discuss how random factors that vary between studies, rather than within them, may induce effect size heterogeneity, with implications for power and the conduct of replication studies.
Keywords
- Type
- Chapter
- Information
- Publisher: Cambridge University PressPrint publication year: 2024
References
Ackerman, R. A., Kashy, D. A., and Corretti, C. A. (2015). A tutorial on analyzing data from speed-dating studies with heterosexual dyads. Personal Relationships, 22, 92–110.CrossRefGoogle Scholar
Barr, D. J., Levy, R., Scheepers, C., and Tily, H. J. (2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68, 255–278.CrossRefGoogle ScholarPubMed
Bates, D., Kliegl, R., Vasishth, S., and Baayen, H. (2015). Parsimonious mixed models. arXiv, 1506.04967v1 [stat.ME].Google Scholar
Beck, A. T., Steer, R. A., and Carbin, M. G. (1988). Psychometric properties of the Beck depression inventory: Twenty-five years of evaluation. Clinical Psychology Review, 8, 77–100.CrossRefGoogle Scholar
Beck, A. T., Ward, C. H., Mendelson, M., Mock, J., and Erbaugh, J. (1961). An inventory for measuring depression. Archives of General Psychiatry, 4, 561–571.CrossRefGoogle ScholarPubMed
Belmi, P., and Schroeder, J. (2021). Human “resources”? Objectification at work. Journal of Personality and Social Psychology, 120, 384–417.CrossRefGoogle ScholarPubMed
Brauer, M., and Curtin, J. J. (2018). Linear mixed-effects models and the analysis of nonindependent data: A unified framework to analyze categorical and continuous independent variables that vary within-subjects and/or within-items. Psychological Methods, 23(3), 389–411.CrossRefGoogle ScholarPubMed
Brysbaert, M., and Stevens, M. (2018). Power analysis and effect size in mixed effects models: A tutorial. Journal of Cognition, 1(1), 9, http://doi.org/10.5334/joc.10.CrossRefGoogle ScholarPubMed
Cronbach, L. J., Gleser, G. C., Nanda, H., and Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. John Wiley.Google Scholar
Eastwick, P. W., Finkel, E. J., Mochon, D., and Ariely, D. (2007). Selective versus unselective romantic D\desire: Not all reciprocity is created equal. Psychological Science, 18, 317–319.CrossRefGoogle Scholar
Fazio, R. H., Jackson, J. R., Dunton, B. C., and Williams, C. J. (1995). Variability in automatic activation as an unobtrusive measure of racial attitudes: A bona fide pipeline? Journal of Personality and Social Psychology, 69(6), 1013–1027.CrossRefGoogle ScholarPubMed
Goenka, S., and Thomas, M. (2020). The malleable morality of conspicuous consumption. Journal of Personality and Social Psychology, 118, 562–583.CrossRefGoogle ScholarPubMed
Greenwald, A. G., McGhee, D. E., and Schwartz, J. L. (1998). Measuring individual differences in implicit cognition: The implicit association test. Journal of Personality and Social Psychology, 74(6), 1464–1480.CrossRefGoogle ScholarPubMed
Higgins, E. T., Rholes, W. S., and Jones, C. R. (1977). Category accessibility and impression formation. Journal of Experimental Social Psychology, 13, 141–154.CrossRefGoogle Scholar
Judd, C. M., Westfall, J., and Kenny, D. A. (2012). Treating stimuli as a random factor in social psychology: A new and comprehensive solution to a pervasive but largely ignored problem. Journal of Personality and Social Psychology, 103, 54–69.CrossRefGoogle ScholarPubMed
Judd, C. M., Westfall, J., and Kenny, D. A. (2017). Experiments with more than one random factor: Designs, analytic models, and statistical power. Annual Review of Psychology, 68, 601–625.CrossRefGoogle ScholarPubMed
Kenny, D. A., and Judd, C. M. (2019). The unappreciated heterogeneity of effect sizes: Implications for power, precision, planning of research, and replication. Psychological Methods, 24, 578–589.CrossRefGoogle ScholarPubMed
Klein, R. A., Vianello, M., Hasselman, F., et al. (2018). Many Labs 2: Investigating Variation in Replicability Across Samples and Settings. Advances in Methods and Practices in Psychological Science, 1, 443–490.CrossRefGoogle Scholar
Kogan, N., and Wallach, M. A. (1967). Risky-shift phenomenon in small decision-making groups: A test of the information-exchange hypothesis. Journal of Experimental Social Psychology, 3, 75–84.CrossRefGoogle Scholar
Linden, A. H., and Hönekopp, J. (2021). Heterogeneity of research results: A new perspective from which to assess and promote progress in psychological science. Perspectives on Psychological Science, 16, 358–376.CrossRefGoogle ScholarPubMed
McArthur, L. A. (1972). The how and what of why: Some determinants and consequences of causal attribution. Journal of Personality and Social Psychology, 22, 171–193.CrossRefGoogle Scholar
McShane, B. B., and Böckenholt, U. (2014). You cannot step into the same river twice: When power analyses are optimistic. Perspectives on Psychological Science, 9, 612–625.CrossRefGoogle ScholarPubMed
Meagher, B. R. (2015). The effects of interpersonal differences within religious communities: A group actor–partner interdependence model of U.S. congregations. International Journal for the Psychology of Religion, 25, 74–90.CrossRefGoogle Scholar
Myers, D. G., and Lamm, H. (1976). The group polarization phenomenon. Psychological Bulletin, 83(4), 602–627.CrossRefGoogle Scholar
Simons, D. J., Holcombe, A. O., and Spellman, B. A. (2014). An introduction to registered replication reports at Perspectives on Psychological Science. Perspectives on Psychological Science, 9, 552–555.CrossRefGoogle ScholarPubMed
Stephenson, B., and Wicklund, R. A. (1984). The contagion of self-focus within a dyad. Journal of Personality and Social Psychology, 46, 163–168.CrossRefGoogle Scholar
Thorson, K. R., Mendes, W. B., and West, T. V. (2021). Controlling the uncontrolled: Are there incidental experimenter effects on physiologic responding? Psychophysiology, 57(3), e13500.CrossRefGoogle Scholar
Vesely, S., & Klöckner, C. A. (2020). Social desirability in environmental psychology research: Three meta-analyses. Frontiers in Psychology, 11, 1395, at https://doi.org/10.3389/fpsyg.2020.01395.CrossRefGoogle ScholarPubMed
Wallach, M. A., Kogan, N., and Bem, D. (1962). Group influence on individual risk taking. Journal of Abnormal and Social Psychology, 2, 72–86.Google Scholar
Westfall, J. (2017). A shiny app for power analysis with random targets and participants, https://jakewestfall.shinyapps.io/two_factor_power.Google Scholar
Westfall, J., Kenny, D. A., and Judd, C. M. (2014). Statistical power and optimal design in experiments in which samples of participants respond to samples of stimuli. Journal of Experimental Psychology: General, 143, 2020–2045.CrossRefGoogle ScholarPubMed
Wolsiefer, K., Westfall, J., and Judd, C. M. (2017). Modeling stimulus variation in three common implicit attitude tasks. Behavior Research Methods, 49, 1193–1209.CrossRefGoogle ScholarPubMed
Word, C. O., Zanna, M. P., and Cooper, J. (1974). The nonverbal mediation of self-fulfilling prophecies in interracial interaction. Journal of Experimental Social Psychology, 10(2), 109–120.CrossRefGoogle Scholar
Xie, S. Y., Flake, J. K., and Hehman, E. (2019). Perceiver and target characteristics contribute to impression formation differently across race and gender. Journal of Personality and Social Psychology, 117, 364–385.CrossRefGoogle ScholarPubMed