Hostname: page-component-cd9895bd7-p9bg8 Total loading time: 0 Render date: 2024-12-28T02:36:14.267Z Has data issue: false hasContentIssue false
  • English
  • Français

External Validity and Multi-Organization Samples: Levels-of-Analysis Implications of Crowdsourcing and College Student Samples

Published online by Cambridge University Press:  28 July 2015

Daniel A. Newman ,
Dana L. Joseph  and
Jennifer Feitosa
Daniel A. Newman*
Affiliation:
University of Illinois at Urbana–Champaign
Dana L. Joseph
Affiliation:
University of Central Florida
Jennifer Feitosa
Affiliation:
University of Central Florida
*
Correspondence concerning this article should be addressed to Daniel A. Newman, Department of Psychology, University of Illinois at Urbana–Champaign, 603 East Daniel Street, Champaign, IL 61820. E-mail: d5n@uiuc.edu
Get access Rights & Permissions [Opens in a new window]

Extract

Here, we expand on Landers and Behrend's (2015) discussion of the external validity of convenience samples. In particular, we note that their focal article failed to mention one important limitation of multi-organization convenience samples (e.g., MTurk samples, student samples): Multi-organization convenience samples tend to confound levels of analysis, which affects the external validity of these samples. Specifically, between-organizations phenomena (i.e., organization-level) and within-organizations phenomena (i.e., individual-level) are distinct and separable (Ostroff, 1993; Robinson, 1950). Unfortunately, multi-organization samples such as those found in MTurk or MBA student samples can confound these two sets of phenomena. The current commentary uses a levels-of-analysis framework to expand on Landers and Behrend's discussion of what external validity is, and then the commentary illustrates how the diversity of convenience samples can actually harm external validity under some common circumstances.

Type
Commentaries
Information
Industrial and Organizational Psychology , Volume 8 , Issue 2 , June 2015 , pp. 214 - 220
Copyright
Copyright © Society for Industrial and Organizational Psychology 2015 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Bliese, P. D. (2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In Klein, K. J. & Kozlowski, S. W. J. (Eds.), Multilevel theory, research, and methods in organizations (pp. 349–381). San Francisco, CA: Jossey-Bass.Google Scholar
Dansereau, F., Alluto, J. A., & Yammarino, F. J. (1984). Theory testing in organizational behavior: The variant approach. Englewood Cliffs, NJ: Prentice Hall.Google Scholar
Feitosa, J., Joseph, D., & Newman, D. (2015). Crowdsourcing survey methods and measurement equivalence: A caveat about countries whose primary language is not English. Personality and Individual Differences, 75, 47–52.Google Scholar
Hernandez, I., Newman, D. A., & Jeon, G. (in press). Twitter analysis: Methods for data management and a word count dictionary to measure city-level job satisfaction. In Tonidandel, S., King, E., & Cortina, J. (Eds.), Big data at work: The data science revolution and organizational psychology. New York, NY: Routledge.Google Scholar
Hulin, C. L. (1966). Effects of community characteristics on measures of job satisfaction. Journal of Applied Psychology, 50 (2), 185–192.Google Scholar
Kozlowski, S. W. J., & Klein, K. J. (2000). A multilevel approach to theory and research in organizations: Contextual, temporal, and emergent processes. In Klein, K. J. & Kozlowski, S. W. J. (Eds.), Multilevel theory, research, and methods in organizations (pp. 3–90). San Francisco, CA: Jossey-Bass.Google Scholar
Landers, R. N., & Behrend, T. S. (2015). An inconvenient truth: Arbitrary distinctions between organizational, Mechanical Turk, and other convenience samples. Industrial and Organizational Psychology: Perspectives on Science and Practice.Google Scholar
Ostroff, C. (1993). The effects of climate and personal influences on individual behavior and attitudes in organizations. Organizational Behavior and Human Decision Processes, 56 (1), 56–90.Google Scholar
Robinson, W. S. (1950). Ecological correlations and the behavior of individuals. American Sociological Review, 15, 351–357.Google Scholar
Simpson, E. H. (1951). The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society, 13 (2), 238–241.Google Scholar
Thorndike, E. L. (1939). On the fallacy of imputing the correlations found for groups to the individuals or smaller groups composing them. American Journal of Psychology, 52, 122–124.CrossRefGoogle Scholar
Yule, G. U. (1903). Notes on the theory of association of attributes in statistics. Biometrika, 2 (2), 121–134.Google Scholar