Hostname: page-component-5f745c7db-8qdnt Total loading time: 0 Render date: 2025-01-06T07:08:24.979Z Has data issue: true hasContentIssue false
  • English
  • Français

Improving the Conduct and Reporting of Statistical Analysis in Psychology

Published online by Cambridge University Press:  01 January 2025

Klaas Sijtsma ,
Coosje L. S. Veldkamp  and
Jelte M. Wicherts
Klaas Sijtsma*
Affiliation:
Tilburg University
Coosje L. S. Veldkamp
Affiliation:
Tilburg University
Jelte M. Wicherts
Affiliation:
Tilburg University
*
Correspondence should be made to Klaas Sijtsma, Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, PO Box 90153, 5000 LE Tilburg, The Netherlands. Email: k.sijtsma@tilburguniversity.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

We respond to the commentaries Waldman and Lilienfeld (Psychometrika, 2015) and Wigboldus and Dotch (Psychometrika, 2015) provided in response to Sijtsma’s (Sijtsma in Psychometrika, 2015) discussion article on questionable research practices. Specifically, we discuss the fear of an increased dichotomy between substantive and statistical aspects of research that may arise when the latter aspects are laid entirely in the hands of a statistician, remedies for false positives and replication failure, and the status of data exploration, and we provide a re-definition of the concept of questionable research practices.

Keywords

data explorationquestionable research practicesremedies for false positivesreplication failure
Type
Original paper
Information
Psychometrika , Volume 81 , Issue 1 , March 2016 , pp. 33 - 38
Copyright
Copyright © 2015 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Bakker, M., & Wicherts, J.M. (2011). The (mis)reporting of statistical results in psychology journals. Behavior Research Methods, 43, 666–678.CrossRefGoogle ScholarPubMed
Bakker, M., & Wicherts, J.M. (2014). Outlier removal, sum scores, and the inflation of the Type I error rate in t tests. The power of alternatives and recommendations. Psychological Methods, 19, 409–427.CrossRefGoogle Scholar
Bakker, M., van Dijk, A., & Wicherts, J.M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543–554.CrossRefGoogle ScholarPubMed
Borsboom, D. (2006). The attack of the psychometricians. Psychometrika, 71, 425–440.CrossRefGoogle ScholarPubMed
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. (2nd ed.). Hillsdale, NJ: Erlbaum.Google Scholar
Cronbach, L.J. (1954). Report on a psychometric mission to Clinicia. Psychometrika, 19, 263–270.CrossRefGoogle Scholar
Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25, 7–29.CrossRefGoogle Scholar
Fanelli, D. (2013). Redefine misconduct as distorted reporting. Nature, 494, 149.CrossRefGoogle ScholarPubMed
Fisher, R. A. (1938). Presidential Address. Talk given at the 1st Indian Statistical Conference, Calcutta, India.Google Scholar
Ioannidis, J.P.A. (2005). Why most published research findings are false. PLoS Med, 2(8), e124.CrossRefGoogle ScholarPubMed
John, L.K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524–532.CrossRefGoogle ScholarPubMed
Kuhn, T.S. (2012). The structure of scientific revolutions. (4th ed.). Chicago, IL: The University of Chicago Press.CrossRefGoogle Scholar
Martinson, B.C., Anderson, M.S., & De Vries, R. (2005). Scientists behaving badly. Nature, 435(7043), 737–738. doi:10.1038/435737a.CrossRefGoogle ScholarPubMed
Pashler, H., & Harris, C.R. (2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7(6), 531–536.CrossRefGoogle Scholar
Rosenthal, R. (1994). Science and ethics in conducting, analyzing, and reporting psychological research. Psychological Science, 5, 127–134.CrossRefGoogle ScholarPubMed
Sijtsma, K. (2015). Playing with data–Or how to discourage questionable research practices and stimulate researchers to do things right. Psychometrika. doi:https://doi.org/10.1007/s11336-015-9446-0.CrossRefGoogle Scholar
Simmons, J.P., Nelson, L.D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.CrossRefGoogle ScholarPubMed
Steneck, N.H. (2006). Fostering integrity in research: Definitions, current knowledge, and future directions. Science and Engineering Ethics, 12, 53–74.CrossRefGoogle ScholarPubMed
Tukey, J.W. (1977). Exploratory data analysis. Reading, MA: Addison-Wesley.Google Scholar
Veldkamp, C.L.S., Nuijten, M.B., Dominguez-Alvarez, L., Van Assen, MALM, & Wicherts, J.M. (2014). Statistical reporting errors and collaboration on statistical analyses in psychological science. PloS ONE, 9, e114876.CrossRefGoogle ScholarPubMed
Waldman, I. D., & Lilienfeld, S. O. (2015). Thinking about data, research methods, and statistical analyses: Commentary on Sijtsma’s (2014) “Playing with data". Psychometrika. doi:https://doi.org/10.1007/s11336-015-9447-z.CrossRefGoogle Scholar
Wigboldus, D. H. J., & Dotch, R. (2015). Encourage playing with data and discourage questionable reporting practices. Psychometrika. doi:https://doi.org/10.1007/s11336-015-9445-1.CrossRefGoogle Scholar