Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-cd9895bd7-dk4vv Total loading time: 0 Render date: 2024-12-26T19:51:49.694Z Has data issue: false hasContentIssue false

Chapter 11 - Evaluating Experimental Research

Published online by Cambridge University Press:  19 December 2019

By
Henry L. Roediger  and
Jeremy K. Yamashiro
Edited by
Robert J. Sternberg  and
Diane F. Halpern
Robert J. Sternberg
Affiliation:
Cornell University, New York
Diane F. Halpern
Affiliation:
Claremont McKenna College, California
Get access

Summary

Experiments allow researchers to determine cause and effect relations between variables. As such, they are a critical component in the advance of scientific psychology. In this chapter, we discuss the theory behind the design of good experiments, and provide a sample study for evaluation. We outline three important types of replication, and give an overview of historical events that led to a renewed vigilance regarding replicability. Finally, we discuss generalizability of research in terms of four factors: Subjects (or participants), materials used in the experiment, dependent measures, and the experimental situation. Effects that generalize across these sets of factors are robust. We end the chapter with a set of 18 critical-thinking questions that should be borne in mind while reading and evaluating experimental research. Referring to these questions will help to sharpen critical thinking skills about experimental research.

Keywords

subjectssystematic replicationvariablewithin subject design
Type
Chapter
Information
Critical Thinking in Psychology , pp. 249 - 279
Publisher: Cambridge University Press
Print publication year: 2020

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Aarts, A. A., et al. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. DOI:10.1126/science.aac4716Google Scholar
Bacon, F. (1620/2000)The new organon. Eds. Jardine, L. & Silverthorne, M.. Originally published as Novum Organum, sive indicia vera de Interpretationes Naturae in 1620. Cambridge: Cambridge University Press.Google Scholar
Banaji, M. R., & Crowder, R. G. (1989). The bankruptcy of everyday memory. American Psychologist, 44, 1185–1193. DOI:10.1037/0003-066X.44.9.1185CrossRefGoogle Scholar
Bem, D. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100(3), 407–425. DOI:10.1037/a0021524CrossRefGoogle ScholarPubMed
Bones, A. K. (2012). We knew the future all along: Scientific hypothesizing is much more accurate than other forms of precognition – A satire in one part. Perspectives on Psychological Science, 7(3), 307–309. DOI:10.1177/1745691612441216CrossRefGoogle ScholarPubMed
Boring, E. G. (1929). A history of experimental psychology. New York and London: Century.Google Scholar
Camerer, C. F., et al. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature: Human Behavior, 2, 637–644. DOI:10.1038/s41562-018–0399–zGoogle Scholar
Craik, F. I. M., & Tulving, E. (1975). Depth of processing and the retention of words in episodic memory. Journal of Experimental Psychology: General, 104(3), 263–294. DOI:10.1037/0096–3445.104.3.268Google Scholar
Cummings, G. (2012). Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. New York: Routledge.Google Scholar
Deese, J. (1959). On the prediction of occurrence of particular verbal instructions in immediate recall. Journal of Experimental Psychology, 58, 17–22. DOI:10.1037/h0046671Google Scholar
Doyen, S., Klein, O., Pichon, C. L., & Cleeremans, A. (2012). Behavioral priming: It’s all in the mind, but whose mind? PloS ONE, 7(1), e2908. DOI:10.1371/journal.pone.0029081Google Scholar
Freyd, J. J., & Gleaves, D. H. (1996). “Remembering” words not presented in lists: Relevance to the current recovered/false memory controversy. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22, 811–813. DOI:10.1037/0278–7393.22.3.811Google Scholar
Ganley, C. M., Mingle, L. A., Ryan, A. M., Ryan, K., Vasilyeva, M., & Perry, M. (2013). An examination of stereotype threat effect on girls’ mathematics performance. Developmental Psychology, 49(10), 1886–1897. DOI:10.1037/a0031412Google Scholar
Gallo, D. A. (2010). False memories and fantastic beliefs: 15 years of the DRM illusion. Memory & Cognition, 37, 833–848. DOI:10.3758/MC.38.7.833Google Scholar
Gates, A. I. (1917). Recitation as a factor in memorizing. Archives of Psychology, 26, 1–104.Google Scholar
Gelman, A., & Loken, E. (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time. Online. http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdfGoogle Scholar
Hagger, M. S., et al. (2016). A multilab preregistered replication of the ego-depletion effect. Perspectives on Psychological Science, 11(4), 546–573. DOI:10.1177/1745691616652873Google Scholar
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015). The extent and consequences of P-hacking in science. PloS Biology, 13(3), e1002106. DOI:10.1371/journal.pbio.1002106CrossRefGoogle ScholarPubMed
Ioannidis, J. P. A. (2005). Why most published research findings are false. PloS Medicine, 2(8), 0696–0701. DOI:10.1371/journal.pmed.0020124Google Scholar
Jenkins, J. J. (1979). Four points to remember: A tetrahedral model of memory experiments. In Cermak, L. S. & Craik, F. I. M. (Eds.), Levels of processing in human memory (pp. 429–446). Hillsdale, NJ: Erlbaum.Google Scholar
Kahneman, D. (2012). A proposal to deal with questions about priming effects. Online. https://tinyurl.com/yyj8v7nxGoogle Scholar
Kantowitz, B. H., Roediger, III, H. L., & Elmes, D. (2015). Experimental psychology (10th ed.). Belmont, CA: Wadsworth.Google Scholar
Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196–217. DOI:10.1207/s15327957pspr0203_4CrossRefGoogle ScholarPubMed
Klein, R. A., et al. (2014). Investigating variation in replicability: A “many labs” replication project. Social Psychology, 45(3), 142–152. DOI:10.1027/1864-9335/a000178Google Scholar
McDermott, K. B., & Naaz, F. (2014). Is recitation an effective tool for adult learners? Journal of Applied Research in Memory and Cognition, 3, 207–213. DOI:10.1016/j.jarmac.2014.06.006Google Scholar
Mook, D. G. (1983). In defense of external invalidity. American Psychologist, 38, 379–387. DOI:10.1037/0003-066X.38.4.379CrossRefGoogle Scholar
Pashler, H., & Wagenmakers, E. J. (2012). Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science, 7(6), 528–530. DOI:10.1177/1745691612465253Google Scholar
Piaget, J. (1962). Play, dreams, and imitation in childhood. New York: Norton.Google Scholar
Popper, K. (2002). The logic of scientific discovery. New York: Routledge.Google Scholar
Ranehill, E., Dreber, A., Johannesson, M., Leiberg, S., Sul, S., & Weber, R. A. (2015). Assessing the robustness of power posing: No effect on hormones and risk tolerance in a large sample of men and women. Psychological Science, 26(5), 653–656. DOI:10.1177/0956797614553946CrossRefGoogle Scholar
Roediger, III, H. L. (2012). Psychology’s woes and a partial cure: The value of replication. The APS Observer, 25(2), 9, 27–29.Google Scholar
Roediger, III, H. L., & Gallo, D. A. (2002). Processes affecting accuracy and distortion in memory: An overview. In Eisen, M. L., Goodman, G. S., & Quas, J. A. (Eds.), Memory and suggestibility in the forensic interview (pp. 3–28). Mahwah, NJ: Erlbaum.Google Scholar
Roediger, III, H. L., & Gallo, D. A. (2016). Associative memory illusions. In Pohl, R. F. (Ed.), Cognitive illusions: Intriguing phenomena in thinking, judgment and memory (2nd ed.) (pp. 390–405). New York: Psychology Press.Google Scholar
Roediger, III, H. L., & Karpicke, J. D. (2006). Test-enhanced learning: Taking memory tests improves long-term retention. Psychological Science, 17, 249–255. DOI:10.1111/j.1467–9280.2006.01693.xGoogle Scholar
Roediger, III, H. L., & McCabe, D. P. (2007) Evaluating Experimental Research: Critical Issues. In Sternberg, R. J., Roediger, III H. L., & Halpern, D. F. (Eds.), Critical thinking in psychology (pp. 15–36). New York: Cambridge University Press.Google Scholar
Roediger, III, H. L., & McDermott, K. (1995). Creating false memories: Remembering words not presented in lists. Journal of Experimental Psychology: Learning, Memory, and Cognition, 21, 803–814. DOI:10.1037/0278–7393.21.4.803Google Scholar
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. DOI:10.1177/0956797611417632Google Scholar
Spellman, B. A. (2015). A short (personal) future history of Revolution 2.0. Perspectives on Psychological Science, 10(6), 886–899. DOI:10.1177/1745691615609918Google Scholar
Tulving, E. (1967). The effects of presentation and recall of material in free-recall learning. Journal of Verbal Learning and Verbal Behavior, 6, 175–184. DOI:10.1016/S0022–5371(67)80092–6Google Scholar
Tulving, E. (1985). Memory and consciousness. Canadian Psychology, 26, 1–12. DOI:10.1037/h0080017CrossRefGoogle Scholar
Underwood, B. J. (1965). False recognition produced by implicit verbal responses. Journal of Experimental Psychology, 70, 122–129. DOI:10.1037/h0022014Google Scholar

Save book to Kindle

To save this book to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×