Hostname: page-component-745bb68f8f-v2bm5 Total loading time: 0 Render date: 2025-01-13T13:30:56.295Z Has data issue: false hasContentIssue false
  • English
  • Français

Outliers in L2 Research in Applied Linguistics: A Synthesis and Data Re-Analysis

Published online by Cambridge University Press:  30 June 2020

Christopher Nicklin  and
Luke Plonsky
Christopher Nicklin*
Affiliation:
Temple University, Japan and Northern Arizona University
Luke Plonsky
Affiliation:
Temple University, Japan and Northern Arizona University
*
*Corresponding author. E-mail: christophernicklin79@gmail.com
Get access Rights & Permissions [Opens in a new window]

Abstract

Data from self-paced reading (SPR) tasks are routinely checked for statistical outliers (Marsden, Thompson, & Plonsky, 2018). Such data points can be handled in a variety of ways (e.g., trimming, data transformation), each of which may influence study results in a different manner. This two-phase study sought, first, to systematically review outlier handling techniques found in studies that involve SPR and, second, to re-analyze raw data from SPR tasks to understand the impact of those techniques. Toward these ends, in Phase I, a sample of 104 studies that employed SPR tasks was collected and coded for different outlier treatments. As found in Marsden et al. (2018), wide variability was observed across the sample in terms of selection of time and standard deviation (SD)-based boundaries for determining what constitutes a legitimate reading time (RT). In Phase II, the raw data from the SPR studies in Phase I were requested from the authors. Nineteen usable datasets were obtained and re-analyzed using data transformations, SD boundaries, trimming, and winsorizing, in order to test their relative effectiveness for normalizing SPR reaction time data. The results suggested that, in the vast majority of cases, logarithmic transformation circumvented the need for SD boundaries, which blindly eliminate or alter potentially legitimate data. The results also indicated that choice of SD boundary had little influence on the data and revealed no meaningful difference between trimming and winsorizing, implying that blindly removing data from SPR analyses might be unnecessary. Suggestions are provided for future research involving SPR data and the handling of outliers in second language (L2) research more generally.

Type
Research Article
Information
Annual Review of Applied Linguistics , Volume 40 , March 2020 , pp. 26 - 55
Check for updates
Open Practices
Open materials Open data
Copyright
Copyright © The Author(s), 2020. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

The experiment in this article earned an Open Data and an Open Materials badge for transparent practices. The data and materials are available at https://www.iris-database.org/iris/app/home/detail?id=york%3a937594&ref=search.

References

Aguinis, H., Gottfredson, R. K., & Joo, H. (2013). Best-practice recommendations for defining, identifying, and handling outliers. Organizational Research Methods, 16(2), 270–301. doi:10.1177/1094428112470848CrossRefGoogle Scholar
Aguinis, H., & Joo, H. (2015). Debunking myths and urban legends about how to identify influential outliers. In Lance, C. E. & Vandenberg, R. J. (Eds.), More statistical and methodological myths and urban legends (pp. 206–223). Routledge.Google Scholar
Al-Hoorie, A. H., & Vitta, J. P. (2019). The seven sins of L2 research: A review of 30 journals’ statistical quality and their CiteScore, SJR, SNIP, JCR Impact Factors. Language Teaching Research, 23(6), 727–744. doi: 10.1177/1362168818767191CrossRefGoogle Scholar
American Psychological Association (2017, January 1). Ethical principles of psychologists and code of conduct. Retrieved from https://www.apa.org/ethics/code/manual-updates.aspxGoogle Scholar
Baayen, R. H. (2008). Analyzing linguistic data: A practical introduction to statistics using R. Cambridge University Press.CrossRefGoogle Scholar
Baayen, R. H., & Milin, P. (2010). Analyzing reaction times. International Journal of Psychological Research, 3(2), 12–28. doi:10.21500/20112084.807CrossRefGoogle Scholar
Bakker, M., & Wicherts, J. M. (2014). Outlier removal, sum scores, and the inflation of the Type I error rate in independent samples t tests: The power of alternatives and recommendations. Psychological Methods, 19(3), 409–427. doi:10.1037/met0000014CrossRefGoogle Scholar
Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68(3), 255–278. doi:10.1016/j.jml.2012.11.001CrossRefGoogle ScholarPubMed
Barnett, V., & Lewis, T. (1994). Outliers in statistical data. Wiley.Google Scholar
Bates, D., Maechler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. doi:10.18637/jss.v067.i01CrossRefGoogle Scholar
Box, G. E. P., & Cox, D. R. (1964). An analysis of transformations. Journal of the Royal Statistical Society, 26(2), 211–252. doi:10.1111/j.2517-6161.1964.tb00553.xGoogle Scholar
Brysbaert, M. (2019). How many participants do we have to include in properly powered experiments? A tutorial of power analysis with reference tables. Journal of Cognition, 2(1), 1–38. doi:10.5334/joc.72CrossRefGoogle ScholarPubMed
Cook, S. V. (2018). Gender matters: From L1 grammar to L2 semantics. Bilingualism: Language and Cognition, 21(1), 13–31. doi:10.1017/S1366728916000766CrossRefGoogle Scholar
Cortina, J. M. (2002). Big things have small beginnings: An assortment of “minor” methodological misunderstandings. Journal of Management, 28(3), 339–362. doi:10.1177/014920630202800305CrossRefGoogle Scholar
Cousineau, D., & Chartier, S. (2010). Outliers detection and treatment: A review. International Journal of Psychological Research, 3(1), 58–67. doi:10.21500/20112084.844CrossRefGoogle Scholar
Craig, J. R., & Reese, S. C. (1973). Retention of raw data: A problem revisited. American Psychologist, 28(8), 723. doi:10.1037/h0035667CrossRefGoogle Scholar
Cunnings, I. (2012). An overview of mixed-effects statistical models for second language researchers. Second Language Research, 28(3), 369–382. doi:10.1177/0267658312443651CrossRefGoogle Scholar
Dekydtspotter, L., & Outcalt, S. D. (2005). A syntactic bias in scope ambiguity resolution in the processing of English-French cardinality interrogatives: Evidence for informational encapsulation. Language Learning, 55(1), 1–36. doi:10.1111/j.0023-8333.2005.00288.xCrossRefGoogle Scholar
Derrick, D. J. (2016). Instrument reporting practices in second language research. TESOL Quarterly, 50(1), 132–153. doi:10.1002/tesq.217CrossRefGoogle Scholar
Dong, Y., Wen, Y., Zeng, X., & Ji, Y. (2015). Exploring the cause of English pronoun gender errors by Chinese learners of English: Evidence from the self-paced reading paradigm. Journal of Psycholinguistic Research, 44(6), 733–747. doi:10.1007/s10936-014-9314-6CrossRefGoogle ScholarPubMed
Dussias, P. E., & Piñar, P. (2010). Effects of reading span and plausibility in the reanalysis of wh-gaps by Chinese-English second language speakers. Second Language Research, 26(4), 443–472. doi:10.1177/0267658310373326CrossRefGoogle Scholar
Field, A. (2018). Discovering statistics using IBM SPSS Statistics (5th Ed.). Sage.Google Scholar
Fine, A. B., Jaeger, T. F., Farmer, T. A., & Qian, T. (2013). Rapid expectation adaptation during syntactic comprehension. PLOS ONE, 8(10), e77661. doi:10.1371/journal.pone.0077661CrossRefGoogle ScholarPubMed
Frank, S. L., Trompenaars, T., & Vasishth, S. (2016). Cross-linguistic differences in processing double-embedded relative clauses: Working-memory constraints or language statistics? Cognitive Science, 40(3), 554–578. doi:10.1111/cogs.12247CrossRefGoogle ScholarPubMed
Gass, S., Loewen, S., & Plonsky, L. (in press). Coming of age: The past, present, and future of quantitative SLA research. Language Teaching.Google Scholar
Gelman, A., & Hill, J. (2007). Data analysis using regression and multilevel/hierarchical models. Cambridge University Press.Google Scholar
Gerth, S., Otto, C., Felser, C., & Nam, Y. (2017). Strength of garden-path effects in native and non-native speakers’ processing of object–subject ambiguities. International Journal of Bilingualism, 21(2), 125–144. doi:10.1177/1367006915604401CrossRefGoogle Scholar
Glass, G. V., Peckham, P. D., & Sanders, J. R. (1972). Consequences of failure to meet assumptions underlying the fixed effects analyses of variance and covariance. Review of Educational Research, 42(3), 237–288. doi:10.3102/00346543042003237CrossRefGoogle Scholar
Godfroid, A. (2020). Eye tracking in second language acquisition and bilingualism: A research synthesis and methodological guide. Routledge.Google Scholar
Hopp, H. (2016). The timing of lexical and syntactic processes in second language sentence comprehension. Applied Psycholinguistics, 37(5), 1253–1280. doi:10.1017/S0142716415000569CrossRefGoogle Scholar
Hsu, C.-H., Lee, C.-Y., & Marantz, A. (2011). Effects of visual complexity and sublexical information in the occipitotemporal cortex in the reading of Chinese phonograms: A single-trial analysis with MEG. Brain and Language, 117(1), 1–11. doi:10.1016/j.bandl.2010.10.002CrossRefGoogle ScholarPubMed
Hu, Y., & Plonsky, L. (2019). Statistical assumptions in L2 research: A systematic review. Second Language Research. Advance online publication. doi:10.1177/0267658319877433CrossRefGoogle Scholar
Jackson, C. (2008). Proficiency level and the interaction of lexical and morphosyntactic information during L2 sentence processing. Language Learning, 58(4), 875–909. doi:10.1111/j.1467-9922.2008.00481.xCrossRefGoogle Scholar
Jackson, C. N., & Roberts, L. (2010). Animacy affects the processing of subject–object ambiguities in the second language: Evidence from self-paced reading with German second language learners of Dutch. Applied Psycholinguistics, 31(4), 671–691. doi:10.1017/S0142716410000196CrossRefGoogle Scholar
Jegerski, J. (2016). Number attraction effects in near-native Spanish sentence comprehension. Studies in Second Language Acquisition, 38(1), 5–33. doi:10.1017/S027226311400059XCrossRefGoogle Scholar
Jegerski, J. (2018). Sentence processing in Spanish as a heritage language: A self-paced reading study of relative clause attachment: Sentence processing in Spanish as a heritage language. Language Learning, 68(3), 598–634. doi:10.1111/lang.12289CrossRefGoogle Scholar
Jiang, N. (2007). Selective integration of linguistic knowledge in adult second language learning. Language Learning, 57(1), 1–33. doi:10.1111/j.1467-9922.2007.00397.xCrossRefGoogle Scholar
Jiang, N. (2012). Conducting reaction time research in second language studies. New York: Routledge.Google Scholar
Johnson, A., Fiorentino, R., & Gabriele, A. (2016). Syntactic constraints and individual differences in native and non-native processing of wh-movement. Frontiers in Psychology, 7, 549. doi:10.3389/fpsyg.2016.00549CrossRefGoogle ScholarPubMed
Judd, C. M., & McClelland, G. H. (1989). Data analysis: A model comparison approach. Harcourt, Brace, Jovanovich.Google Scholar
Juffs, A. (1998). Main verb versus reduced relative clause ambiguity resolution in L2 sentence processing. Language Learning, 48(1), 107–147. doi:10.1111/1467-9922.00034CrossRefGoogle Scholar
Juffs, A. (2005). The influence of first language on the processing of wh-movement in English as a second language. Second Language Research, 21(2), 121–151. doi:10.1191/0267658305sr255oaCrossRefGoogle Scholar
Juffs, A., & Harrington, M. (1996). Garden path sentences and error data in second language sentence processing. Language Learning, 46(2), 283–323. doi:10.1111/j.1467-1770.1996.tb01237.xCrossRefGoogle Scholar
Keselman, H. J., Wilcox, R. R., Othman, A. R., & Fradette, K. (2002). Trimming, transforming statistics, and bootstrapping: Circumventing the biasing effects of heteroscedasticity and non-normality. Journal of Modern Applied Statistics methods, 1(2), 288–309. doi:10.22237/jmasm/1036109820CrossRefGoogle Scholar
Kim, H. (2018). Second language processing of Korean floating numeral quantifiers. Journal of Psycholinguistic Research, 47(5), 1101–1119. doi:10.1007/s10936-018-9581-8CrossRefGoogle ScholarPubMed
Kim, J. H., & Christianson, K. (2017). Working memory effects on L1 and L2 processing of ambiguous relative clauses by Korean L2 learners of English. Second Language Research, 33(3), 365–388. doi:10.1177/0267658315623322CrossRefGoogle Scholar
Kim, M., Crossley, S. A., & Skalicky, S. (2018). Effects of lexical features, textual properties, and individual differences on word processing times during second language reading comprehension. Reading and Writing, 31(5), 1155–1180. doi:10.1007/s11145-018-9833-xCrossRefGoogle Scholar
Kruskal, W. H. (1960). Some remarks on wild observations. Technometrics, 2(1), 1–3. doi:10.2307/1266526CrossRefGoogle Scholar
Lachaud, C. M., & Renaud, O. (2011). A tutorial for analyzing human reaction times: How to filter data, manage missing values, and choose a statistical model. Applied Psycholinguistics, 32(2), 389–416. doi:10.1017/S0142716410000457CrossRefGoogle Scholar
Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4. doi:10.3389/fpsyg.2013.00863CrossRefGoogle ScholarPubMed
Larson-Hall, J., & Herrington, R. (2010). Improving data analysis in second language acquisition by utilizing modern developments in applied statistics. Applied Linguistics 31(3) 368–390. doi:10.1093/applin/amp038CrossRefGoogle Scholar
Leal, T., Slabakova, R., & Farmer, T. A. (2017). The fine tuning of linguistic expectations over the course of L2 learning. Studies in Second Language Acquisition, 39(3), 493–525. doi:10.1017/S0272263116000164CrossRefGoogle Scholar
Linck, J. A., & Cunnings, I. (2015). The utility and application of mixed-effects models in second language research. Language Learning, 65(S1), 185–207. doi:10.1111/lang.12117CrossRefGoogle Scholar
Litcofsky, K. A., & Van Hell, J. G. (2017). Switching direction affects switching costs: Behavioral, ERP and time-frequency analyses of intra-sentential codeswitching. Neuropsychologia, 97, 112–139. doi:10.1016/j.neuropsychologia.2017.02.002CrossRefGoogle ScholarPubMed
Liu, Q., & Brown, D. (2015). Methodological synthesis of research on the effectiveness of corrective feedback in L2 writing. Journal of Second Language Writing, 30, 66–81. doi:10.1016/j.jslw.2015.08.011CrossRefGoogle Scholar
Lo, S., & Andrews, S. (2015). To transform or not to transform: Using generalized linear mixed models to analyze reaction time data. Frontiers in Psychology, 6. doi:10.3389/fpsyg.2015.01171CrossRefGoogle ScholarPubMed
Luce, R. D. (1991). Response times: Their role in inferring elementary mental organization. Oxford University Press.CrossRefGoogle Scholar
Maienschein, J., Parker, J. N., Laubichler, M., & Hackett, E. J. (2019). Data management and data sharing in science and technology studies. Science, Technology, & Human Values, 44(1), 143–160. doi:10.1177/0162243918798906CrossRefGoogle Scholar
Marinis, T., Roberts, L., Felser, C., & Clahsen, H. (2005). Gaps in second language sentence processing. Studies in Second Language Acquisition, 27(1), 53–78. doi:10.1017/S0272263105050035CrossRefGoogle Scholar
Marsden, E., Mackey, A., & Plonsky, L. (2016). The IRIS Repository: Advancing research practice and methodology. In Mackey, A. & Marsden, E. (Eds.), Advancing methodology and practice: The IRIS Repository of instruments for research into second languages (pp. 1–21). New York: Routledge.Google Scholar
Marsden, E., Morgan-Short, K., Trofimovich, P., & Ellis, N. C. (2018). Introducing registered reports at Language Learning: Promoting transparency, replication, and a synthetic ethic in the language sciences. Language Learning, 68, 309–320. doi: 10.1111/lang.12284CrossRefGoogle Scholar
Marsden, E., & Plonsky, L. (2018). Data, open science, and methodological reform in second language acquisition research. In Gudmestad, A., & Edmonds, A. (Eds.), Critical reflections on data in second language acquisition (pp. 219–228). John Benjamins.CrossRefGoogle Scholar
Marsden, E., Thompson, S., & Plonsky, L. (2018). A methodological synthesis of self-paced reading in second language research. Applied Psycholinguistics, 39(5), 861–904. doi:10.1017/S0142716418000036CrossRefGoogle Scholar
Norouzian, R., & Plonsky, L. (2018). Eta- and partial eta-squared in L2 research: A cautionary review and guide to more appropriate usage. Second Language Research, 34(2), 257–271. doi:10.1177/0267658316684904CrossRefGoogle Scholar
Orr, J. M., Sackett, P. R., & Dubois, C. L. Z. (1991). Outlier detection and treatment in I/O psychology: A survey of researcher beliefs and an empirical illustration. Personnel Psychology, 44(3), 473–486. doi:10.1111/j.1744-6570.1991.tb02401.xCrossRefGoogle Scholar
Osborne, J. W. (2002). Notes on the use of data transformations. Practical Assessment, Research & Evaluation, 8(6), 1–7. Retrieved from https://pareonline.net/getvn.asp?v=8&n=6Google Scholar
Osborne, J. W., & Overbay, A. (2004). The power of outliers (and why researchers should ALWAYS check for them). Practical Assessment, Research & Evaluation, 9(6), 1–8. Retrieved from https://pareonline.net/getvn.asp?v=9&n=6Google Scholar
Pek, J., Wong, O., & Wong, C. M. (2017). Data transformations for inference with linear regression: Clarifications and recommendations. Practical Assessment, Research & Evaluation, 22(9), 1–11. Retrieved from https://pareonline.net/getvn.asp?v=22&n=9Google Scholar
Plonsky, L. (2013). Study quality in SLA: An assessment of designs, analyses, and reporting practices in quantitative L2 research. Studies in Second Language Acquisition, 35(4), 655–687. doi:10.1017/S0272263113000399CrossRefGoogle Scholar
Plonsky, L., & Derrick, D. J. (2016). A meta-analysis of reliability coefficients in second language research. The Modern Language Journal, 100(2), 538–553. doi:10.1111/modl.12335CrossRefGoogle Scholar
Plonsky, L., Egbert, J., & LaFlair, G. T. (2015). Bootstrapping in applied linguistics: Assessing its potential using shared data. Applied Linguistics, 36(5), 591–610. doi:10.1093/applin/amu001Google Scholar
Plonsky, L., & Gass, S. (2011). Quantitative research methods, study quality, and outcomes: The case of interaction research. Language Learning, 61(2), 325–366. doi:10.1111/j.1467-9922.2011.00640.xCrossRefGoogle Scholar
Plonsky, L., & Ghanbar, H. (2018). Multiple regression in L2 research: A methodological synthesis and guide to interpreting R 2 values. The Modern Language Journal, 102(4), 713–731. doi:10.1111/modl.12509CrossRefGoogle Scholar
Plonsky, L., & Gonulal, T. (2015). Methodological synthesis in quantitative L2 research: A review of reviews and a case study of exploratory factor analysis. Language Learning, 65(S1), 9–36. doi:10.1111/lang.12111CrossRefGoogle Scholar
Plonsky, L., Marsden, E., Crowther, D., Gass, S., & Spinner, P. (2019). A methodological synthesis of judgement tasks in second language research. Second Language Research. Advance online publication. doi:10.1177/0267658319828413CrossRefGoogle Scholar
Plonsky, L., & Oswald, F. L. (2014). How big is “big”? Interpreting effect sizes in L2 research. Language Learning, 64(4), 878–912. doi:10.1111/lang.12079CrossRefGoogle Scholar
R Core Team (2018). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/Google Scholar
Ratcliff, R. (1993). Methods for dealing with reaction time outliers. Psychological Bulletin, 114(3), 510–532. doi:10.1037/0033-2909.114.3.510CrossRefGoogle ScholarPubMed
Rayner, K., & Pollatsek, A. (1989). The psychology of reading. Englewood Cliffs: Prentice Hall.Google ScholarPubMed
Revelle, W. (2019). psych: Procedures for Personality and Psychological Research, Northwestern University, Evanston, Illinois, USA. https://CRAN.R-project.org/package=psychVersion=1.8.4.Google Scholar
Roth, P. L., & Switzer, F. S. III (2004). Outliers and influential cases: Handling those discordant contaminated maverick rogues. In Rogelberg, S. L. (Ed.), The handbook of research methods in industrial and organizational psychology (pp. 297–309). Hoboken: Wiley-Blackwell.Google Scholar
Sagarra, N., & Herschensohn, J. (2010). The role of proficiency and working memory in gender and number agreement processing in L1 and L2 Spanish. Lingua, 120(8), 2022–2039. doi:10.1016/j.lingua.2010.02.004CrossRefGoogle Scholar
Sagarra, N., & Herschensohn, J. (2011). Proficiency and animacy effects on L2 gender agreement processes during comprehension: Proficiency, animacy, and L2 gender concord. Language Learning, 61(1), 80–116. doi:10.1111/j.1467-9922.2010.00588.xCrossRefGoogle Scholar
Sagarra, N., & Herschensohn, J. (2012). Processing of gender and number agreement in late Spanish bilinguals. International Journal of Bilingualism, 17(5), 607–627. doi:10.1177/1367006912453810CrossRefGoogle Scholar
Schramm, P., & Rouder, J. (2019, March 5). Are reaction time transformations really beneficial? doi.org/10.31234/osf.io/9ksa6CrossRefGoogle Scholar
Seltman, H. (n.d.). Function for obtaining SES and SEK. Retrieved from http://www.stat.cmu.edu/~hseltman/files/spssSkewKurtosis.RGoogle Scholar
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. doi:10.1177/0956797611417632CrossRefGoogle ScholarPubMed
Sok, S., Kang, E. Y., & Han, Z. H. (2019). Thirty-five years of ISLA on form-focused instruction: A methodological synthesis. Language Teaching Research, 23(4). doi:10.1177/1362168818776673CrossRefGoogle Scholar
Streiner, D. L. (2018). Dealing with outliers. Journal of Clinical Psychopharmacology, 38(3), 170–171. doi:10.1097/JCP.0000000000000865CrossRefGoogle ScholarPubMed
Tabachnick, B. G., & Fidell, L. S. (2013). Using multivariate statistics. Boston: Pearson.Google Scholar
Tamura, Y., Fukuta, J., Nishimura, Y., Harada, Y., Hara, K., & Kato, D. (2019). Japanese EFL learners’ sentence processing of conceptual plurality: An analysis focusing on reciprocal verbs. Applied Psycholinguistics, 40(1), 59–91. doi:10.1017/S0142716418000450CrossRefGoogle Scholar
Vafaee, P., Suzuki, Y., & Kachisnke, I. (2017). Validating grammaticality judgement tests: Evidence from two new psycholinguistic measures. Studies in Second Language Acquisition. 39(1), 59–95. doi:10.1017/S0272263115000455CrossRefGoogle Scholar
VanPatten, B., & Smith, M. (2019). Word-order typology and the acquisition of case marking: A self-paced reading study in Latin as a second language. Second Language Research, 35(3), 397–420. doi:10.1177/0267658318785652CrossRefGoogle Scholar
Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006). The poor availability of psychological research data for reanalysis. American Psychologist, 61(7), 726–728. doi:10.1037/0003-066X.61.7.726CrossRefGoogle ScholarPubMed
Williams, J. N., Möbius, P., & Kim, C. (2001). Native and non-native processing of English wh- questions: Parsing strategies and plausibility constraints. Applied Psycholinguistics, 22(04), 509–540. doi:10.1017/S0142716401004027CrossRefGoogle Scholar
Williams, M. N., Grajales, C. A. G., Kurkiewicz, D. (2013). Assumptions of multiple regression: Correcting two misconceptions. Practical Assessment, Research & Evaluation, 18(11), 1–14. Retrieved from https://pareonline.net/getvn.asp?v=18&n=11Google Scholar
Wolins, L. (1962). Responsibility for raw data. American Psychologist, 17(9), 657–658. doi:10.1037/h0038819CrossRefGoogle Scholar
Supplementary material: File

Nicklin and Plonsky supplementary material

Nicklin and Plonsky supplementary material

Download Nicklin and Plonsky supplementary material(File)
File 68.2 KB