Hostname: page-component-7f64f4797f-kg7gq Total loading time: 0.001 Render date: 2025-11-04T14:42:43.026Z Has data issue: false hasContentIssue false
  • English
  • Français

One by One: Accumulating Evidence by using Meta-Analytical Procedures for Single-Case Experiments

Published online by Cambridge University Press:  23 November 2017

Patrick Onghena ,
Bart Michiels ,
Laleh Jamshidi ,
Mariola Moeyaert  and
Wim Van den Noortgate
Patrick Onghena*
Affiliation:
Faculty of Psychology and Educational Sciences, KU Leuven, University of Leuven, Leuven, Belgium
Bart Michiels
Affiliation:
Faculty of Psychology and Educational Sciences, KU Leuven, University of Leuven, Leuven, Belgium
Laleh Jamshidi
Affiliation:
Faculty of Psychology and Educational Sciences, KU Leuven, University of Leuven, Leuven, Belgium
Mariola Moeyaert
Affiliation:
Department of Educational and Counseling Psychology, Division of Educational Psychology and Methodology, School of Education, University at Albany, SUNY, Albany, USA
Wim Van den Noortgate
Affiliation:
Faculty of Psychology and Educational Sciences, KU Leuven, University of Leuven, Leuven, Belgium
*
Address for correspondence: Patrick Onghena, Faculty of Psychology and Educational Sciences, KU Leuven, University of Leuven, Tiensestraat 102, BE-3000 Leuven, Belgium. E-mail: patrick.onghena@kuleuven.be
Get access

Abstract

This paper presents a unilevel and multilevel approach for the analysis and meta-analysis of single-case experiments (SCEs). We propose a definition of SCEs and derive the specific features of SCEs’ data that have to be taken into account when analysing and meta-analysing SCEs. We discuss multilevel models of increasing complexity and propose alternative and complementary techniques based on probability combining and randomisation test wrapping. The proposed techniques are demonstrated with real-life data and corresponding R code.

Keywords

single-case experimentsingle-case experimental designN-of-1 trialmeta-analysismultilevel modelprobability combiningrandomisation testpermutation test

Information

Type
Articles
Information
Brain Impairment , Volume 19 , Special Issue 1: Quantitative Data Analysis; by Robyn Tate and Michael Perdices , March 2018 , pp. 33 - 58
Copyright
Copyright © Australasian Society for the Study of Brain Impairment 2017 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Aiken, L.S., West, S.G., & Pitts, S.C. (2003). Multiple linear regression. In Schinka, J. & Velicer, W. (Eds.), Comprehensive handbook of psychology: Vol. 2. Research methods in psychology (pp. 481–507. New York, NY: Wiley.Google Scholar
Anumendem, N.D., De Fraine, B., Onghena, P., & Van Damme, J. (2013). The impact of coding time on the estimation of school effects. Quality & Quantity, 47, 1021–1040. doi:10.1007/s11135-011-9581-3.Google Scholar
Araujo, A., Julious, S., & Senn, S. (2016). Understanding variation in sets of N-of-1 trials. PLoS ONE, 11 (12), e0167167. doi:10.1371/journal.pone.0167167.CrossRefGoogle ScholarPubMed
Baek, E., Moeyaert, M., Petit-Bois, M., Beretvas, S., Van den Noortgate, W., & Ferron, J. (2014). The use of multilevel analysis for integrating single-case experimental design results within a study and across studies. Neuropsychological Rehabilitation, 24, 590–606. doi:10.1080/09602011.2013.835740.Google Scholar
Barlow, D.H., Nock, M.K., & Hersen, M. (2009). Single case experimental designs: Strategies for studying behavior change (3rd ed.). Boston, MA: Pearson.Google Scholar
Bates, D.M. (2010). Lme4: Mixed-effects modeling with R. Springer. Retrieved from http://lme4.r-forge.r-project.org/book/.Google Scholar
Beretvas, S.N., & Chung, H. (2008). A review of meta-analyses of single-subject experimental designs: Methodological issues and practice. Evidence-Based Communication Assessment and Intervention, 2, 129–141. doi:10.1080/17489530802446302.CrossRefGoogle Scholar
Booth, A., Clarke, M., Dooley, G., Ghersi, D., Moher, D., Petticrew, M., & Stewart, L. (2012). The nuts and bolts of PROSPERO: An international prospective register of systematic reviews. Systematic Reviews, 1, 2. doi:10.1186/2046-4053-1-2.Google Scholar
Borenstein, M., Hedges, L.V., Higgins, J.P.T., & Rothstein, H.R. (2009). Introduction to meta-analysis. Chichester, UK: Wiley. doi:10.1002/ 9780470743386.ch43.CrossRefGoogle Scholar
Box, G.E.P., Jenkins, G.M., Reinsel, G.C., & Ljung, G.M. (2016). Time series analysis: Forecasting and control (5th ed.). New York, NY: Wiley.Google Scholar
Brosnan, J., Moeyaert, M., Brooks Newsome, K., Healy, O., Heyvaert, M., Onghena, P., & Van den Noortgate, W. (2017). Multilevel analysis of multiple-baseline data evaluating precision teaching as an intervention for improving fluency in foundational reading skills for at risk readers. Exceptionality. doi:10.1080/09362835.2016.1238378.Google Scholar
Bulté, I., & Onghena, P. (2008). An R package for single-case randomization tests. Behavior Research Methods, 40, 467–478. doi:10.3758/BRM.40.2.467.Google Scholar
Bulté, I., & Onghena, P. (2009). Randomization tests for multiple baseline designs: An extension of the SCRT-R package. Behavior Research Methods, 41, 477–485. doi:10.3758/BRM.41.2.477.CrossRefGoogle ScholarPubMed
Bulté, I., & Onghena, P. (2013). The Single-Case Data Analysis package: Analysing single-case experiments with R software. Journal of Modern Applied Statistical Methods, 12, 450–478. doi:10.22237/jmasm/1383280020.Google Scholar
Carey, T.A., & Stiles, W.B. (2016). Some problems with randomized controlled trials and some viable alternatives. Clinical Psychology & Psychotherapy, 23, 87–95. doi: 10.1002/cpp.1942.Google Scholar
Cassell, D.L. (2002). A randomization-test wrapper for SAS® PROCs. SAS User's Group International Proceedings, 27, 108–111. Retrieved from http://www.lexjansen.com/wuss/2002/WUSS02023.pdf.Google Scholar
Chen, L.-T., Peng, C.-Y.J., & Chen, M.-E. (2015). Computing tools for implementing standards for single-case designs. Behavior Modification, 39, 835–869. doi: 10.1177/0145445515603706.CrossRefGoogle ScholarPubMed
Cohen, L.L., Feinstein, A., Masuda, A., & Vowles, K.E. (2014). Single-case research design in pediatric psychology: Considerations regarding data analysis. Journal of Pediatric Psychology, 39, 124–137. doi:10.1093/jpepsy/jst065.Google Scholar
Cools, W., Van den Noortgate, W., & Onghena, P. (2009). Design efficiency for imbalanced multilevel data. Behavior Research Methods, 41, 192–203. doi:10.3758/BRM.41.1.192.Google Scholar
Cools, W., Van den Noortgate, W., & Onghena, P. (2008). ML-DEs: A program for designing efficient multilevel studies. Behavior Research Methods, 40, 236–249. doi:10.3758/BRM.40.1.236.Google Scholar
Cooper, H., Hedges, L.V., & Valentine, J.C. (Eds.) (2009). The handbook of research synthesis and meta-analysis (2nd ed.). New York, NY: Russell Sage Foundation.Google Scholar
Dart, E.H., & Radley, K.C. (2017). The impact of ordinate scaling on the visual analysis of single-case data. Journal of School Psychology, 63, 105–118. doi:10.1016/j.jsp.2017.03.008.CrossRefGoogle ScholarPubMed
de Vries, R.M., & Morey, R.D. (2013). Bayesian hypothesis testing for single-subject designs. Psychological Methods, 18, 165–185. doi: 10.1037/ a0031037.Google Scholar
Douglas, J.M., Knox, L., De Maio, C., & Bridge, H. (2015). Improving communication-specific coping after traumatic brain injury: Evaluation of a new treatment using single-case experimental design. Brain Impairment, 15, 190–201. doi: 10.1017/BrImp.2014.25.Google Scholar
Dugard, P., File, P., & Todman, J. (2011). Single-case and small-N experimental designs. New York, NY: Routledge Academic.Google Scholar
Edgington, E.S. (1967). Statistical inference from N=1 experiments. The Journal of Psychology, 65, 195–199.CrossRefGoogle Scholar
Edgington, E.S. (1972). An additive method for combining probability values from independent experiments. The Journal of Psychology, 80, 351–363. doi:10.1080/00223980.1972.9924813.Google Scholar
Edgington, E.S. (1975). Randomization tests for one-subject operant experiments. Journal of Psychology, 90, 57–68. doi:10.1080/00223980.1975.9923926.Google Scholar
Edgington, E.S. (1980). Validity of randomization tests for one-subject experiments. Journal of Educational Statistics, 5, 235–251. doi:10.2307/1164966.CrossRefGoogle Scholar
Edgington, E.S. (1996). Randomized single-subject experimental designs. Behaviour Research and Therapy, 34, 567–574. doi:10.1016/ 0005-7967(96)00012-5.CrossRefGoogle ScholarPubMed
Edgington, E.S., & Onghena, P. (2007). Randomization tests (4th ed.). Boca Raton, FL: Chapman & Hall/CRC.CrossRefGoogle Scholar
Eysenck, H.J. (1978). An exercise in mega-silliness. American Psychologist, 33, 517. Retrieved from https://doi.org/10.1037/0003-066X.33.5.517.a.Google Scholar
Fay, M. (2010). Exact or asymptotic permutation tests: The perm package (version 1.0-0.0) in R. Retrieved from https://cran.r-project.org/web/packages/perm/perm.pdf.Google Scholar
Ferron, J.M., & Levin, J.R. (2014). Single-case permutation and randomization statistical tests: Present status, promising new developments. In Kratochwill, T.R. & Levin, J.R. (Eds.), Single-case intervention research: Methodological and statistical advances (pp. 153–183). Washington, DC: American Psychological Association.CrossRefGoogle Scholar
Ferron, J.M., Farmer, J.L., & Owens, C.M. (2010). Estimating individual treatment effects from multiple-baseline data: A Monte Carlo study of multilevel-modeling approaches. Behavior Research Methods, 42, 930–943. doi:10.3758/BRM.42.4.930.Google Scholar
Fisher, R.A. (1925). Statistical methods for research workers. Edinburgh, UK: Oliver and Boyd.Google Scholar
Fisher, R.A. (1938). Presidential address. Sankhyā: The Indian Journal of Statistics, 4, 14–17.Google Scholar
Gast, D.L., & Ledford, J.R. (2014). Single case research methodology: Applications in special education and behavioral sciences (2nd ed.). New York, NY: Routledge.Google Scholar
Gentile, J.R., Roden, A.H., & Klein, R.D. (1972). An analysis of variance model for the intrasubject replication design. Journal of Applied Behavior Analysis, 5, 193–198.Google Scholar
Guyatt, G., Jaeschke, R., & McGinn, T. (2002). PART 2B1: Therapy and validity. N-of-1 randomized controlled trials. In Guyatt, G., Rennie, D., Meade, M.O., & Cook, D.J. (Eds.), Users’ guides to the medical literature (pp. 275–290). New York, NY: McGraw-Hill.Google Scholar
Harrington, M., & Velicer, W.F. (2015). Comparing visual and statistical analysis in single-case studies using published studies. Multivariate Behavioral Research, 50, 162–183. doi:10.1080/00273171.2014.973989.Google Scholar
Hartley, H.O., & Hocking, R.R. (1971). The analysis of incomplete data. Biometrics, 27, 783–823.CrossRefGoogle Scholar
Heyvaert, M., & Onghena, P. (2014). Randomization tests for single-case experiments: State of the art, state of the science, and state of the application. Journal of Contextual Behavioral Science, 3, 51–64. doi:10.1016/j.jcbs.2013.10.002.Google Scholar
Heyvaert, M., Hannes, K., & Onghena, P. (2017). Using mixed methods research synthesis for literature reviews. Thousand Oaks, CA: Sage.Google Scholar
Heyvaert, M., Maes, B., & Onghena, P. (2010). A meta-analysis of intervention effects on challenging behaviour among persons with intellectual disabilities. Journal of Intellectual Disability Research, 54, 634–649. doi:10.1111/j.1365-2788.2010.01291.x.Google Scholar
Heyvaert, M., Maes, B., & Onghena, P. (2013). Mixed methods research synthesis: Definition, framework, and potential. Quality & Quantity, 47, 659–676. doi:10.1007/s11135-011-9538-6.Google Scholar
Heyvaert, M., Maes, B., Van den Noortgate, W., Kuppens, S., & Onghena, P. (2012). A multilevel meta-analysis of single-case and small-n research on interventions for reducing challenging behavior in persons with intellectual disabilities. Research in Developmental Disabilities, 33, 766–780. doi:10.1016/j.ridd.2011.10.010.CrossRefGoogle ScholarPubMed
Heyvaert, M., Moeyaert, M., Verkempynck, P., Van den Noortgate, W., Vervloet, M., Ugille, M., & Onghena, P. (2017). Testing the intervention effect in single-case experiments: A Monte Carlo simulation study. Journal of Experimental Education, 85, 175–196. doi:10.1080/00220973.2015.1123667.Google Scholar
Heyvaert, M., Wendt, O., Van den Noortgate, W., & Onghena, P. (2015). Randomization and data-analysis items in quality standards for single-case experimental studies. Journal of Special Education, 49, 146–156. doi:10.1177/0022466914525239.Google Scholar
Horner, R.H., Carr, E.G., Halle, J., McGee, G., Odom, S., & Wolery, M. (2005). The use of single-subject research to identify evidence-based practice in special education. Exceptional Children, 71, 165–179. doi:10.1177/001440290507100203.Google Scholar
Huitema, B.E., & McKean, J.W. (2000). Design specification issues in time-series intervention models. Educational and Psychological Measurement, 60, 38–58. doi:10.1177/00131640021970358.Google Scholar
Jacobson, K.H. (2017). Introduction to health research methods (2nd ed.). Burlington, MA: Jones & Bartlett.Google Scholar
Jamshidi, L., Heyvaert, M., Declercq, L., Fernández Castilla, B., Ferron, J. M., Moeyaert, M., . . . Van den Noortgate, W. (2017). Review of single-subject experimental design meta-analyses and reviews: 1985–2015. Manuscript submitted.Google Scholar
Jones, R.R., Weinrot, R., & Vaught, R.S. (1978). Effects of serial dependence on the agreement between visual and statistical inferences. Journal of Applied Behavioral Analysis, 11, 277–283. doi:10.1901/jaba.1978.11-277.Google Scholar
Kaptchuk, T.J. (2001). The double-blind, randomized, placebo-controlled trial: Gold standard or golden calf? Journal of Clinical Epidemiology, 54, 541–549. doi:10.1016/S0895-4356(00)00347-4.Google Scholar
Kazdin, A.E. (2011). Single-case research designs: Methods for clinical and applied settings. New York, NY: Oxford University Press.Google Scholar
Koch, G.G., & Gillings, D.B. (1984). Inference, design based vs. model based. In Johnson, N.L. & Kotz, S. (Eds.), Encyclopedia of statistical sciences, vol. 4 (pp. 84–88). New York, NY: Wiley.Google Scholar
Kratochwill, T.R., & Levin, J.R. (2010). Enhancing the scientific credibility of single-case intervention research: Randomization to the rescue. Psychological Methods, 15, 124–144. doi:10.1037/a0017736.Google Scholar
Kratochwill, T.R., & Levin, J.R. (2014). Meta- and statistical analysis of single-case intervention research data: Quantitative gifts and a wish list. Journal of School Psychology, 52, 231–235. doi:10.1016/j.jsp.2014.01.003.Google Scholar
Kratochwill, T.R., Hitchcock, J., Horner, R. H., Levin, J. R., Odom, S. L., Rindskopf, D. M., & Shadish, W. R. (2010 ). Single-case designs technical documentation. Retrieved from http://ies.ed.gov/ncee/wwc/pdf/wwc_scd.pdf.Google Scholar
Kratochwill, T.R., Hitchcock, J.H., Horner, R.H., Levin, J.R., Odom, S.L., Rindskopf, D.M., & Shadish, W.R. (2013). Single-case intervention research design standards. Remedial and Special Education, 34, 26–38. doi:10.1177/0741932512452794.Google Scholar
Kravitz, R.L., & Duan, N. (2014). The DEcIDE Methods Center N-of-1 Guidance Panel. In Duan, N., Eslick, I., Gabler, N.B., Kaplan, H.C., Kravitz, R.L., Larson, E.B., Pace, W.D., Schmid, C.H., Sim, I., & Vohra, S. (Eds.), Design and implementation of N-of-1 trials: A user's guide. AHRQ Publication No. 13(14)-EHC122-EF. Rockville, MD: Agency for Healthcare Research and Quality. Retrieved from https://effectivehealthcare.ahrq.gov/topics/n-1-trials/research-2014-5/.Google Scholar
Kuppens, S., & Onghena, P. (2010). Are there enough pieces to unravel the puzzle? A method to determine sufficiency in single-case research synthesis. Annual meeting of the american educational research association (AERA). Denver, 30 April–4 May 2010.Google Scholar
Kuppens, S., & Onghena, P. (2012). Sequential meta-analysis to determine the sufficiency of cumulative knowledge: The case of early intensive behavioral intervention for children with autism spectrum disorders. Research in Autism Spectrum Disorders, 6, 168–176. doi:10.1016/j.rasd.2011.04.002.Google Scholar
Kuppens, S., Heyvaert, M., Van den Noortgate, W., & Onghena, P. (2011). Sequential meta-analysis of single-case experimental data. Behavior Research Methods, 43, 720–729. doi:10.3758/s13428-011-0080-1.Google Scholar
Lakens, D., Hilgard, J., & Staaks, J. (2016). On the reproducibility of meta-analyses: Six practical recommendations. BMC Psychology, 4, 24. doi:10.1186/s40359-016-0126-3.CrossRefGoogle ScholarPubMed
Lane, J.D., & Gast, D.L. (2014). Visual analysis in single case experimental design studies: Brief review and guidelines. Neuropsychological Rehabilitation, 24, 445–463. doi:10.1080/09602011.2013.815636.Google Scholar
Levin, J.R., Ferron, J. M., & Gafurov, B. S. (2014). Improved randomization tests for a class of single-case intervention designs. Journal of Modern Applied Statistical Methods, 13 (2), 2–52. doi:10.22237/jmasm/1414814460.Google Scholar
Little, R.J.A., & Rubin, D.B. (1987). Statistical analysis with missing data. New York, NY: Wiley.Google Scholar
Lobo, M.A., Moeyaert, M., Cunha, A.B., & Babik, I. (2017). Single-case design, analysis, and quality assessment for intervention research. Journal of Neurologic Physical Therapy, 41, 187–197. doi:10.1097/NPT.0000000000000187.Google Scholar
Manolov, R., & Moeyaert, M. (2017a). How can single-case data be analyzed? Software resources, tutorial, and reflections on analysis. Behavior Modification, 41, 179–228. doi:10.1177/0145445516664307.Google Scholar
Manolov, R., & Moeyaert, M. (2017b). Recommendations for choosing single-case data analytical techniques. Behavior Therapy, 48, 97–114. doi:10.1016/j.beth.2016.04.008.Google Scholar
Manolov, R., & Solanas, A. (2009). Percentage of nonoverlapping corrected data. Behavior Research Methods, 41, 1262–1271. doi:10.3758/BRM.41.4.1262.Google Scholar
McCullagh, P., & Nelder, J. (1989). Generalized linear models (2nd ed.). Boca Raton, FL: Chapman and Hall/CRC.Google Scholar
McDonald, S., & Davidson, K.W. (2016). Using N-of-1 methodology to study or change health-related behaviour. The European Health Psychologist, 18, 38–42.Google Scholar
Michiels, B., & Onghena, P. (2017). Nonparametric meta-analysis for single-case research: Confidence intervals for combined effect sizes. Manuscript submitted for publication.Google Scholar
Michiels, B., Heyvaert, M., Meulders, A., & Onghena, P. (2017). Confidence intervals for single-case effect size measures based on randomization test inversion. Behavior Research Methods, 49, 363–381. doi: 10.3758/s13428-016-0714-4.Google Scholar
Moeyaert, M., Ferron, J., Beretvas, S., & Van den Noortgate, W. (2014). From a single-level analysis to a multilevel analysis of single-subject experimental data. Journal of School Psychology, 52, 191–211. doi:10.1016/j.jsp.2013.11.003.Google Scholar
Moeyaert, M., Rindskopf, D., Onghena, P., & Van den Noortgate, W. (2017). Multilevel modeling of single-case data: A comparison of maximum likelihood and Bayesian estimation. Psychological Methods. doi:10.1037/met0000136.Google Scholar
Moeyaert, M., Ugille, M., Beretvas, S., Ferron, J., Bunuan, R., & Van den Noortgate, W. (2016). Methods for dealing with multiple outcomes in meta-analysis: A comparison between averaging effect sizes, robust variance estimation and multilevel meta-analysis. International Journal of Social Research Methodology: Theory & Practice. doi:10.1080/13645579.2016. 1252189.Google Scholar
Moeyaert, M., Ugille, M., Ferron, J., Beretvas, S., & Van den Noortgate, W. (2013a). The three-level synthesis of standardized single-subject experimental data: A Monte Carlo simulation study. Multivariate Behavioral Research, 48, 719–748. doi:10.1080/00273171.2013.816621.Google Scholar
Moeyaert, M., Ugille, M., Ferron, J., Beretvas, S., & Van den Noortgate, W. (2013b). Three-level analysis of single-case experimental data: Empirical validation. Journal of Experimental Education, 82, 1–21. doi:10.1080/00220973.2012.745470.Google Scholar
Moeyaert, M., Ugille, M., Ferron, J., Beretvas, S., & Van den Noortgate, W. (2014). The influence of the design matrix on treatment effect estimates in the quantitative analyses of single-case experimental design research. Behavior Modification, 38, 665–704. doi:10.1177/0145445514535243.Google Scholar
Moeyaert, M., Ugille, M., Ferron, J., Onghena, P., Heyvaert, M, & Van den Noortgate, W. (2015). Estimating intervention effects across different types of single-subject experimental designs: Empirical illustration. School Psychology Quarterly, 25, 191–211. doi:10.1037/spq0000068.Google Scholar
Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G., & the PRISMA group (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA Statement. PLoS Medicine, 6 (7), e1000097. doi:10.1371/journal.pmed1000097.Google Scholar
Molenaar, P.C.M. (2004). A manifesto on psychology as idiographic science: Bringing the person back into scientific psychology, this time forever. Measurement: Interdisciplinary Research and Perspectives, 2, 201–211. doi:10.1207/s15366359mea0204_1.Google Scholar
Molenaar, P.C.M., & Campbell, C.G. (2009). The new person-specific paradigm in psychology. Current Directions in Psychology, 18, 112–117. doi:10.1111/j.1467-8721.2009.01619.x.Google Scholar
Moore, D.S., McCabe, G.P., & Craig, B.A. (2017). Introduction to the practice of statistics (9th ed.). New York, NY: D. H. Freeman.Google Scholar
Onghena, P. (1992). Randomization tests for extensions and variations of ABAB single-case experimental designs: A rejoinder. Behavioral Assessment, 14, 153–171.Google Scholar
Onghena, P. (2005). Single-case designs. In Everitt, B. & Howell, D. (Eds.), Encyclopedia of statistics in behavioral science, vol. 4 (pp. 1850–1854). Chichester, UK: Wiley.Google Scholar
Onghena, P. (2007). N-of-1 randomized clinical trials. In Berger, V. (Ed.), Design and analysis of randomized clinical trials: Design, analysis & theory. London: Henry Stewart Talks Ltd. Retrieved from https://hstalks.com/t/555/.Google Scholar
Onghena, P. (2016). Randomization in N-of-1 clinical trials: Is it possible to draw causal inferences from single-patient data? In Berger, V. (Ed.), The risk of bias in randomized clinical trials. London: Henry Stewart Talks Ltd. Retrieved from https://hstalks.com/bs/3311/.Google Scholar
Onghena, P., & Edgington, E.S. (2005). Customization of pain treatments: Single-case design and analysis. Clinical Journal of Pain, 21, 56–68.Google Scholar
Onghena, P., & Struyve, C. (2015). Case studies. In Balakrishnan, N., Brandimarte, P., Everitt, B., Molenberghs, G., Piegorsch, W., & Ruggeri, F. (Eds.), Wiley StatsRef: Statistics reference online (pp. 1–5). Chichester, UK: Wiley. doi:10.1002/9781118445112.stat06656.pub2.Google Scholar
Onghena, P., Vlaeyen, J.W.S., & de Jong, J. (2007). Randomized replicated single-case experiments: Treatment of pain-related fear by graded exposure in vivo. In Sawilowsky, S. (Ed.), Real data analysis (pp. 387–396). Charlotte, NC: Information Age Publishing.Google Scholar
Ottenbacher, K.J. (1993). Interrater agreement of visual analysis in single-subjects designs: Quantitative review and analysis. American Journal of Mental Retardation, 98, 135–142.Google Scholar
Ottoboni, K., Lewis, F., & Salmaso, L. (2017). A comparison of parametric and permutation tests for regression analysis of randomized experiments. arXiv:1702.04851v1.Google Scholar
Peat, J. (2002). Health science research: A handbook of quantitative methods. London, UK: Sage.Google Scholar
Perdices, M., & Tate, R.L. (2009). Single-subject designs as a tool for evidence-based clinical practice: Are they unrecognised and undervalued? Neuropsychological Rehabilitation, 19, 904–927. doi:10.1080/09602010903040691.Google Scholar
Pesarin, F., & Salmaso, L. (2010). Permutation tests for complex data: Theory, applications and software. Chichester, UK: Wiley.Google Scholar
Pinheiro, J.C., & Bates, D.M. (2000). Mixed-effects models in S and S-PLUS. New York: Springer.Google Scholar
Piwek, L., Ellis, D. A., Andrews, S., & Joinson, A. (2016). The rise of consumer health wearables: Promises and barriers. PLoS Medicine, 13 (2), e1001953. doi:10.1371/journal.pmed.1001953.Google Scholar
Punja, S., Schmid, C.H., Hartling, L., Urichuk, L., Nikles, C.J., & Vohra, S. (2016). To meta-analyze or not to meta-analyze? A combined meta-analysis of N-of-1 trial data with RCT data on amphetamines and methylphenidate for pediatric ADHD. Journal of Clinical Epidemiology, 76, 76–81. doi:10.1016/j.jclinepi.2016.03.021.Google Scholar
Pustejovsky, J.E., Hedges, L.V., & Shadish, W.R. (2014). Design-comparable effect sizes in multiple baseline designs: A general modeling framework. Journal of Educational and Behavioral Statistics, 39, 368–393. doi:10.3102/1076998614547577.Google Scholar
Rosenthal, R. (1978). Combining the results of independent studies. Psychological Bulletin, 85, 185–193. doi:10.1037/0033-2909.85.1.185.Google Scholar
Sackett, D.L., Rosenberg, W.M., Gray, J.A., Haynes, R.B., & Richardson, W.S. (1996). Evidence based medicine: What it is and what it isn't. BMJ, 312, 71–72. doi:10.1136/bmj.312.7023.71.Google Scholar
Scargle, J.D. (2000). Publication bias: The “file-drawer” problem in scientific inference. Journal of Scientific Exploration, 14, 91–106.Google Scholar
Schlosser, R.W., Lee, D.L., & Wendt, O. (2008). Application of the percentage of non-overlapping data (PND) in systematic reviews and meta-analyses: A systematic review of reporting characteristics. Evidence-Based Communication Assessment and Intervention, 2, 163–187. doi:10.1080/17489530802505412.Google Scholar
Schork, N.J. (2015). Personalized medicine: Time for one-person trials. Nature, 520 (7549), 609–611. doi:10.1038/520609a.Google Scholar
Scruggs, T.E., & Mastropieri, M.A. (2013). PND at 25: Past, present, and future trends in summarizing single-subject research. Remedial and Special Education, 34, 9–19. doi:10.1177/0741932512440730.Google Scholar
Senn, S.J. (2017, 8 February 2017). Consult two medics and you'll get two opinions but consult two statisticians and you could easily get three #thewonderofstats [Twitter moment]. Retrieved from https://twitter.com/stephensenn/status/829593923123343363.Google Scholar
Shadish, W.R. (2014). Analysis and meta-analysis of single-case designs: An introduction. Journal of School Psychology, 52, 109–122. Retrieved from http://doi.org/10.1016/j.jsp.2013.11.009.Google Scholar
Shadish, W.R., & Sullivan, K.J. (2011). Characteristics of single-case designs used to assess intervention effects in 2008. Behavior Research Methods, 43, 971–980. doi:10.3758/s13428-011-0111-y.Google Scholar
Shadish, W.R., Kyse, E.N., & Rindskopf, D.M. (2013). Analyzing data from single-case designs using multilevel models: New applications and some agenda items for future research. Psychological Methods, 18, 385–405. doi:10.1037/a0032964.Google Scholar
Shadish, W.R., Rindskopf, D.M., & Boyajian, J.G. (2016). Single-case experimental design yielded an effect estimate corresponding to a randomized controlled trial. Journal of Clinical Epidemiology, 76, 82–88. doi:10.1016/j.jclinepi.2016.01.035.Google Scholar
Shadish, W.R., Zelinsky, N.A., Vevea, J.L., & Kratochwill, T.R. (2016). A survey of publication practices of single-case design researchers when treatments have small or large effects. Journal of Applied Behavioral Analysis, 49, 656–673. doi: 10.1002/jaba.308.Google Scholar
Shamseer, L., Sampson, M., Bukutu, C., Schmid, C.H., Nikles, J., Tate, R., . . . & the CENT group (2015). CONSORT extension for reporting N-of-1 trials (CENT) 2015: Explanation and elaboration. British Medical Journal, 350, h1793. doi:10.1136/bmj/h1793.Google Scholar
Shapiro, M.B. (1966). The single case in clinical-psychological research. The Journal of General Psychology, 74, 3–23. doi:10.1080/ 00221309.1966.9710306.Google Scholar
Shea, B.J., Hamel, C., Wells, G.A., Bouter, L.M., Kristjansson, E., Grimshaw, J., . . . Boers, M. (2009). AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology, 62, 1013–1020. doi:10.1016/j.jclinepi.2008.10.009.Google Scholar
Shine, L.C., & Bower, S.M. (1971). A one-way analysis of variance for single-subject designs. Educational and Psychological Measurement, 31, 105–113. doi:10.1177/001316447103100108.Google Scholar
Sidman, M. (1952). A note on functional relations obtained from group data. Psychological Bulletin, 49, 263–269. doi:10.1037/h0063643.Google Scholar
Simmons, J.P., Nelson, L.D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366. doi:10.1177/0956797611417632.Google Scholar
Smith, J.D. (2012). Single-case experimental designs: A systematic review of published research and current standards. Psychological Methods, 17, 510–550. doi:10.1037/a0029312.Google Scholar
Solmi, F., & Onghena, P. (2014). Combining P-values in replicated single-case experiments with multivariate outcome. Neuropsychological Rehabilitation, 24, 607–633. doi:10.1080/09602011.2014.881747.Google Scholar
Solomon, B.G. (2014). Violations of assumptions in school-based single-case data: Implications for the selection and interpretation of effect sizes. Behavior Modification, 38, 477–496. doi:10.1177/0145445513510931.Google Scholar
Sterba, S.K. (2009). Alternative model-based and design-based frameworks for inference from samples to populations: From polarization to integration. Multivariate Behavioral Research, 44, 711–740. doi:10.1080/00273170903333574.Google Scholar
Strube, M.J., & Miller, R.H. (1986). Comparison of power rates for combined probability procedures: A simulation study. Psychological Bulletin, 99, 407–415. doi:10.1037/0033-2909. 99.3.407.Google Scholar
Tate, R. L., Aird, V., & Taylor, C. (2013). Bringing single-case methodology into the clinic to enhance evidence-based practices. Brain Impairment, 13, 347–359. doi:10.1017/BrImp.2012.32.Google Scholar
Tate, R.L., Perdices, M., Rosenkoetter, U., McDonald, S., Togher, L., Shadish, W., . . . Vohra, S. (2016a). The single-case reporting guideline in BEhavioural Interventions (SCRIBE) 2016: Explanation and elaboration. Archives of Scientific Psychology, 4, 10–31. doi:10.1037/arc0000027.Google Scholar
Tate, R.L., Perdices, M., Rosenkoetter, U., Shadish, W., Vohra, S., Barlow, D. H., . . . Wilson, B. (2016b). The single-case reporting guideline In BEhavioural interventions (SCRIBE) 2016 statement. Archives of Scientific Psychology, 4, 1–9. doi:10.1037/arc0000026.Google Scholar
Tate, R.L., Perdices, M., Rosenkoetter, U., Wakim, D., Godbee, K., Togher, L., & McDonald, S. (2013). Revision of a method quality rating scale for single-case experimental designs and N-of-1 trials: The 15-item Risk of Bias in N-of-1 Trials (RoBiNT) Scale. Neuropsychological Rehabilitation, 23, 619–638. Retrieved from https://doi.org/10.1080/09602011.2013.824383.Google Scholar
Terrin, N., Schmid, C.H., Lau, J., & Olkin, I. (2003). Adjusting for publication bias in the presence of heterogeneity. Statistics in Medicine, 22, 2113–2126. doi:10.1002/sim.1461.Google Scholar
Tierney, J.F., Vale, C., Riley, R., Smith, C.T., Stewart, L., Clarke, M., & Rovers, M. (2015). Individual participant data (IPD) meta-analyses of randomised controlled trials: Guidance on their use. PLoS Medicine, 12 (7), e1001855. doi:10.1371/journal.pmed.1001855.Google Scholar
Tukey, J.W. (1969). Analyzing data: Sanctification or detective work? American Psychologist, 24, 83–91. doi:10.1037/h0027108.Google Scholar
Turner, L., Shamseer, L., Altman, D. G., Weeks, L., Peters, J., Kober, T., . . . Moher, D. (2012). Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Database of Systematic Reviews, 11, MR000030. Retrieved from https://doi.org/10.1002/14651858.mr000030.pub2.Google Scholar
Ugille, M., Moeyaert, M., Beretvas, S., Ferron, J., & Van den Noortgate, W. (2012). Multilevel meta-analysis of single-subject experimental designs: A simulation study. Behavior Research Methods, 44, 1244–1254. doi:10.3758/s13428-012-0213-1.Google Scholar
Vallverdú, J. (2016). Bayesians versus frequentists: A philosophical debate on statistical reasoning. New York, NY: Springer.Google Scholar
Van den Noortgate, W., & Onghena, P. (2003a). Combining single-case experimental data using hierarchical linear models. School Psychology Quarterly, 18, 325–346.Google Scholar
Van den Noortgate, W., & Onghena, P. (2003b). Hierarchical linear models for the quantitative integration of effect sizes in single-case research. Behavior Research Methods, Instruments & Computers, 35, 1–10. doi:10.3758/BF03195492.Google Scholar
Van den Noortgate, W., & Onghena, P. (2003c). Multilevel meta-analysis: a comparison with traditional meta-analytical procedures. Educational and Psychological Measurement, 63, 765–790. doi:10.1177/0013164403251027.Google Scholar
Van den Noortgate, W., & Onghena, P. (2007). The aggregation of single-case results using hierarchical models. Behavior Analyst Today, 8, 196–208. doi:10.1037/h0100613.Google Scholar
Van den Noortgate, W., & Onghena, P. (2008). A multilevel meta-analysis of single-subject experimental design studies. Evidence-Based Communication Assessment and Intervention, 2, 142–151. doi:10.1080/17489530802505362.Google Scholar
Van den Noortgate, W., López-López, J., Marín-Martínez, F., & Sánchez-Meca, J. (2013). Three-level meta-analysis of dependent effect sizes. Behavior Research Methods, 45, 576–594. doi:10.3758/s13428-012-0261-6.Google Scholar
Van den Noortgate, W., López-López, J., Marín-Martínez, F., & Sánchez-Meca, J. (2015). Meta-analysis of multiple outcomes: A multilevel approach. Behavior Research Methods, 47, 1274–1294. doi:10.3758/s13428-014-0527-2.Google Scholar
Vanderkerken, L., Heyvaert, M., Maes, B., & Onghena, P. (2013). Psychosocial interventions for reducing vocal challenging behaviour in persons with autistic disorder: A multilevel meta-analysis of single-case experiments. Research in Developmental Disabilities, 34, 4515–4533. doi:10.1016/j.ridd.2013.09.030.Google Scholar
Velicer, W.F., & Molenaar, P. (2013). Time series analysis. In Schinka, J. & Velicer, W.F. (Eds.), Handbook of psychology: Research methods in psychology (2nd ed., Vol. 50, pp. 628–660). New York: Wiley.Google Scholar
Vohra, S., Shamseer, L., Sampson, M., Bukutu, C., Schmid, C.H., Tate, R., . . . & the CENT group (2015). CONSORT extension for reporting N-of-1 trials (CENT) 2015 Statement. British Medical Journal, 350, h1738. doi:10.1136/bmj/h1738.Google Scholar
Wheeler, B., & Torchiano, M. (2016). Permutation tests for linear models: the lmPerm package (version 2.1.0) in R. Retrieved from https://cran.r-project.org/web/packages/lmPerm/lmPerm.pdf.Google Scholar
Wicherts, J.M., Veldkamp, C.L.S., Augusteijn, H.E.M., Bakker, M., van Aert, R.C.M., & van Assen, M.A. L.M. (2016). Degrees of freedom in planning, running, analyzing, and reporting psychological studies: A checklist to avoid p-hacking. Frontiers in Psychology, 7, 1832. doi:10.3389/fpsyg.2016.01832.Google Scholar
Supplementary material: File

Onghena et al supplementary material

Onghena et al supplementary material 1

Download Onghena et al supplementary material(File)
File 295 Bytes
Supplementary material: File

Onghena et al supplementary material

Onghena et al supplementary material 2

Download Onghena et al supplementary material(File)
File 12 Bytes
Supplementary material: File

Onghena et al supplementary material

Onghena et al supplementary material 3

Download Onghena et al supplementary material(File)
File 7.1 KB
Supplementary material: File

Onghena et al supplementary material

Onghena et al supplementary material 4

Download Onghena et al supplementary material(File)
File 121 Bytes
Supplementary material: File

Onghena et al supplementary material

Onghena et al supplementary material 5

Download Onghena et al supplementary material(File)
File 142 Bytes