Hostname: page-component-78c5997874-m6dg7 Total loading time: 0 Render date: 2024-11-10T16:16:23.021Z Has data issue: false hasContentIssue false
  • English
  • Français

Transparency of reporting in CALL meta-analyses between 2003 and 2015

Published online by Cambridge University Press:  11 October 2017

Huifen Lin ,
Tsuiping Chen  and
Hsien-Chin Liou Open the ORCID record for Hsien-Chin Liou [Opens in a new window]
Huifen Lin
Affiliation:
National Tsing Hua University, Taiwan (email: huifen@mx.nthu.edu.tw)
Tsuiping Chen
Affiliation:
Kun Shan University, Taiwan (email: tsuiping0925@gmail.com)
Hsien-Chin Liou
Affiliation:
Feng Chia University, Taiwan (email: liuhsienc@gmail.com)
Get access Rights & Permissions [Opens in a new window]

Abstract

Since its introduction by Glass in the 1970s, meta-analysis has become a widely accepted and the most preferred approach to conducting research synthesis. Overcoming the weaknesses commonly associated with traditional narrative review and vote counting, meta-analysis is a statistical method of systematically aggregating and analyzing empirical studies by following well-established procedures. The findings of a meta-analysis, when appropriately conducted, are able to inform important policy decisions and provide practical pedagogical suggestions. With the growing number of publications employing meta-analysis across a wide variety of disciplines, it has received criticism due to its inconsistent findings derived from multiple meta-analyses in the same research domain. These inconsistencies have arisen partly due to the alternatives available to meta-analysts in each major meta-analytic procedure. Researchers have therefore recommended transparent reporting on the decision-making for every essential judgment call so that the results across multiple meta-analyses become replicable, consistent, and interpretable. This research explored the degree to which meta-analyses in the computer-assisted language learning (CALL) discipline transparently reported their decisions in every critical step. To achieve this aim, we retrieved 15 eligible meta-analyses in CALL published between 2003 and 2015. Features of these meta-analyses were extracted based on a codebook modified from Cooper (2003) and Aytug, Rothstein, Zhou and Kern (2012). A transparency score of reporting was then calculated to examine the degree to which these meta-analyses are compliant with the norms of reporting as recommended in the literature. We then discuss the strengths and weaknesses of the methodologies and provide suggestions for conducting quality meta-analyses in this domain.

Keywords

transparencyreportingmeta-analysisCALLsystematic reviewsynthesis
Type
Regular papers
Information
ReCALL , Volume 30 , Issue 3 , September 2018 , pp. 253 - 277
Copyright
© European Association for Computer Assisted Language Learning 2017 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

*Abraham, L. B. (2008) Computer-mediated glosses in second language reading comprehension and vocabulary learning: A meta-analysis. Computer Assisted Language Learning, 21(3): 199–226. https://doi.org/10.1080/09588220802090246 Google Scholar
Ahn, S., Ames, A. J. and Myers, N. D. (2012) A review of meta-analyses in education: Methodological strengths and weaknesses. Review of Educational Research, 82(4): 436–476. https://doi.org/10.3102/0034654312458162 Google Scholar
Aytug, Z. G., Rothstein, H. R., Zhou, W. and Kern, M. C. (2012) Revealed or concealed? Transparency of procedures, decisions, and judgment calls in meta-analyses. Organizational Research Methods, 15(1): 103–133. https://doi.org/10.1177/1094428111403495 Google Scholar
Berkeljon, A. and Baldwin, S. A. (2009) An introduction to meta-analysis for psychotherapy outcome research. Psychotherapy Research, 19(4–5): 511–518. https://doi.org/10.1080/10503300802621172 Google Scholar
Borenstein, M., Hedges, L. V., Higgins, J. P. T. and Rothstein, H. R. (2009) Introduction to meta-analysis. Chichester, UK: Wiley. https://doi.org/10.1002/9780470743386 Google Scholar
*Chang, M.-M. and Lin, M.-C. (2013) Strategy-oriented web-based English instruction: A meta-analysis. Australasian Journal of Educational Technology, 29(2): 203–216. https://doi.org/10.14742/ajet.67 Google Scholar
*Chiu, Y.-H. (2013) Computer-assisted second language vocabulary instruction: A meta-analysis. British Journal of Educational Technology, 44(2): E52–E56. https://doi.org/10.1111/j.1467-8535.2012.01342.x Google Scholar
*Chiu, Y.-H., Kao, C.-W. and Reynolds, B. L. (2012) The relative effectiveness of digital game-based learning types in English as a foreign language setting: A meta-analysis. British Journal of Educational Technology, 43(4): E104–E107. https://doi.org/10.1111/j.1467-8535.2012.01295.x Google Scholar
*Cobb, T. and Boulton, A. (2015) Classroom applications of corpus analysis. In In Biber, D. and Reppen, R. (eds.), The Cambridge handbook of English corpus linguistics. Cambridge: Cambridge University Press, 478–497.Google Scholar
Cooper, H. (1998) Synthesizing research: A guide for literature reviews (3rd ed.). Thousand Oaks, CA: Sage.Google Scholar
Cooper, H. (2003) Editorial. Psychological Bulletin, 129(1): 3–9. https://doi.org/10.1037/0033-2909.129.1.3 Google Scholar
Cooper, H. (2007) Evaluating and interpreting research syntheses in adult learning and literacy (NCSALL occasional paper). Boston, MA: National Center for the Study of Adult Learning and Literacy. http://www.worlded.org/WEIInternet/inc/common/_download_pub.cfm?id=16699&lid=3 Google Scholar
Copas, J. and Shi, J. Q. (2000) Meta-analysis, funnel plots and sensitivity analysis. Biostatistics, 1(3): 247–262. https://doi.org/10.1093/biostatistics/1.3.247 Google Scholar
Durlak, J. A., Weissberg, R. P. and Pachan, M. (2010) A meta-analysis of after-school programs that seek to promote personal and social skills in children and adolescents. American Journal of Community Psychology, 45(3–4): 294–309. https://doi.org/10.1007/s10464-010-9300-6 Google Scholar
Egger, M., Smith, G. D. and Phillips, A. N. (1997) Meta-analysis: Principles and procedures. British Medical Journal, 315: 1533–1537. https://doi.org/10.1136/bmj.315.7121.1533 Google Scholar
Elvik, R. (2005) Can we trust the results of meta-analyses? A systematic approach to sensitivity analysis in meta-analyses. Transportation Research Record: Journal of the Transportation Research Board, 1908: 221–229. https://doi.org/10.3141/1908-27 Google Scholar
Garg, A. X., Hackam, D. and Tonelli, M. (2008) Systematic review and meta-analysis: When one study is just not enough. Clinical Journal of the American Society of Nephrology, 3(1): 253–260. https://doi.org/10.2215/CJN.01430307 Google Scholar
Glass, G. V. (1976) Primary, secondary, and meta-analysis of research. Educational Researcher, 5(10): 3–8. https://doi.org/10.3102/0013189X005010003 Google Scholar
Glass, G. V., McGaw, B. and Smith, M. L. (1981) Meta-analysis in social research. Beverly Hills, CA: Sage.Google Scholar
Goo, J., Granena, G., Yilmaz, Y. and Novella, M. (2015) Implicit and explicit instruction in L2 learning. In Rebuschat, P. (ed.), Implicit and explicit learning of languages (Vol. 48). Amsterdam: John Benjamins, 443–482. https://doi.org/10.1075/sibil.48.18goo Google Scholar
*Grgurović, M., Chapelle, C. A. and Shelley, M. C. (2013) A meta-analysis of effectiveness studies on computer technology-supported language learning. ReCALL, 25(2): 165–198. https://doi.org/10.1017/S0958344013000013 Google Scholar
Heyland, D. K. (n.d.) Merits and limitations of meta-analyses. http://scholar.googleusercontent.com/scholar?q=cache:pbTgrH5h_RUJ:scholar.google.com/+Merits+and+Limitations+of+Meta-analyses&hl=zh-TW&as_sdt=0,5 Google Scholar
Higgins, J. P. T. and Green, S. (2011) Cochrane handbook for systematic reviews of interventions (Vol. 4). Chichester, UK: Wiley.Google Scholar
In’nami, Y. and Koizumi, R. (2009) A meta-analysis of test format effects on reading and listening test performance: Focus on multiple-choice and open-ended formats. Language Testing, 26(2): 219–244. https://doi.org/10.1177/0265532208101006 Google Scholar
In’nami, Y. and Koizumi, R. (2010) Database selection guidelines for meta-analysis in applied linguistics. TESOL Quarterly, 44(1): 169–184. https://doi.org/10.5054/tq.2010.215253 Google Scholar
Levy, M. (1997) Computer-assisted language learning: Context and conceptualization. Oxford: Oxford University Press.Google Scholar
Li, S., Shintani, R. and Ellis, R. (2012) Doing meta-analysis in SLA: Practice, choices, and standards. Contemporary Foreign Language Studies, 384(12): 1–17.Google Scholar
Liao, Y.-K. C. and Hao, Y. (2008) Large-scale studies and quantitative methods. In Voogt, J. and Knezek, G. (eds.), International handbook of information technology in primary and secondary education (Vol. 20). New York: Springer Science & Business Media, 1019–1035. https://doi.org/10.1007/978-0-387-73315-9_64 Google Scholar
*Lin, H. (2014) Establishing an empirical link between computer-mediated communication (CMC) and SLA: A meta-analysis of the research. Language Learning & Technology, 18(3): 120–147. http://llt.msu.edu/issues/october2014/lin.pdf Google Scholar
*Lin, H. (2015a) Computer-mediated communication (CMC) in L2 oral proficiency development: A meta-analysis. ReCALL, 27(3): 261–287. https://doi.org/10.1017/S095834401400041X Google Scholar
*Lin, H. (2015b) A meta-synthesis of empirical research on the effectiveness of computer-mediated communication (CMC) on SLA. Language Learning & Technology, 19(2): 85–117. http://llt.msu.edu/issues/june2015/lin.pdf Google Scholar
*Lin, W.-C., Huang, H.-T. and Liou, H.-C. (2013) The effects of text-based SCMC on SLA: A meta analysis. Language Learning & Technology, 17(2): 123–142. http://llt.msu.edu/issues/june2013/linetal.pdf Google Scholar
Liou, H.-C. and Lin, H.-F. (2017) CALL meta-analyses and transparency analysis. In Chapelle, C. A. and Sauro, S. (eds.), The handbook of technology and second language teaching and learning. Hoboken, NJ: Wiley, 409–427. https://doi.org/10.1002/9781118914069.ch27 Google Scholar
Lipsey, M. W. and Wilson, D. B. (2001) Practical meta-analysis. Thousand Oaks, CA: Sage.Google Scholar
Littell, J. H., Corcoran, J. and Pillai, V. (2008) Systematic reviews and meta-analysis. Oxford: Oxford University Press. https://doi.org/10.1093/acprof:oso/9780195326543.001.0001 Google Scholar
Norris, J. M. and Ortega, L. (2000) Effectiveness of L2 instruction: A research synthesis and quantitative meta-analysis. Language Learning, 50(3): 417–528. https://doi.org/10.1111/0023-8333.00136 Google Scholar
Norris, J. M. and Ortega, L. (2006) The value and practice of research synthesis for language learning and teaching. In Norris, J. M. and Ortega, L. (eds.), Synthesizing research on language learning and teaching. Amsterdam: John Benjamins, 3–50. https://doi.org/10.1075/lllt.13.04nor.Google Scholar
Plonsky, L. (2012) Replication, meta-analysis, and generalizability. In Porte, G. (ed.), Replication research in applied linguistics. Cambridge: Cambridge University Press, 116–132.Google Scholar
Plonsky, L. (2013) Study quality in SLA: An assessment of designs, analyses, and reporting practices in quantitative L2 research. Studies in Second Language Acquisition, 35(4): 655–687. https://doi.org/10.1017/S0272263113000399 Google Scholar
Plonsky, L. and Ziegler, N. (2016) The CALL–SLA interface: Insights from a second-order synthesis. Language Learning & Technology, 20(2): 17–37. http://llt.msu.edu/issues/june2016/plonskyziegler.pdf Google Scholar
Rosenthal, R. and DiMatteo, M. R. (2001) Meta-analysis: Recent developments in quantitative methods for literature reviews. Annual Review of Psychology, 52: 59–82. https://doi.org/10.1146/annurev.psych.52.1.59 Google Scholar
Rothstein, H. R. and McDaniel, M. A. (1989) Guidelines for conducting and reporting meta-analyses. Psychological Reports, 65(3): 759–770. https://doi.org/10.2466/pr0.1989.65.3.759 Google Scholar
Rothstein, H. R., Sutton, A. J. and Borenstein, M. (eds.) (2006) Publication bias in meta-analysis: Prevention, assessment and adjustments . Hoboken, NJ: Wiley.Google Scholar
Scammacca, N., Roberts, G. and Stuebing, K. K. (2014) Meta-analysis with complex research designs: Dealing with dependence from multiple measures and multiple group comparisons. Review of Educational Research, 84(3): 328–364. https://doi.org/10.3102/0034654313500826 Google Scholar
Schmidt, F. L. and Oh, I.-S. (2013) Methods for second order meta-analysis and illustrative applications. Organizational Behavior and Human Decision Processes, 121(2): 204–218. https://doi.org/10.1016/j.obhdp.2013.03.002 Google Scholar
Shadish, W. R. and Sweeney, R. B. (1991) Mediators and moderators in meta-analysis: There’s a reason we don’t let dodo birds tell us which psychotherapies should have prizes. Journal of Consulting and Clinical Psychology, 59(6): 883–893. https://doi.org/10.1037/0022-006X.59.6.883 Google Scholar
Shea, B. J., Hamel, C., Wells, G. A., Bouter, L. M., Kristjansson, E., Grimshaw, J., Henry, D. A. and Boers, M. (2009) AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology, 62(10): 1013–1020. https://doi.org/10.1016/j.jclinepi.2008.10.009 Google Scholar
*Taylor, A. (2006) The effects of CALL versus traditional L1 glosses on L2 reading comprehension. CALICO Journal, 23(2): 309–318.Google Scholar
*Taylor, A. M. (2010) CALL-based versus paper-based glosses: Is there a difference in reading comprehension? CALICO Journal, 27(1): 147–160. https://doi.org/10.11139/cj.27.1.147-160 Google Scholar
*Taylor, A. M. (2013) CALL versus paper: In which context are L1 glosses more effective? CALICO Journal, 30(1): 63–81. https://doi.org/10.11139/cj.30.1.63-81 Google Scholar
Turner, L., Boutron, I., Hróbjartsson, A., Altman, D. G. and Moher, D. (2013) The evolution of assessing bias in Cochrane systematic reviews of interventions: Celebrating methodological contributions of the Cochrane Collaboration. Systematic Reviews, 2: 2–8. https://doi.org/10.1186/2046-4053-2-79 Google Scholar
Valentine, J. C., Cooper, H., Patall, E. A., Tyson, D. and Robinson, J. C. (2010) A method for evaluating research syntheses: The quality, conclusions, and consensus of 12 syntheses of the effects of after-school programs. Research Synthesis Methods, 1(1): 20–38. https://doi.org/10.1002/jrsm.3 Google Scholar
Wanous, J. P., Sullivan, S. E. and Malinak, J. (1989) The role of judgment calls in meta-analysis. Journal of Applied Psychology, 74(2): 259–264. https://doi.org/10.1037/0021-9010.74.2.259 Google Scholar
Willis, B. H. and Quigley, M. (2011) The assessment of the quality of reporting of meta-analyses in diagnostic research: A systematic review. BMC Medical Research Methodology, 11: 1–11. https://doi.org/10.1186/1471-2288-11-163 Google Scholar
*Yun, J. (2011) The effects of hypertext glosses on L2 vocabulary acquisition: A meta-analysis. Computer Assisted Language Learning, 24(1): 39–58. https://doi.org/10.1080/09588221.2010.523285 Google Scholar
*Zhao, Y. (2003) Recent developments in technology and language learning: A literature review and meta-analysis. CALICO Journal, 21(1): 7–27.Google Scholar
Supplementary material: File

Lin et al supplementary material

Lin et al supplementary material 1

Download Lin et al supplementary material(File)
File 71.5 KB