Investigating the status of mixed-methods research in CALL published research articles

Abstract

This study examines the status of mixed-methods research (MMR) in computer-assisted language learning (CALL). A total of 204 studies employing MMR were analyzed. Manual coding was carried out to reveal MMR purposes, designs, features, and rhetorical justifications. Findings indicate CALL authors mostly adopt MMR for triangulation and complementarity purposes. Core designs are more favored in CALL MMR research articles, compared to complex designs. Moderate size random sampling prevails in the data, where data sources are sequentially collected and analyzed using parametric tests. Symptomatic argumentative schemes are found to be the most common justification of MMR. Based on the findings, it is evident that most CALL researchers employ conventional MMR designs. The study concludes with implications for CALL stakeholders and authors.

1. Background

Applied linguistics (AL) researchers combine quantitative and qualitative methodologies to investigate research issues that if studied through either approach alone would result in a less comprehensive understanding of those phenomena. In contrast, combining quantitative and qualitative approaches has been documented to heighten the quality and nature of inquiry in social and educational research (Hashemi & Babaii, 2013). Recently, Riazi and Amini Farsani (2024) have summarized review studies dealing with mixed-methods research (MMR) status and trends in AL research. Examining the data in these review studies reveals that CALL and its core journals have received insufficient attention when investigated through the lens of MMR. This implies that the adoption of MMR in CALL empirical research has not been explored sufficiently although CALL is exponentially evolving from its subdisciplinary status to becoming an autonomous strand of inquiry in the realm of AL (Hampel & Stickler, 2024) with its own core journals (Computer Assisted Language Learning now stands over and above other journals in the category of “Languages and Linguistics” in Scimago Journal & Country Rank), dedicated associations such as EUROCALL (Gillespie, 2020), and its current position as an established interdisciplinary field of study (Hubbard & Colpaert, 2019).

In this study, MMR trends in CALL research are assumed to be compared with the counterpart trends in the AL tradition. This assumption is premised on the fact that although CALL is increasingly growing out of AL due to its technological orientation, it still shares key concerns with AL in language learning, pedagogy, and methodology (Hampel & Stickler, 2024). Therefore, situating and exploring CALL’s MMR methodological trends within the broader AL field is both rationally relevant and, at the same time, practically feasible. These pragmatic realities should justify why AL-based MMR perspectives have been adopted to address CALL’s evolving research demands.

Against this backdrop, charting the development of MMR in CALL published research deserves more attention, as researchers publishing in CALL journals are increasingly employing MMR (Choubsaz, Jalilifar & Boulton, 2024). Choubsaz et al. (2024) found 22 out of 426 high-impact CALL articles exploiting MMR. This is due not only to the influence of various disciplines on conceptions of CALL but also to the complexities arising from technological transformations such as generative AI, with their implications for education and more specifically for language learning/teaching (Hampel & Stickler, 2024).

Considering Levy’s (2024) assertion about the maturity of CALL and CALL research, we believe it is timely to report on how MMR has been used in CALL empirical research over the past 14 years (2010–2023) with regard to purposes and designs, the cornerstones of the MMR approach (Creswell & Plano Clark, 2018), as well as methodological features. In fact, mapping the use of MMR designs in CALL is well timed as the field evolves alongside digital innovation, requiring research designs that can capture both technological affordances and effectiveness (quantitative studies) and learner experiences of using EdTech (qualitative studies). Charting MMR in CALL can also bridge quantitative-qualitative gaps to enhance methodological precision and cater for interdisciplinary dialogues necessary for the field’s progress. The 2010–2023 time span in this study replicates the practice of Sántha and Malomsoki-Sántha (2023) and Riazi and Amini Farsani (2024), as these studies contend that 2010 onward is the period of expansion of MMR in AL and, as Choubsaz et al. (2024) maintain, rising popularity in CALL. Furthermore, of particular interest in this study is to determine whether MMR is used out of conventionality or innovation in CALL empirical research in view of the rhetorical justifications that authors make by referencing previous theoretical frameworks to support their choice of MMR (Guetterman, 2017; Riazi, 2018). In this study, however, we go beyond merely referencing previous frameworks and look at the argumentations authors present to justify their MMR choice in the research articles (RAs) published in top-tier CALL journals, specifically the top Q1 journals in Web of Science and Scopus databases (see Section 2.2).

2. Methods

The focus of this systematic methodological synthesis (Chong & Plonsky, 2024; Smith & Bazis, 2021) is to spell out the purposes, designs, features, and rhetorical justification that shape the methodology of MMR RAs in CALL research. We employed Plonsky and Oswald’s (2015) guidelines for conducting a synthetic study, namely, (1) study identification and retrieval, (2) developing a coding sheet, (3) delineating the coding process, and (4) analyzing and interpreting the results. With these aims in mind, the following research questions (RQs) are posed:

• • RQ1. What are the MMR purposes in empirical RAs published in top-tier CALL journals between 2010 and 2023?

• • RQ2. What are the MMR designs in empirical RAs published in top-tier CALL journals between 2010 and 2023?

• • RQ3. How are methodological features represented in the MMR empirical RAs published in top-tier CALL journals between 2010 and 2023?

• • RQ4. How do authors justify the conventionality or innovation in their choice of MMR in top-tier CALL journal empirical RAs?

2.1 Analytical scheme

2.1.1 MMR purposes

Understanding why CALL researchers may opt for MMR entails identifying the purposes they pursue, which is one of the criteria contributing to transparency that Riazi and Amini Farsani (2024) emphasized in their six-pronged MMR quality framework. To analyze the purposes of using MMR, Greene, Caracelli and Graham’s (1989) five purposes of MMR are used, as they “have often been cited by authors using mixed methods” (Creswell & Plano Clark, 2018: 290). The main purposes Greene et al. identified for conducting MMR include:

Triangulation (seeking convergence and corroboration by comparing findings from qualitative data with the quantitative results), complementarity (seeking elaboration, illustration enhancement, and clarification of the findings from one strand with the other strand), development (using the results from one strand to help inform the other strand), initiation (discovering paradoxes and contradictions that emerge when findings from the two strands are compared), and expansion (expand breadth and range of a study by using multiple strands for different study components). (Greene et al., 1989: 259)

Triangulation and complementarity were identified as the most frequent purposes in studies using MMR, while the other three were less attended (Greene et al., 1989; Riazi & Amini Farsani, 2024).

2.1.2 MMR designs

Creswell and Plano Clark (2023: 23) characterize design as “research procedures, including data collection, data analysis, and interpretation.” They proposed six MMR designs based on four key elements, namely, implementation (timing), priority (emphasis), integration, and theory use (Tashakkori & Teddlie, 1998). In retrospect, Creswell and Plano Clark (2023) admit that their initial classification was inchoate because they gave undue importance to certain elements, such as timing and priority. In their recent reformulation of MMR designs, they instead stressed integration, intent (i.e. the underlying logic), and theory as the central elements to classify MMR designs. In view of the current developments in MMR designs, Creswell and Plano Clark (2023) proposed two main categories: core designs and complex designs. Core designs (adopted to combine one quantitative strand and one qualitative strand in either a concurrent or sequential fashion) include convergent designs (overall domains are developed from merging the two databases), explanatory sequential designs (quantitative results are further explained qualitatively), and exploratory sequential designs (qualitative results are further tested or measured quantitatively). Complex designs (adopted to extend one or more core designs within a larger, multiphase, or theory-driven framework/processes) include experimental or intervention trials, participatory research processes, multiple case studies, and evaluation approaches (Creswell & Plano Clark, 2023). Riazi and Amini Farsani (2024) report that AL researchers use both core and complex designs, with the prevalence of core MMR designs. They further attribute the scarcity of complex designs in AL research to researchers’ insufficient “familiarity with all MMR purposes and design structures” (Riazi & Amini Farsani, 2024: 173). A point concerning educational research designs, which have roots in pragmatism (Cohen, Manion & Morrison, 2018), is in order. These designs are essentially question/problem-oriented, and that is what MMR designs are premised on. For this reason, they do not constitute a separate category of research designs in this study.

2.1.3 MMR methodological features

Our analytical framework for MMR features (i.e. sampling designs, data sources, data analysis, and procedures) accords with Riazi and Amini Farsani (2024). In his seminal paper, Riazi (2016) reflects on the method versus methodology polarization, where methodology refers to the broader theoretical lens encompassing philosophical assumptions, design, and methods, while method denotes the procedural aspects of data collection and analysis. We adopted the latter view, common in pragmatic MMR approaches (Johnson & Onwuegbuzie, 2004; Tashakkori & Teddlie, 2010), which sees sampling strategies, data types, and analytic procedures as central to method, and investigated MMR sampling explicitness, temporal sequence (concurrent, sequential), and schemes (type, size). Data source concerns the data collection techniques used in quantitative and qualitative strands (e.g. test, survey, interview). For data analysis procedures, we looked for inferential tests of significance (i.e. parametric, nonparametric, or both techniques) as well as descriptive statistics.

2.1.4 MMR rhetorical justification

Creswell and Plano Clark (2018: 290) assert that MMR purposes “do not address the added values” of combining quantitative and qualitative methods. Thus, a further development is required to not only justify the use of MMR but also clearly articulate the added value of this integration (point addressed in Q4). Creswell and Plano Clark (2018: 52) classify MMR core designs into fixed and emergent, where in the former the use of MMR “is predetermined and planned at the start of the research process and the procedures are implemented as planned,” while in the latter the use of MMR “arises due to issues that develop during the process of conducting the research.” We argue that this improvisation not only transpires during research but might also manifest itself any time in the research process when researchers innovate methods – that is, integrate the quantitative and qualitative methods creatively – to suit their research problems. Riazi (2016) corroborates that by allowing RQs to guide the choice of methodology, avoiding paradigmatic impositions, and drawing on hybrid ontological and epistemological perspectives when conceptualizing the research problems, researchers are in fact shaping an innovative MMR (IMMR). Riazi (2016: 38–42) identifies the distinct features of IMMR studies as follows:

• • having a clear purpose and logic for integrating methods from both approaches

• • showing how this mixture is done genuinely

• • addressing the need of interweaving the different types of knowledge related to the research problem

• • conceptualizing the research problem as multi-layered

• • showing how each layer lends itself to a particular type of investigation

• • mixing methods at all (conceptualization, implementation, and inferential) stages in the research process

• • drawing on hybrid theories and paradigms at the conceptualization stage.

Most of these features reflect what Creswell and Plano Clark (2023) characterize as complex MMR designs. However, neither Riazi nor Creswell and Plano Clark have thoroughly discussed how authors justify their IMMR. Riazi and Amini Farsani (2024), for example, associated what they called MMR rhetoric with the type of references authors make to multidisciplinary (i.e. sources used not only in AL but also in other disciplines), discipline-specific (i.e. sources frequently used in AL), and general sources. In addition to this surface-based approach, we believe it is necessary to examine authors’ argumentative justifications of IMMR designs. We drew on the three pragma-dialectical argumentation schemes (van Eemeren, 2018; van Eemeren & Garssen, 2020). These schemes show how the arguments are deployed in the text, provide common argumentative grounds that enable dialogues between authors and readers, and boost the credibility of the arguments presented. Some authors do not put forth any arguments to defend their choice of MMR. For cases like these, in our analysis, we used Not Applicable, while for studies in which more than one argumentation scheme was exploited, the term Combined is used.

1. (A) In the symptomatic scheme, accepting the author’s standpoint relies on it being a symptom of a more comprehensive argument(s) – that is, when an author refers to an authority in the field whose claims in MMR are much more inclusive than the claim put forth by the author.

2. (B) In the causal scheme, the standpoint’s acceptability (the effect) depends on the acceptability of its cause(s) – that is, the effect/cause of using MMR is justified by the circumstances surrounding the choice of research design.

3. (C) In the analogy-based scheme, the acceptability of a standpoint is linked to the resemblance between the arguments – that is, the claim offered by an author of why MMR is used is identical to the claim of another author.

Argumentation schemes can help differentiate strong from weak justifications by evaluating the logical soundness and appropriateness of the reasoning employed using critical questions. In the context of CALL MMR RAs, these questions might pertain but are not limited to such issues as to why this particular MMR design is employed rather than other alternative MMR designs or single-strand quantitative/qualitative approaches (symptomatic), how the adoption of MMR contributes to meta-inferences – that is, integrating the findings obtained from the two approaches to provide understanding of the issue under investigation (causal) (Hashemi & Gohari Moghaddam, 2019) – and who else in the literature has applied this approach to address this type of RQ (analogy-based) (Riazi & Amini Farsani, 2024).

2.2 Data retrieval

The first step was to investigate the “Languages and Linguistics” category of the most prestigious social sciences databases, specifically the Scimago Journal Ranking (powered by Scopus) and Web of Science Core Collection (powered by Clarivate Analytics), in May 2024, when both databases released their last report on 2023, to find the pertinent journals. Considering their strong rankings and impact over the years (Gillespie, 2020; Mohsen et al., 2024) and following Smith and Bazis’s (2021) guidelines, five Q1 top-tier CALL journals – that is, Computer Assisted Language Learning (CALL), ReCALL, Language Learning & Technology (LL&T), CALICO Journal, and the International Journal of Computer-Assisted Language Learning and Teaching (IJCALLT) – were selected for data retrieval. Lower-ranked journals were excluded to uphold consistency in our analyses and to maintain the quality of findings. Including these journals, however, might have caused variability in our analyses, potentially undermining the comparability and interpretation of the findings.

After that, following Sántha and Malomsoki-Sántha (2023), who attest to the growth of MMR in AL in the 2010s and its increasing popularity in the CALL realm (Choubsaz et al., 2024), all 1,615 published RAs from 2010 up to and including 2023 were downloaded, categorized, and coded manually. Only empirical RAs (containing first-hand data) adopting MMR were selected using specific search terms (see Supplementary Material A). To elucidate, we followed Riazi and Amini Farsani’s (2024; 153) manual search procedure to include solely the RAs that explicitly used collocations of “mix*, mixed-methods, mixed-methods design, mixed-methodology, mixed research, and mixed approach”. We excluded the RAs that used the terms “quantitative AND qualitative” in their research designs or adopted MMR to justify the way they reviewed/synthesized their papers and managed to collect a representative dataset of 204 RAs (Table 1).

Table 1. Breakdown of research articles (RAs) in the final dataset

A preliminary fallout of the dataset indicates a growing number of CALL RAs that have adopted MMR between 2010 and 2023, particularly in the past four years (n = 97, 48%), attesting to the selected time span in this study (Figure 1).

2.3 Coding process

Our developed coding book included five main headings (i.e. RA profiles, MMR purposes, MMR designs, MMR features, and MMR rhetorical justification) and several subheadings (see Supplementary Material A). The design of the coding book was based on the analytical scheme provided in Section 2.1. The first main heading pertains to RA profile details and the journal name. However, coding MMR purposes, designs, methodological features (i.e. sampling designs and data analysis procedures), and the MMR rhetorical justification (i.e. MMR reference types and argumentation schemes) required several rounds of readings, a certain level of interpretation, and therefore intercoder discussions. Ten percent of the data was coded by the two researchers independently. To have a representative sample with balanced distribution across the five journals and over the 14-year time frame, four RAs from each journal were selected from equal intervals and were analyzed to identify pitfalls. This initial coding of an evenly distributed sample assisted us in compromising the discrepancies. The coding book was refined and finalized, and the rest of the sample was coded manually. Finally, Cohen’s kappa was computed since the data was categorical, resulting in a high level of agreement (κ = 0.87) (McHugh, 2012). Descriptive statistics (i.e. frequencies, percentages, and ranks) were implemented to analyze the data.

3. Results and discussion

3.1 MMR purposes in CALL RAs

Regarding MMR purposes, we found that triangulation (n = 119, 58.3%) is by far the most used purpose, followed by complementarity (n = 40, 19.6%). Development and expansion were equally ranked third, as each of these purposes appeared in 21 studies (10.2%). Our findings also indicated that initiation (n = 3, 1.4%) was proven to be the rarest purpose in CALL RAs. Overall, these results are consistent with Hashemi and Babaii (2013), Riazi and Candlin (2014), and Riazi and Amini Farsani’s (2024) findings in AL, signifying the leading role that triangulation plays in MMR studies and how authors use it to cross-validate findings from qualitative and quantitative strands.

Despite the strong presence of triangulation in MMR studies, its long-standing ontological issue, previously apparent in AL (Riazi & Candlin, 2014), was also observable in CALL RAs. In several studies, the adoption of heterogenous constructs, data, or analysis procedures, rather than dealing with homogenous constructs, “to cross-validate findings from one methodological approach against findings from another” (Riazi & Candlin, 2014: 154), made it difficult for us to make sense of the data/analysis and thus identify the MMR purposes. We identified some discrepancies between what authors explicitly stated and the actual purposes they pursued based on Greene et al.’s (1989) taxonomy. For example, Sundqvist (2019) used the term triangulation as her MMR purpose, and this was noted in our coding sheet; however, our reading of the methodology of this RA identified complementarity as the purpose. Nevertheless, in analyzing explicitly mentioned MMR purposes in the dataset, we heavily relied on the RA authors’ verbatim accounts. Such mismatches, which were also apparent in Riazi and Amini Farsani’s dataset (2024), might be ascribed to the fine lines of differences laid out between MMR purposes leading to wrong purpose selections. Moreover, they might be rooted in authors’ insufficient familiarity with MMR purposes and journal gatekeepers’ leniency concerning this matter.

Discussing MMR purposes in CALL RAs, we should also highlight the importance of complementarity, which was present in almost one-fifth of the studies, a result that might be related to the nature of CALL and the way technology is integrated into CALL daily practices. In other words, researchers in CALL tend to adopt new technologies to offer elaborations on single-strand data, which could be insufficient if not complemented with data from the other strand, and therefore assist the readership in perceiving CALL as a multi-layered social phenomenon (Riazi & Candlin, 2014). The remaining purposes (i.e. development, expansion, and initiation) have attracted less attention, probably owing to the challenges researchers face in comprehending and implementing them. Understanding the appropriate time and the adequate measures to develop and expand the strands of data/level of analysis cannot be instantly clear to CALL researchers when deciding on the purpose and design of their MMR studies (Riazi & Amini Farsani, 2024).

3.2 MMR designs in CALL RAs

In response to RQ2, our results demonstrated that core MMR designs were adopted in 116 (56.8%) RAs, whereas the rest of the data (n = 88, 43.2%) employed complex MMR designs (Figure 2), indicating the relative popularity of core designs in CALL research. Although the difference is not too large, it is safe to claim that core designs are more favored due to their less complicated data collection, analysis, and interpretation processes (Creswell & Plano Clark, 2023). In RAs built on core designs, explanatory sequential designs (n = 52, 25.5%) – that is, using qualitative data and analysis to expound the quantitative results – were the most frequent, followed by convergent designs (n = 35, 17.1%), in which quantitative and qualitative results are used for triangulation. The least exploited core design was exploratory sequential designs (n = 29, 14.2%), where quantitative data and analysis are utilized to explore qualitative findings. These findings confirm Riazi and Amini Farsani’s (2024) results, where the same order of core designs was identified.

Figure 1. Distribution of 204 MMR CALL RAs over the time span.

Figure 2. MMR designs in CALL research.

Complex designs showed rather remarkable disparities among its four subcategories. Experimental (interventional) trial designs (n = 59, 28.9%) stood out highly, compared to the other three (Figure 2). Multiple case studies (n = 20, 9.8%) were the second most used, while evaluation approaches (n = 7, 3.4%) and participatory research processes (n = 2, 0.98%) were the least used complex designs. In general, complex MMR designs are used less often than core designs in CALL research due to their apparent complexity, time-consuming nature, resource demands, and the need for advanced expertise, as pointed out by Creswell and Plano Clark (2018). Unlike Riazi and Amini Farsani (2024), who followed Johnson, Meeker, Loomis and Onwuegbuzie’s (2004) tripartite framework of MMR priorities or drives – (1) qualitatively driven, (2) quantitatively driven, and (3) qualitative-quantitative continuum – to classify MMR studies as principled and innovative regardless of whether they are core or complex, we classified complex MMR designs as innovative only if the authors provided sound argumentations to justify their choice. This will be unpacked later in Section 3.5.

3.3 MMR features in CALL RAs

Our findings show that CALL researchers have recognized the importance of describing the sampling designs and data sources, as 199 (97.5%) RAs explicitly mention these two aspects, 1 (.5%) RA had an implicit mention, and 4 (2%) RAs had no description of sampling and data sources. Research shows that explicit mention of these two facets is important (Hashemi & Babaii, 2013; Riazi & Amini Farsani, 2024). However, it is how quantitative and qualitative samples are selected (i.e. randomly or non-randomly) and how data sources of the two strands are integrated (i.e. concurrently or sequentially) that contribute to MMR transparency (Riazi & Candlin, 2014).

3.3.1 Sampling designs

As mentioned before, the significance of the sampling designs, or how the collected data from both quantitative and qualitative strands relate to one another, has been considered one of the most prominent features of MMR RAs (Fetters, 2020). Unlike Riazi and Amini Farsani (2024), who followed Onwuegbuzie and Collins’s (2007) bidimensional model of MMR sampling, probably due to the complexities of sampling procedures in various subfields of AL, we decided to zero in on the level of explicitness in reporting the collected samples along with their schemes (i.e. type and size). This decision was made because of our focus on one specific subfield of AL (i.e. CALL), not its entirety; it can also be an indication of the previously discussed pragmatism (Johnson & Onwuegbuzie, 2004; Tashakkori & Teddlie, 2010) in the analysis of CALL MMR (see Section 2.1.3).

As Table 2 presents, the fact that almost 98% of studies in the dataset specify the sampling designs and data sources explicitly attests to the awareness of CALL authors of the significance of mentioning these features of MMR, particularly when compared to the results of Riazi and Candlin (2014) as well as Riazi and Amini Farsani (2024) in AL journals, where the main components of the sampling designs were implicitly identified. In the analysis of the schemes of CALL RAs, we found that random sampling was used in more than 60% of the studies, while non-random sampling was employed in less than 40% of the dataset. By looking at the breakdown of each sampling, we realized that while 38 studies (18.6%) adopted intact classes, 9 (4.4%) chose purposive, 7 (3.4%) utilized convenience, 3 (1.4%) employed cluster, and only 1 study used stratified sampling. On the other hand, 46 studies (22.5%) in the strand of non-random samples adopted purposive sampling, 23 (11.2%) used convenience sampling, 5 (2.4%) did not specify the category of non-random sampling, and finally 4 (1.9%) employed stratified sampling. Overall, and contrary to the results of Riazi and Amini Farsani (2024), who found that nonprobability sampling designs are more prevalent in AL journals, our findings revealed that CALL MMR authors are more likely to adopt intact classes when they choose random sampling, and convenience selection when they adopt non-random sampling, owing to the feasibility issue and accessibility of such a cohort of participants in CALL research contexts (Choubsaz et al., 2024). However, when they shift toward non-random sampling, they are more inclined to use purposive sampling, probably due to a particular MMR design they have in mind at the outset of research or to adopt/evaluate a new technology in the learning/research environment. Transparency-wise, our results, at least in the random strand, align with Riazi and Amini Farsani’s (2024) in that one-third of the CALL authors did not specify the kind of random sampling they used.

Table 2. Sampling designs and data sources in MMR CALL RAs

In further analysis of the scheme (i.e. sample size), it was found that almost two-thirds of the studies had moderate or large sample sizes (see Table 2). This rather promising finding, which might be linked to the dominance of quantitative samples in lieu of qualitative ones in CALL research (cf. Riazi & Amini Farsani, 2024), testifies to the attempts of CALL authors to include larger sample sizes not only to achieve generalizability but also to boost their chances of publication in core CALL journals, an unstated convention in the CALL realm (Gillespie, 2020). On the flip side, the sample size in one-third of the studies in the dataset reported 30 or fewer participants. Sticking to the bare minimum sample size (Mackey & Gass, 2022), though understandable due to the limitations that CALL researchers might have by choosing intact classes, convenience sampling, or institution-related restrictions, is considered a long-standing issue in researching CALL (Gillespie, 2020). All in all, we can cautiously claim that sampling transparency and sample size in CALL MMR have not been taken into account either by CALL researchers or journal gatekeepers.

3.3.2 Data sources

CALL researchers have used various sources in their studies to collect qualitative and quantitative data. Interviews (n = 126, 61.7%) were notably the most used data source, followed by surveys (n = 78, 38.2%), questionnaires (n = 74, 36.2%), (pre/post) tests (n = 46, 22.5%), (classroom) observations (n = 23, 11.2%), and a range of academic tasks (n = 17, 8.33%) assigned to research participants/language learners. The immense popularity of interviews as a qualitative source of data was an expected finding given the empirical nature of CALL studies we investigated. Moreover, the presence of observations, either in the classroom or in the research context, through which CALL researchers were able to take field/reflective notes, among the top five data sources, was an unexpected finding. On the other hand, the prevalence of surveys, questionnaires, and tests can be seen as a point of similarity between CALL and other subfields of AL (Riazi & Amini Farsani, 2024). A major difference, however, might be the heavier weight of various computer-related tasks (online, storytelling, usage, chat-based, etc.), which might be related to the nature of CALL programs and courses worldwide, as found in our data.

The analysis of combined data sources in the dataset (we only present the highlights in Sections 3.3.2 and 3.3.3 to save space) could shed light on how CALL researchers mixed qualitative and quantitative data sources to achieve the MMR purpose they had in mind and answer the developed RQs more thoroughly. Based on the findings, the adoption of “interviews + surveys” and “interviews + questionnaires” were the most frequent combinations, appearing in 47 and 44 studies, respectively. These two are considered common combinations in MMR due to their simple design and implementation (Riazi & Amini Farsani, 2024). Additionally, “tests + interviews” and “tests + questionnaires” should also be recognized as relatively frequent combinations (19 and 15, respectively).

In the analysis of the MMR features in CALL RAs, it is also important to understand how data sources of the two strands are integrated. Regarding the sequential integration, in which the sources of data were used one at a time based on CALL researchers’ needs, most of the studies (n = 180, 88%) considered this approach fit for various design-related or logistical reasons (Garret, 2009). To elaborate, as discussed in Section 3.2, 81 (39.7%) of the MMR CALL RAs adopted exploratory/explanatory sequential designs and naturally leaned toward the sequential domain. Moreover, the MMR logistical aspects and the five MMR purposes pursued might explain the scarcity (n = 24, 12%) of concurrent data collection.

In the studies with a sequential design, the authors started collecting the data with a form of quantification, usually a survey, and then moved to the qualitative strand by conducting interviews with the research participants (n = 40, 22.2%). This corroborates the results of Riazi and Amini Farsani’s (2024), since they also perceive the QUAN → QUAL order a more frequent sequence in AL journals and probably an indication of researchers’ tendency to collect quantitative data first, understand the participants’ opinions using surveys/questionnaires, and then use that data to develop questions for the various forms of interviews. However, following this sequence in CALL RAs with this low occurrence can indicate that authors have taken a relaxed approach to design/feature transparency. This is because a certain degree of transparency in design hinges on the emphasis that authors lay on quantitative or qualitative strands or even both, the priority that we could not find in many of the CALL RAs. It was for the very same reason that no pattern or trend could be recognized in the studies with a concurrent design.

3.3.3 Data analysis procedures

Only 48 (23.4%) RAs in our study explicitly stated whether their statistical procedures were descriptive, parametric or non-parametric. Therefore, to investigate the statistical analyses, we relied on Field’s (2024) classification. Inferential tests of significance are broadly divided into parametric (assuming normal distribution) and non-parametric (not requiring distribution assumptions and used for ordinal data and small samples) (Mackey & Gass, 2022). As Figure 3 illustrates, parametric tests (n = 90, 44.1%) were used the most, followed by descriptive statistics (n = 65, 31.9%), both parametric and non-parametric tests (n = 24, 11.7%), non-parametric tests (n = 23, 11.3%), and 2 (1%) RAs did not include any statistical tests. These results are in line with Riazi and Amini Farsani’s (2024) findings, in which parametric tests were reported to be more preponderant than non-parametric counterparts. However, in their study, Riazi and Amini Farsani (2024) did not consider descriptive statistics. It is worth mentioning that most RAs in our study had more than one statistical test, resulting in a total of 353 instances.

Figure 3. Data analysis procedures in MMR CALL RAs.

Concerning descriptive statistical tests, frequencies (n = 35, 14.46%), percentages (n = 33, 13.61%), means (n = 14, 5.78%), and standard deviations (n = 11, 4.54%) were frequently used. The most prevalent parametric tests were t-tests (n = 67, 15.72%), ANOVAs (n = 58, 13.98%), and correlation (n = 24, 5.63%). Regarding non-parametric tests, Wilcoxon tests (n = 19, 4.46%), chi-square tests (n = 14, 3.28%), and Mann–Whitney U test (n = 11, 2.58%) were the most common ones (Table 3). Broadly corroborating Riazi and Amini Farsani’s (2024) results, the trends emerging from our analysis show that authors of MMR CALL papers use a wide range of statistical tests to accommodate both the qualitative and quantitative dimensions of CALL research. As Chapelle (2009) and Hubbard and Colpaert (2019) contend, the interdisciplinary nature of CALL research might be one main reason why authors opt for MMR designs and thus use various statistical tests. Another point perceived from the scarcity of non-parametric tests is that in MMR CALL RAs, the data lend themselves better to parametric tests, which are based on normal distributions and large samples. Moreover, despite their simplicity, descriptive statistics in MMR CALL RAs should be the least favored, as they lack the generalizability feature, which is highly in demand in CALL research.

Table 3. Statistical tests in MMR CALL RAs

Reporting qualitative data analysis procedures, alongside quantitative data analysis modes, is a key contributor to transparency in MMR studies (Creswell & Plano Clark, 2018; Riazi & Amini Farsani, 2024). Out of the 204 RAs, 151 (74%) RAs mentioned the qualitative data analysis procedures they adopted, while the rest, 53 (26%) RAs, did not. A total of 254 instances of qualitative data analysis procedures were identified in the data. Thematic analysis was used far more often than other procedures (n = 157, 61.8%), and content analysis stood second, followed by other less frequently used procedures (Supplementary Material B summarizes the qualitative data analysis procedures with their frequencies). While examining this aspect, we found that some studies mentioned what procedures they had used, but without unpacking how the analysis was carried out. In contrast, some authors described the procedures extensively and explained how they were adapted to suit the purposes they pursued. At times, the process of coding and theme identification was followed by exemplifications and reliability and validity measures to justify the suitability and practicality of the procedures.

3.4 MMR rhetorical justification in CALL RAs

As discussed in Section 2.1.4, we built on recent research (i.e. Creswell & Plano Clark, 2018; Riazi & Amini Farsani, 2024) to show how authors justify their adoption of IMMR by examining the MMR reference types, and we added that identifying argumentation schemes can enable us to assess the quality of justifications (van Eemeren & Garssen, 2020) the authors present in defense of their choice of MMR in CALL RAs. To our surprise, the findings (Figure 4) indicated that more than half of the CALL studies (56.3%) did not have any reference to the MMR literature.

Figure 4. MMR reference types in CALL RAs.

In other words, 115 CALL RAs did not include any MMR-related references to frame the purpose, design, or features of their MMR studies. On the flip side, 89 (43.7%) RAs ¹ referenced MMR literature to firmly establish ties with the MMR rudiments, informing readers of authors’ familiarity with the key figures in the MMR literature or their consultation with research methodology books/pertinent articles/book chapters related to MMR. Overall, 108 MMR references were used in these 89 CALL studies. Contrary to the results of Riazi and Amini Farsani (2024) in which MMR sources from various disciplines were the dominant category, in our analysis, general sources (i.e. sources with a general scope in research methodology) had the highest weight with 67 out of 108 instances (62%), followed by discipline-specific sources (the ones frequently used in AL) with 27 (25%) cases, and multidisciplinary (i.e. sources used not only in AL but also in other disciplines) with only 14 instances (13%).

To discuss, it might be understandable if we say CALL authors are less familiar with discipline-specific and multidisciplinary MMR sources compared to authors in the broad field of AL. This might be justified since CALL authors have integrated MMR into their research mostly in the past 14 to 15 years (see Figure 1), a trend that is certainly growing but needs more time to be incorporated into CALL authors’ research designs. Hitherto, when CALL authors have used a discipline-specific or multidisciplinary MMR source, they have mostly referred to John Creswell and collaborators (n = 9). Similarly, when referring to general sources, they have used Creswell and collaborators (n = 45) more than any other key figure in MMR. That said, the large number of studies that did not include any MMR-related references might be a downside of the existing CALL MMR RAs, an issue that needs to be addressed not just by CALL authors but also by CALL core journal gatekeepers.

Concerning argumentation schemes, 91 RAs (44%) did not have any argumentation in support of the MMR approaches employed. The authors briefly mentioned they used MMR, with no references to the literature, nor offering any reasons why they had done so. Perhaps they assumed that their employment of MMR can be perceived through the stated objectives/RQs of the study or the procedures. It is necessary to assert that justifying MMR in CALL RAs should not be overlooked or understated since this can harm the transparency of the methodology (Hashemi & Gohari Moghaddam, 2019; Riazi & Amini Farsani, 2024) and thus the credibility of the entire study (Creswell & Plano Clark, 2018). The argumentation schemes identified in the data are shown in Figure 5.

Figure 5. Argumentation schemes in MMR CALL RAs.

In symptomatic schemes, authors justified the use of MMR approaches by either juxtaposing their study context to a broader context, espoused by direct or indirect quotes from experts, or non-integral citations. In this manner, authors rely on evidential justifications (Arizavi, Jalilifar & Riazi, 2023), which add to the credibility of their methodology. Causal schemes should be seen as true instances of reasoning in which authors put forth any context-dependent justifications to support their choice of MMR and the effects that this selection might bring about. This argumentation scheme might be considered stronger than symptomatic schemes, as authors hold accountability for their MMR selection. Analogy-based schemes, where justification relies on similarity between authors’ MMR choices and other authors’ approaches, were almost absent, except for one instance in a combined scheme. The reason for this paucity might be that even in replicatory studies, the methodological features vary depending on contextual factors; therefore, it is self-evident that authors avoid stating that their MMR is an exact replica of other studies. Combined schemes (n = 31, 15%) can be viewed as the strongest scenarios because authors justify their MMR approach by deploying various argument types. In this category, the causal + symptomatic combination preponderates (n = 30), compared to causal + analogy-based and symptomatic + analogy-based variants.

Overall, these findings represent the prototypical argumentation schemes in academic discourse, which are in tandem with Arizavi et al.’s (2023) results, who found causal, symptomatic, and combinatory (causal + symptomatic) schemes as the most frequent argumentation schemes in the published ESP RA discussion sections. It is important to clarify that exploring the paradigmatic patterns of argumentations – that is, qualitative in-depth analysis rather than quantitative frequency-based examination of such patterns (van Eemeren, 2018) – though revealing, is beyond the scope of this study.

3.5 MMR adoption: Conventionality or innovation?

Based on our findings in Section 3 and to answer RQ4, we tend to argue that most CALL researchers have employed MMR approaches out of conventionality, which is characterized by adherence to conventionalized MMR norms and practices that are borrowed from sister AL subdisciplines or other social sciences disciplines. Most authors follow exemplary MMR designs and put up resistance to diversion from established conventions. This is evident from the type of data sources and data analysis procedures utilized in their studies, as well as the higher frequencies of fixed and/or core MMR designs, as compared to emergent or more advanced designs (Creswell & Plano Clark, 2018). In fact, overuse of surveys, interviews, and pre-/post-tests in predictable sequences attests to CALL researchers’ proclivity toward conventionality (Hashemi & Babaii, 2013).

Innovation in research, in contrast, is manifested by experimenting with new data sources, methods, and procedures, or by using the long-established ones in innovative ways by integrating various ontological and epistemological orientations (Riazi, 2016). Drawing on Riazi’s notion of IMMR, we can identify two potential contributors to innovation, namely, “complex MMR designs,” which are guided by RQs, as well as “causal or combined argumentative schemes,” which justify the innovation in the MMR utilized. Our analysis revealed that a few studies tried new possibilities in their MMR approaches. This is understandable because, as Riazi (2016) suggests, when authors fully comply with paradigmatic impositions and rely on traditional perspectives when conceptualizing research problems, rather than allowing RQs to guide their choice of MMR, they may hinder the development of IMMR. Surprisingly, even in studies where MMR could be perceived as innovative, the rhetorical justification for the innovativeness was either absent or poorly crafted. It is a truism that conventionality lays the foundation and paves the way for innovation; however, this does not mean that CALL researchers should blindly confine themselves to MMR-established norms and never try alternative ways of exploring CALL-related phenomena.

4. Conclusions and implications

This methodological synthesis examined the MMR purposes, designs, methodological features, and rhetorical justifications in 204 CALL RAs to reveal the status quo of MMR in CALL research. The results indicated that triangulation is the most used purpose, core MMR designs are predominantly favored, and the data is characterized by moderately sized random samples, with sources collected and analyzed in a sequential manner using parametric tests. Additionally, symptomatic argumentation schemes emerge as the most prevalent form of justification for the use of MMR: the findings that verify the prevalence of conventional rather than innovative designs. Based on our discussions and observations of CALL international research contexts in this article, it is now evident that CALL authors are implementing MMR at a higher rate, CALL core journals are welcoming submissions with MMR designs and CALL associations are attempting to raise awareness of MMR CALL research in their seasonal and annual meetings, gatherings, and conferences. However, contrary to Amini Farsani, Babaii, Beikmohammadi and Babaii Farsani (2022) and Riazi and Amini Farsani (2024), who staunchly defend the status quo of MMR in AL, we are unable to claim that the overall state of MMR CALL research is fully grown and healthy. One reason for this might be the rather short time that MMR has been implemented in CALL studies (about 15 years). Our reliance on MMR syntheses in the field of AL, but not MMR synthetic RAs in CALL, for referencing, comparing, and contrasting results has been an indication of the paucity of such studies in the CALL literature. This reliance underscores the need for further exploration of MMR syntheses within CALL. Additionally, more steps can be taken to familiarize the community of CALL researchers with design and implementation when adopting IMMR in their studies (Plonsky, 2024).

While it is true that MMR is gaining traction in CALL research, this should not be interpreted as castigating other research approaches due to their unviability and propagating only MMR. The implication is that to raise the number of high-quality IMMR studies in CALL research and encourage appropriate reporting of this approach, CALL core publishers and journal gatekeepers can dedicate special issues to MMR-based research practices. As one of the reviewers of this paper suggested, conducting interviews with CALL journal editors and analyses of submission guidelines could help us understand why insufficient attention has been paid to MMR implementation in CALL research, and how authors can best report MMR designs. This will be embarked on in a prospective study. Additionally, CALL associations such as WorldCALL, EUROCALL, and JALTCALL can work on establishing a new special interest group (SIG) that concentrates on promoting MMR in CALL publications. Prospective authors are expected to clearly state why they have chosen MMR, what design they have adopted, attempt to be as transparent as possible in writing about their data sources/data analyses tools, and refer to relevant MMR literature in their work (see Chen, Wang & Rodway, 2022, in CALL as an example of a study that clearly outlines the MMR purpose, design, features, and rhetorical justification).