The impact of using Memrise on student perceptions of learning Latin vocabulary and on long-term memory of words

Introduction

During my PGCE I had a class of eight students in Year 11 studying for the OCR GCSE in Latin who had to master knowledge of 475 words for their exams. Their recent unseen translations demonstrated weak performances due to poor vocabulary knowledge. On interviewing the students, most lacked a systematic approach to learning vocabulary. Since the end-goal of studying Latin today is more often the reading of ancient texts, the requirements for vocabulary learning differs from that of other languages. Composing sentences in Latin is an increasingly rare skill too. Thus by rarely needing to produce Latin, students have no means of actively practising the language. Whilst students often find learning vocabulary arduous, I found that with Latin students find it particularly difficult to retain knowledge. In the school the GCSE Latin lessons are teacher-led and mostly involve the dissemination of information for the students to record and process. The pressure of covering the exam syllabus and difficulty of integrating vocabulary activities into this class led me to the possible solution presented by Memrise.

Memrise is an online self-study programme which I had used myself to learn Latin vocabulary. Its main benefits for me were its portability, accessibility (via computers and Apps on smartphones and tablets) and ease of use. Created by researchers in psychology and languages, the programme is designed to build up long-term memory of vocabulary through systematic and frequent testing of vocabulary items. The school in which I was based relied on assessing students on new clusters of words each week, thus assessing the students' short term memories. By contrast, Memrise utilises an algorithmic review system whereby students must revisit words repeatedly but at a particular scheduled time, with reminders to review occurring at the time when the memory is most likely to fade. Particular focus is given to words that individual students continually forget, encouraging focussed effort in weaker areas of their memory. I felt that Memrise's design might assist in establishing long-term memories of words for the students and may compensate for limitations of class time. Also, the prevalent use of smartphones in students' free time suggested that Memrise might be particularly compatible with student habits and dispositions.

Thus my research questions were:

• • What are Latin students' perceptions of learning vocabulary?

• • What are the students' perceptions of learning vocabulary through Memrise?

• • Is Memrise more effective in improving Latin vocabulary test performance?

What can other research tell us about learning vocabulary?

Student perceptions

A number of research papers, suggested that having a variety of methods for learning vocabulary was a benefit for students. For example, Macaro states that for learning large volumes of words ‘…the learner needs to rely less on the teaching process and more on themselves and their own self-study practices’ (Macaro, 2001, p. 38). This was echoed by Deagon who cites a study by Gregorc which indicated that ‘most individuals in the sample indicated a desire for a variety of approaches in order to avoid boredom’ (Deagon, 2006, p. 35; Gregorc, 1984, p. 54). However, this says little about its effectiveness. Kornell (2009) found that students wrongly perceived that their preferred strategy of re-studying word-pairs was more effective than testing themselves. The study of Huesler and Metcalfe (2012) on learning word pairs demonstrates how participants in their study had false beliefs about the most effective learning methods. They concluded that participants ‘failed to realize that generating errors greatly facilitated recall’ (2012, p. 519). Although variety and avoidance of testing may keep students interested, it alone cannot address the need for students to be trained in how to learn effectively, something which Kornell believes in strongly.

Brown, Roediger and McDaniel suggest that learning that requires students to generate responses or testing is far more effective but is more effortful, saying that ‘the greater the effort at retrieval, the more will be retained’ (2014, p. 44). Yet students tend to drop items when they perceive they have been learned and that they do this far earlier than they should (Brown et al., 2014, p. 124). Pyc et al. suggest that learning is a structured and logical process, stating that using self-regulated learning can be powerful ‘provided that they continue to review information until they can correctly recall items multiple times at spaced intervals’ (2014, p. 86). These evidence-based studies therefore suggest that effective self-regulated learning must be taught rather than assumed.

Scheduled retrieval practice

Overall, the literature favours testing as a means of learning. Carrier and Pashler (1992) demonstrated testing was more effective than just studying. Similarly Roediger and Karpicke (2006b) confirmed the benefits of testing using three conditions on sizeable participant populations: STTT, SSST and SSSS (where S stands for study and T for test). Those who studied for longer retained more knowledge in the short term, but those tested more gained long term retention and forgot less.

Arnold and McDermott (2013) conducted a study that confirms that testing also affects studying positively as students become more aware of gaps in their knowledge and their learning is strengthened through practised retrieval. This ‘test-potentiating effect’ leads to better retention of knowledge in later study phases (Arnold & McDermott, 2013).

The nature of the study correlates with Memrise, where word-pairings are learned then tested. Moreover, since Memrise uses algorithms to test students on the words they find more difficult (See ‘Learn more about the garden: http://www.memrise.com/about/’), it seems to aid the user in maximising the test-potentiating effect. Moreover, Kornell, Hays and Bjork (2009) found that ‘unsuccessful retrieval attempts followed by feedback led to more learning than did spending an equal amount of time studying the cue and target together’ (Kornell et al., 2009, pp. 995-6).

Overall, previous research highlights the importance of focussed and directed self-study for consolidation of knowledge, the benefits of effortful retrieval practice, and that students instinctively are poor judges of their own performance and the best methods of learning vocabulary. Thus I am keen to see what perceptions they have of Memrise and its effectiveness.

Methodology

There were two arms to this study. The first was a quantitative arm comparing students' using Memrise to learn vocabulary with their normal self-study habits. The second arm qualitatively looked at student self-reported study habits, and their experience of using Memrise.

Qualitative Arm

Prior to exposure to Memrise, students from Years 8-11 were asked to complete a ten question questionnaire on their perceptions of Latin vocabulary learning within the wider context of learning Latin. Year 11 students were also invited to have an interview discussing their questionnaire responses, based on a semi-structured format. Following a four-week exposure to Memrise, Year 11 students were invited to attend a group interview to discuss the experience of using Memrise. A phenomenological approach was used to analyse transcripts to facilitate my understanding of the students' lived experiences.

Quantitative Data:

A pilot study was performed where Year 11 students were tested on the 475 words they are required to learn for their GCSE exam. The words were then stratified on the basis of percentage of students correctly translating the word.

A selection of 100 words was chosen from the group of words that 62-75% of students correctly translated in the pilot study. These words were then split again into two groups of 50 that were matched for difficulty based on the pilot test. Students from Years 8-11 inclusive were provided with access to Memrise, which had been prepared with one of the groups of 50 words (from now on known as “Red” words). They were instructed to use Memrise to learn the Red words, and to use their normal self-study methods to learn the other group (the “Blue” words). A pre-intervention test was performed on both Red and Blue words, and then a repeat test was performed four weeks later. Paired t-tests were performed using each student's original results as the control for their post-test results. Cohen's d was then calculated to examine effect size.

Findings

Data from 34 participants were rejected on the basis that they did not complete either a pre- and/or post-intervention test on the hundred words or use Memrise for reasons such as lack of Internet access.

Pre-intervention

The pre-intervention data questionnaires and interviews revealed five main themes and 26 sub-themes (Figure 1).

Figure 1. | Themes and sub-themes from pre-intervention data.

1. Intrinsic difficulty of Latin

Latin was perceived as intrinsically difficult to learn, some words more so than others:

“Some words are easier because they have derivatives, although sometimes the verbs are harder” (Year 11 student). Other difficulties were described as “variations [of] word endings” (Year 10 student) and “lots of different forms” (Year 9 student). The quantity of words was an issue too: “We have so many words to learn” (Year 8 student).

2. Internal experience of vocabulary learning

Derivatives and similar appearances to the Latin or English word were perceived to ease difficulties by “being close to English” (Year 8 student). Yet, when “many words… are similar” (Year 11 student), Latin is perceived as difficult. Although, younger students were generally vague on what determined word difficulty, two higher-attaining Year 8 students described derivatives as helpful.

Students often described Latin in figurative language. When explaining how they perceived the process of vocabulary learning, they spoke of memory as being good or bad: “I don't have a good memory” (Year 11 student). Some spoke of memorising as a struggle: “I find it hard to memorise lots of words” (Year 8 student). Words were perceived as either ‘sticking’ (or not) in their head: “It seems I can't get the words in my head” (Year 10 student). Notably, Year 11 students dominated the sub-theme of memories ‘sticking’ in the mind. Others thought vocabulary learning was “a natural process” and yet a “struggle sometimes” (Year 11 student). Students perceived language learning as boring and not motivating: “It is difficult to motivate yourself… [to] start doing something slow and boring” (Year 8 student).

3. Vocabulary in context

I observed comments about how encountering the Latin words in stories enabled students to consolidate their meanings: “Going through the books… get those words into my head” (Year 11 student). Similarly, others mentioned how valuable vocabulary knowledge was for reading stories: “You could do better if you translate it [unseen Latin] with just vocabulary rather than if you knew all the cases.” (Year 11 student). The medium of story was described as helpful for deducing unfamiliar words: “If I can understand some of the words… then try and work it [unseen Latin] out from there” (Year 11 student).

Thus these sub-themes I grouped together as student perspective of vocabulary in context (as opposed to vocabulary in isolation).

4. Logistics and organisation of vocabulary learning

Also present were various comments about the difficulties students encountered when trying to set aside time for vocabulary learning. Some spoke about being busy generally: “I do not have much time” (Year 9 student). Concerns in school and home life competed for time in student lives: “I have lots of things to do other than school work” (Year 8 student). Students felt pressure from other subjects: “I have other homework to complete” (Year 9 student). Some were concerned about how long to spend learning vocabulary and the deadline affected when (and if) revision took place: “Sometimes on the weekend or just before a test” (Year 11 student). Notably Year 8 students feature prominently in factors that serve as barriers to learning as shown in Figure 2.

Figure 2. | Occurrence of themes in pre-intervention qualitative data.

5. Methods and strategies employed in vocabulary learning

Students spoke about the ways in which they learned vocabulary and mentioned the role of notions such as the novelty of “silly sentences” (mnemonics or made-up phrases): “I just remember them” (Year 11 student). The strategy of repetition was also mentioned by a number of students: “I'm not very good at remembering things so just repeating a lot” (Year 11 student). Students mentioned how they had clear preferences: “Those three [methods] help me so I don't want to do anything that might confuse me” (Year 11 student). Some benefited from the help of others: “I prefer to have someone say a word to me and I say it back” (Year 11 student).

Other students mentioned specific ways such as: “Looking over lists, memorising” (Year 8 student).

Pre-intervention analysis:

The younger years and the Year 11 students differed dramatically in the way they perceived the Latin language and processes of vocabulary learning (see Figures 3 and 4). They saw more obstacles and difficulties to organising regular vocabulary learning including quantity of words and the logistics of organising revision. Moreover, they perceived it as intrinsically difficult, with one's memory being either good or bad. Yet Year 11 students perceived that one's ability to retain memories varies over time (“remembering them is harder after a while”) and shifted difficulty to the words rather than the process, with some words being more difficult than others to remember. The Year 11 students also mentioned derivatives and stories as helpful and mentioned this more than any other year group. More often than the Year 11 students, younger students talked about being busy with other subjects and only learning vocabulary for tests. Year 11 students reflect more on the usefulness of “going back over” course books and consolidating knowledge, in spite of the difficulties of developing long term memories for words. Thus older students, with exams at the end of the year, seemed to perceive the benefits of effortful recapitulation of material. Perhaps without looming exams, the Year 8 and 9 students will have less impetus for learning vocabulary and understandably see less importance in regular vocabulary learning. Clearly then students seem to need extrinsic motivation for revision in the form of assessment or perhaps the ability to comprehend an interesting narrative.

Figure 3. | Years 8, 9 and 10 questionnaire responses.

Figure 4. | Year 11 questionnaire responses.

These trends may indicate that younger students lack perspective about the purpose of vocabulary learning in the long term. Thus only the Year 11 students and those higher-attaining Year 8 students seemed able to talk about solutions for overcoming difficulties of vocabulary learning as the others fail to see value in the endeavour (see Figure 2). What also emerges from the data is that individual students generally have clear preferences of methods. Thus it would seem that whether or not a method is effective or not, habit and personal preference play a key role in determining methods for vocabulary learning.

Figures 5 and 6 show the viewpoints of students on effective and ineffective methods (taking into account only responses that were “very” or “quite” effective / ineffective). Derivatives and communal learning through family and friends demonstrate an overall preference for communicative approaches to Latin. Year 11 students differ from Year 8 students in perceiving look-cover-write-check as ineffective. As Year 11 students rely on long term vocabulary retention, one could hypothesise that this method may have proved ineffective for their purposes yet suits Year 8 students who will not be assessed in national exams but rather formatively in class. Neither group displays much preference for testing programmes online.

Figure 5. | Perceptions of effective (blue) and ineffective (red) methods (year 8, 9 and 10).

Figure 6. | Perceptions of effective (blue) and ineffective (red) methods (Year 11).

It may be that older and/or higher-achieving students may be tacitly aware of the benefits of meta-learning for vocabulary learning overall and have extrinsic motivation for seeing long term retention as achievable in theory but elusive. Middle- and low- attaining younger students, lacking perspective, seem less concerned with finding effective vocabulary learning for long term retention, thus devaluing or seeing it as unattainable.

Post-intervention Findings

From the post-intervention focus group interview six main themes and 18 sub-themes were identified (Figure 7).

Figure 7. | Themes and sub-themes from post-intervention data on Year 11s.

1. Advantages of the Memrise platform for learning

Students perceived there were a number of advantages of learning vocabulary with Memrise including being both “quicker to access” and more time-efficient: “It helped me learn the vocabulary quicker so it made sense to put aside some time for it” (Year 11 student). The notion that it was helpful or useful featured strongly: “It's been so helpful” (Year 11 student). Students considered it easier to use: “I love it. I wouldn't go back to look-cover-write-check. I found it so much easier” (Year 11 student). Yet by far the most recurring sub-theme was the convenience of Memrise with a typical occurrence being: “I was able to do it anywhere and at any time” (Year 11 student).

2. Perceptions of improvement

Students felt confident in their perceptions that Memrise has improved their vocabulary learning. Typical responses include: “Convenient and [it] works” (Year 11 student) and “I have improved thanks to this program” (Year 11 student). In addition to perceived improvement in confidence, a notable theme was that students felt they could have had better knowledge and would have been more successful in Latin had they had Memrise at an early stage in their study of the language. For example: “I wish we… had it earlier on so we could know more words” (Year 11 student).

3. Sense of a learning community

Whilst the idea of sharing learning through ‘Mem’ cards on Memrise is a positive side to the perceived learning community, competition dominates this theme with strong feelings about experiencing the Memrise leader-board: “I'm at the bottom, and [my score's] not even low” (Year 11 student). There was strong sentiment about being ‘behind’ others. This is closely linked to the theme of intrinsic motivation, where the leader-board and the level of points that determines one's ranking, features prominently.

4. Meta-learning

This theme covered features that demonstrated students' gains in the area of meta-learning, knowledge about the learning process. Students perceived that Memrise repeats the words you get wrong and reinforces the words they considered more difficult: “It enforces my learning and stops me from ignoring those words which I don't know” (Year 11 student). Similarly it is a noteworthy that one student seemed to deduce how assessment facilitates learning: “Makes me revise it every day… testing to make sure I actually know the words” (Year 11 student). Along similar lines, students perceived gains from repetition: “If you think you've remembered a word and you go back… and you don't know it. You can go back over it… it keeps it in your head more” (Year 11 student). Students expressed interest in the ability to personalise mnemonics: “I did the paene pasta one. paene is nearly” (Year 11 student).

Linked to this, students described ways they would extend Memrise features: “For another bit of our course, I'd put all the verb endings on” (Year 11 student); and also more universality of features: “I'd like there to be more of the features on Apple products like the Leader-board” (Year 11 student). Students also saw merit in the gradual and progressive development of knowledge: “[It] builds you up to remembering whole words in stages” (Year 11 student).

5. Intrinsic motivation

The notion of competition and scoring points for learning and reviewing words was perceived as motivating by the students: “You had a set target to work towards… able to see how many words you have learned…this offers a form of motivation” (Year 11 student). Discussion on the leader-board involved many of the students: “You want to beat people… I was like “yeh!” I'm at the top” (Year 11 student). Others mentioned that learning vocabulary with Memrise was engaging: “It makes it more fun” (Year 11 student). A minor feature was a more explicit perception of success:

“So [I] thought it was better to spend time on Memrise than reading the flashcards I made” (Year 11 student).

6. Desire to change integral features

Although the second smallest theme, some students suggested changes that would undermine the way in which Memrise is designed to work. The email reminders to review words in spaced practice (an effective method for long-term retention) were perceived to be annoying: “I just ignore them” (Year 11 student). Similarly, students wanted to drop words after they perceived that they definitely knew them: “So if you know that you know a word you don't want to have to go over it when there are other words you might be struggling on more” (Year 11 student). One student mentioned that she would “…allow slight spelling mistakes… when I'm typing fast” (Year 11 student), which threatens the need to differentiate between easily confused and similar words.

Post-intervention analysis:

Most of all students mentioned the advantages of Memrise as a method of vocabulary learning. Following close behind is the theme of meta-learning which possesses the greatest number of sub-themes. It appears that even though I chose not to explain how Memrise works, students still saw many advantages and gained a new perception of vocabulary learning through Memrise that was not evident before the intervention. The accessibility and portability of Memrise were a significant benefit that is likely to have increased student exposure to the vocabulary.

The lower frequency of discrete disadvantages and inconveniences is also worth noting. One might deduce that the onus on the student to find time and space to formally schedule vocabulary learning and deciding on an effective method is reduced using Memrise.

Students spoke very positively about their perceptions of their improvement in vocabulary and unequivocal terms to talk about improvement, saying such things as “I have improved…” and “It allows me to improve”. Moreover, students demonstrated that they were driven by a desire to be top of the Leader-board. Memrise seems to foster some sense of learning community with students experiencing cooperative learning through sharing their revision (‘Mem’) cards, which they created on Memrise themselves.

This perception of success and improvement in vocabulary knowledge and learning is also reflected in the students' beliefs that, had Memrise been introduced earlier, they would have improved in Latin more rapidly and would now be better at Latin. This demonstrates the students' confidence in their perceived improvement through Memrise. The qualitative data will provide some idea as to whether this perception is well founded or not. Nevertheless, I hypothesise that it is in part a self-fulfilling prophecy: the more successful they perceive the programme is, the more time they devote to it and investing greater effort in learning enhances retention of knowledge as the literature suggested.

The Meta-learning theme featured prominently and seems to suggest that students have developed a better understanding of how Memrise works and why it might be effective. They talked about notions such as spaced practice, reinforcement through repetition and even testing as learning. Not only are these features which the research from the literature review strongly suggests are effective, but the students also seem to be transferring these observations to their perceptions of improvement. Despite these student comments, students still desired to change integral features of Memrise. Their keenness to reduce or ignore spaced and repeated practice would, according to the research in this area, reduce any positive impact Memrise has. Still, this was a relatively less important theme overall, with students speaking more about advantages and demonstrating ownership of Memrise.

Quantitative data: findings

Paired T tests were performed for the Red words and Blue words, stratifying by year. Significance was set at p < 0.05. The results can be seen in Table 1.

Table 1

Both methods proved significant. Therefore, Cohen's d was calculated to assess the effect size of Memrise compared to other methods.

Figure 8 shows the effect size for other methods of vocabulary and Memrise overall and according to year group. The results suggest overall Memrise is more effective over the population of Latin students. It is significantly more effective for Year 9 students and also very effective for Year 8 students. For Year 11 students Memrise is still more effective than other methods. However, for the Year 10 students Memrise is less effective than the other vocabulary methods which the students used. It is important to note that the Year 10 students received fairly regular vocabulary tests prior to the intervention, which may account for the findings that their habitual vocabulary learning methods are more familiar and therefore more effective for them.

Figure 8. | Effect size of blue words and red words according to year group (using Cohen, J. (1988)).

Limitations of the research

The study is slightly limited as it was not able to separate the well-known phenomenon of learning by regular assessment, from any potential extra benefit from using Memrise. There are also limitations due to the lack of post – intervention data from Years 8, 9 and 10 because of time constraints. In addition, since students failed to retain and use learning journals, it was not possible to assess the time devoted to learning either Red or Blue words. Thus whilst the data seemed to show the successes of Memrise for retention of words, it is difficult to ascertain how much of this progress is due to the students simply not revising the other words.

In light of these limitations, it would be appropriate to undertake studies comparing Memrise with other methods that account for which methods are being used and for how long. With more controls on the amount of time to be spent on each, a greater amount of accuracy would be possible when analysing which method is more effective. Similarly, delayed testing and repeated testing of students after the interventions would enable the researcher to observe the long-term retention of words and which method has a longer effect after the intervention. An additional control that would enhance the reliability of the data would be to conduct a crossover study so that whilst half of the participant group used Memrise for the red words and other methods for the blue words the other half of the participant group did the opposite.

Conclusion

The quantitative data strongly support the superiority of Memrise as a way of learning Latin vocabulary, over the methods that students would otherwise self-select. The students found using Memrise enjoyable and convenient, and therefore it would be a useful tool for Latin teachers to employ in teaching vocabulary. Based on the Year 10 results, it is possible that the improvement due to Memrise is at least partially due to the phenomenon of learning by regular assessment.