Introduction
The considerable contribution of vocabulary knowledge to reading comprehension (RC) (Pearson et al., Reference Pearson, Palincsar, Biancarosa and Berman2020) has led to the development of receptive and expressive vocabulary assessments in different languages. Research reviewing extant vocabulary assessment tasks has emphasized the importance of defining the purpose of the test, describing the development procedures both theoretically and practically, as well as the need for stricter validation procedures (Pearson et al., Reference Pearson, Hiebert and Kamil2007; Schmitt et al., Reference Schmitt, Nation and Kremmel2020). The importance of vocabulary knowledge is also evident by the fact that academic word lists, which set out explicit instruction goals in this domain, have been compiled for different grade levels (Coxhead, Reference Coxhead2000, Reference Coxhead2011; Gardner & Davies, Reference Gardner and Davies2013). However, existing assessment measures and word lists typically approach vocabulary as a single construct without distinguishing between word types, and in particular, do not include emotion words (Coxhead, Reference Coxhead2000, Reference Coxhead2011). The distinctions between emotion and nonemotion words are supported by broad theoretical justifications. Emotion words are different from both other abstract and concrete words in their tangibility, imageability, and degree of connection and availability in a semantic network (Altarriba & Basnight-Brown, Reference Altarriba and Basnight-Brown2010; Altarriba et al., Reference Altarriba, Bauer and Benvenuto1999; Pavlenko, Reference Pavlenko2008). Differences between emotion words and nonemotion words are also evident in information processes; for example, emotion words were found to be better recalled than nonemotion words (Kissler et al., Reference Kissler, Herbert, Peyk and Junghofer2007; Kissler et al., Reference Kissler, Herbert, Winkler and Junghofer2009).
From a developmental perspective, emotion words are acquired at later ages, usually after entering school, compared to general verbs (for example, action verbs or movement), and even later than the development of cognitive words (e.g., “thinking” and “conclusion”) (Ravid & Egoz-Liebstein, Reference Ravid and Egoz-Liebstein2012). Given these differences, it is important to study the role of emotion words in RC as well, since it may diverge from that of other types of words. In the present study, we developed a curriculum-based theoretically motivated vocabulary assessment in Hebrew, with a specific focus on emotion words. We then tested how well this assessment might predict RC compared to a previously validated measure of general vocabulary.
Vocabulary and RC
Vocabulary is the conceptual knowledge of words and their meanings (Argaman, Reference Argaman2006; Perfetti, Reference Perfetti2007; Silverman et al., Reference Silverman, Proctor, Harring, Doyle, Mitchell and Meyer2014). Readers need to understand the meaning of the words they encounter in texts, in order to be able to construct their meaning. This is especially important in higher grades, since text complexity increases and requires knowledge of more sophisticated vocabulary (Neuman & Wright, Reference Neuman and Wright2014). A well-established base of broad and varied vocabulary contributes to the comprehension of unfamiliar words as well (Ricketts et al., Reference Ricketts, Bishop, Pimperton and Nation2011; Shahar-Yames & Prior, Reference Shahar-Yames and Prior2018). A meta-analysis conducted by the US Institute of Education Sciences (Pearson et al., Reference Pearson, Palincsar, Biancarosa and Berman2020) found that vocabulary was the most powerful predictor of RC. However, one of the critical conclusions of the study was that while the different vocabulary measures included across studies were robust in predicting RC, there has not been a systematic effort in the reading field to justify the procedures of selecting words for assessment and instruction. Others have also expressed the critique that many vocabulary assessment measures are not theory-based (Pearson et al., Reference Pearson, Hiebert and Kamil2007) and refer to vocabulary as a single general structure without addressing different types of vocabulary.
Vocabulary assessment
The contribution of vocabulary to academic achievements in general and to RC in particular has led to the development of various tools for assessing this knowledge, as well as to the development of academic word lists. These lists contain words that are essential for comprehension in various disciplines (Coxhead, Reference Coxhead2000) and are based on the frequency of occurrence in academic materials and content databases (Coxhead, Reference Coxhead2000, Reference Coxhead2011; Gardner & Davies, Reference Gardner and Davies2013). One of the best-known lists is the Academic Words List, published in 2000 by Coxhead for university students. Subsequent word lists were also developed for younger students, such as the Oxford wordlist used in Australian primary schools (2021). Most of these lists refer to vocabulary as a single general construct and do not identify different types of words.
Similarly, the educational field of vocabulary assessment has yet to offer distinctions between vocabulary types. Over the years, principles for word choice were formulated for assessment tasks. The National Assessment Governing Board (NAGB, 2005) defined a set of word selection criteria: words which are the characteristics of mature language and appear in written language; words that are not basic or common but also not uncommon, or that reflect the specific subject (Beck et al., Reference Beck, McKeown and Kucan2013); words that are found necessary for understanding the main ideas in texts; and words found in learners’ reading material. However, there are studies that indicate gaps and limitations both in the measurement procedures and in how the tools are validated (Schmitt et al., Reference Schmitt, Nation and Kremmel2020), as well as in how words are chosen for common vocabulary assessment tools (Pearson et al., Reference Pearson, Hiebert and Kamil2007).
The first gap is related to methodological aspects. Schmitt et al. (Reference Schmitt, Nation and Kremmel2020) claim that background materials published with vocabulary measures provide no theory or guidance on how words were chosen. Thus, there is no way of knowing why these specific items were chosen rather than others (Pearson et al., Reference Pearson, Hiebert and Kamil2007) and whether assessment measures should be based on vocabulary from the curriculum or on another criterion, such as the frequency of the word in the language.
The second gap is related to validation. Schmitt et al. (Reference Schmitt, Nation and Kremmel2020) evaluated vocabulary measures and found that most measures were launched with poor or insufficient evidence as to their validity. Thus, it is important to develop vocabulary assessment tools while adhering to higher requirements and standards of tool validity.
Beyond the gaps in vocabulary measure development procedures and how they are validated, extant vocabulary measures also do not include different types of words. The importance of integrating diverse words into vocabulary assessment is also expressed in the National Assessment of Educational Progress (NAEP, 2009), which emphasized that selected words should be necessary for a variety of contexts and areas and can express concepts, ideas, actions, or emotions.
It seems that even in widely used vocabulary measures, such as the Wechsler Intelligence Scale for Children (WISC) (Kaufman, Reference Kaufman1994) and the Peabody Picture Vocabulary Test (PPVT) (Dunn & Dunn, Reference Dunn and Dunn1981), insufficient information is given on how words were selected to be included in the test. The PPVT Manual (Dunn, Reference Dunn1965) for example does not include a comprehensive explanation of how the words are chosen. Thus, it is noted that out of a collection of 3,885 words taken from a dictionary, 2,055 words from a variety of topics were utilized, but there is no explanation as to why specific words were chosen over others. In a more recent version of the PPVT (PPVT-4_2007), there is only general information about the choice of words for the test. For example, it is stated that the words cover a wide range of vocabulary levels. The words represent 20 content areas (e.g., actions, vegetables, tools) and parts of speech. However, there is no detailed information about how the words were chosen in each category or information about word frequency and length. Similar gaps can be identified in common vocabulary tests in Hebrew and in additional languages. One common vocabulary measure in Hebrew comes from the Elul battery (Shatil et al., Reference Shatil, Nevo and Breznitz2007). In this test, a word is presented in each question and the students are asked to identify the picture that represents the word most appropriately. According to the description, in order to select the words, the research team was exposed to diverse items in the professional literature. However, it is not clear exactly what professional literature was at the base of the selection process and how these items were selected. The manual also mentions that in formulating the items for the test, each item was run through a sequence of tests, but it is unclear what these tests entailed and what were the criteria for selecting certain items over others. The Elul battery was administered to students from 120 classes from different schools. Based on the data collected, norms for each grade level were calculated for each subtest.
In a recent revision of the Elul test, conducted by Shany and colleagues (Reference Shany, Prior and Blicher2017), all the words appearing in the original subtests for the third and sixth grades versions of the test were administered to 20 students from each age grade level (from first to sixth grade). Based on the accuracy of the various age groups, the original items were ranked by level of difficulty. The revised test includes a single list of words to be administered to all age groups, and words were presented with increasing level of difficulty. The test begins with common words in the language (such as “jump”), and towards the end of the test, more complex, specific, and rarer Hebrew words are presented (such as “cycle”). However, the revised version relies on the words included in the original Elul subtest, again, for which there is only scant information about how they were identified and selected. To the best of our knowledge, emotion words have not yet been found systematically in vocabulary measures.
Characterizing different emotions
According to Ekman’s model of emotions (Reference Ekman and Cole1972), basic emotion develops in early childhood and often arises as a result of direct and specific circumstances that are easy to identify from facial expressions, even without knowing the context (Ekman, Reference Ekman1982; Lewis, Reference Lewis1993a). Ekman identified six basic emotions (sadness, fear, disgust, joy, anger, and surprise). In contrast, nonbasic emotions arise as a result of more complex circumstances and require sophisticated cognitive ability to understand (Lewis, Reference Lewis1993b). They develop at a later stage and continue to develop during the school-age years (Harris, Reference Harris1993; Lewis, Reference Lewis1993a). The assessment task developed for this study included both basic and nonbasic emotions
According to Scherer’s (Reference Scherer2005) Alternative Dimensional Structures of the Semantic Space for Emotions model, emotions can be mapped across four dimensions: valence (positive vs. negative), level of arousal (active vs. passive), degree of activation (obstructive vs. conducive), and degree of control (high vs. low). These four dimensions form a wheel divided into eight parts. Each part reflects two main characteristics and includes several emotions within it. One slice includes all the emotions that correspond to the active qualities with high control (e.g., lust, pride, ambition). The second, opposite slice, includes the set of emotions that correspond to passive characteristics and low control (for example, indifferent, hesitant, embarrassed, bored, and sad). The third slice includes the set of emotions that correspond to traits of negative values and low control (e.g., lonely, despairing, depressed, dissatisfied, disappointed, and unhappy). The opposite fourth slice includes emotions with positive values and controllability (e.g., enthusiastic, eager, interested, amused, and happy). The next slice contains negative values and a degree of activation (e.g., frustrated, suspicious, rejected, anxious). Opposite to this are positive emotions and high degree of activation (e.g., in love, confident, hopeful, happy, calm, sociable, satisfied). The final two slices relate to a high degree of arousal (active) with low degree of activation (e.g., jealous, afraid, angry) and opposite it a low degree of arousal (passive, calm) with a high degree of activation (e.g., longing, empathic). In the current study, we adopted Scherer’s model (Reference Scherer2005) for designing a vocabulary assessment tool in Hebrew, such that it included words from all eight slices of the model.
From emotions to emotion words
Emotion words are recognized as a lexical choice that directly reflects an emotional state, for example, joy, pride, and frustration (Pavlenko, Reference Pavlenko2008). Emotion words provide information about inner states, attitudes, beliefs, social connections, motives, values, and behaviors (Shiota & Kelther, Reference Shiota and Kelther2005). The words can be negative or positive and can be connected to the arousal of physiological responses (Pavlenko, Reference Pavlenko2008; Hsu et al., Reference Hsu, Jacobs, Citron and Conrad2015). This definition does not include emotionally laden words which describe expressions of emotion indirectly, such as curses or affectionate phrases (Pavlenko, Reference Pavlenko2008). Emotion words express the connection between the subjective experience of an individual and her/his environment on the one hand and the emotional experience and her/his future action tendencies on the other (Hobbs & Gordon, Reference Hobbs, Gordon and Khurshid2011). Emotion words are often very meaningful and play an important role in the perception of experiences and interpretation of the individual in diverse emotional states (Gendron et al., Reference Gendron, Lindquist, Barsalou and Barrett2012).
The ability to name emotion vocabulary is a basic ability among a variety of emotional abilities (Denham, Reference Denham2007; Harris, Reference Harris1993; Izard et al., Reference Izard, Schultz and Ackerman1996). Emotion words are most often explored in the context of social behavior (Izard et al., Reference Izard, Fine, Schultz, Mostow, Ackerman and Youngstrom2001; Miller et al., Reference Miller, Gouley, Seifer, Zakriski, Eguia and Vergnani2005), and they may reflect cultural differences in the emotional lexicon (Fontaine et al., Reference Fontaine, Scherer and Soriano2013).
Emotion words vs. nonemotion words
Emotion vocabulary has been less explored in the context of cognitive and reading processes and rarely in the context of vocabulary assessment. We hypothesize that there is a strong basis for treating emotion vocabulary as separate from nonemotion vocabulary for several reasons.
The first reason is that emotion words have unique characteristics. Emotion words are conceived as mental words, namely abstract words that describe internal processes or situations, including emotions (Hall & Nagy, Reference Hall, Nagy, Gopnik and Gopnik1986; Ravid & Egoz-Liebstein, Reference Ravid and Egoz-Liebstein2012; Ruffman et al., Reference Ruffman, Slade and Crowe2002; Shiota & Keltner, Reference Shiota and Kelther2005). Mental words have abstract and complex meanings based on the comprehension of mental states and are challenging to learn from context (Ravid & Egoz-Liebstein, Reference Ravid and Egoz-Liebstein2012). People report more difficulty in representing emotion words in their imagination compared to abstract words. Emotion words are also more challenging to recall than concrete words or abstract words (Altarriba & Basnight-Brown, Reference Altarriba and Basnight-Brown2010; Altarriba et al., Reference Altarriba, Bauer and Benvenuto1999; Pavlenko, Reference Pavlenko2008). In addition, written emotion words were detected very rapidly and automatically about 250 ms after exposure, unlike neutral words that took 500 ms (Kissler et al., Reference Kissler, Herbert, Peyk and Junghofer2007).
The second reason to include emotion words in vocabulary assessment is research that suggests that they have a different course of language acquisition rate compared to nonemotion words. Mental verbs have been consistently acquired relatively late compared to nonmental verbs (Ravid & Egoz-Liebstein, Reference Ravid and Egoz-Liebstein2012). Within mental words, it seems that emotion words are acquired especially late. Studies have found that children aged 5–8 were more likely to use cognitive verbs (such as thinking and knowing) than emotion verbs (such as loving and feeling sad). The later acquisition of emotion words may be related to the later development of abstraction abilities (Egoz-Liebstein, Reference Egoz-Liebstein2009; Ravid & Egoz-Liebstein, Reference Ravid and Egoz-Liebstein2012). We could not identify studies examining the proficiency of older children with emotion words.
These differences reinforce the notion that emotion words might function independently than other types of words and provide the theoretical basis for assessing emotion words as a specific category of vocabulary and examining their contribution to RC.
The current study
In the current study, we developed a novel vocabulary assessment (the Herut test) that attempts to address the gaps identified in the literature. The assessment we developed is curriculum-based and includes different categories of emotions according to Scherer’s model (Reference Scherer2005).
The main goal of the current study was to validate this newly developed vocabulary measure (Herut) and examine the contribution of emotion words to RC. To achieve an equal balance of 3 emotions from each of the 8 slices in the Scherer’s model, 24 emotion words were selected from the curriculum. According to Ekman’s model (Reference Ekman and Cole1972), another 6 basic emotions were added, for a total of 30 emotion words. These items were split between two versions of the test, one of which was used in the present study. Because the current study was focused specifically on emotion words in the context of RC abilities and based on the theoretical framework presented above that reflects different types of emotion words, the Herut measure we designed included more emotion than nonemotion words.
After developing the measure, we next examined its psychometric characteristics. The first aim will be to validate the Herut measure. To examine convergent validity, Pearson correlations were calculated between the Herut measure and the Elul vocabulary measure. Elul and Herut are both receptive measures of vocabulary, but Herut is curriculum-based, whereas Elul is based on increasing level of difficulty and is age normed. In addition, Herut requires children to match words to definitions, whereas Elul requires matching words to visual representations. Based on the research conducted by Beck et al. (Reference Beck, Kumschick, Eid and Klann-Delius2012), which found a correlation between emotional abilities and linguistic abilities such as vocabulary, we hypothesized that there would be positive correlations between the two vocabulary measures. We also examined the correlation between the two subscales within the Herut measure, namely emotion and nonemotion words.
In the next step, we assessed the construct validity of Herut. To this end, we calculated Pearson correlations between the Herut and reading fluency and RC measures. Based on Beck et al. (Reference Beck, Kumschick, Eid and Klann-Delius2012), we hypothesized that there would be positive correlations between the two subscales of the Herut measure (emotion and nonemotion words) and reading fluency and RC measures.
The second question addressed in the current study is the contribution of vocabulary measures and words types (emotion and nonemotion words) to predicting RC by using a hierarchical regression analysis. To our knowledge, no studies have evaluated the differences between the contributions of different vocabulary measures to RC, so there was no basis for an apriori hypothesis.
Development of the Herut vocabulary measure
The Herut test is a unique tool aimed to assess different types of vocabulary in Hebrew-speaking upper elementary school aged children (grades 4 and 5). The purpose of the assessment is to evaluate knowledge of emotion and nonemotion words, based on the curriculum.
The measure was developed in several steps: As a first step, texts were selected from fourth-grade curriculum materials, including textbooks, assessment tasks, and teaching units published by the Ministry of Education and independent publishers approved by the Ministry. The selected texts were either narrative or expository and were written by various authors. The sample included texts from different domains: RC and literature (15,000 words), science (8,000 words), geography (5,000 words), and social skills (5,000 words). All texts together included 33,441 words (about half of them narrative and half expository). Based on accepted pedagogical principles (Beck et al., Reference Beck, McKeown and Kucan2013; Graves et al., Reference Graves, Baumann, Blachowicz, Manyak, Bates, Cieply, Davis and Von Gunten2013), candidate words for the vocabulary assessment were identified first by the first author and then by two trained research assistants. We selected second-tier words (such as “embarrassment,” “inspiration”), which are not basic or highly frequent words on the one hand, but are also not rare words specialized for a specific topic (Beck et al., Reference Beck, McKeown and Kucan2013). Of the selected candidate words, some appeared explicitly in the text, and some were imported words (words that did not appear in the actual text but were deemed to be important for supporting understanding of the text, Graves et al., Reference Graves, Baumann, Blachowicz, Manyak, Bates, Cieply, Davis and Von Gunten2013). For example, in a text that describes several people working together for a long time to fix something, the words “collaboration” and “goal” are necessary for understanding the main idea in the text even though the words themselves did not appear explicitly.
Based on these characteristics, 2,057 words (6%) were selected from the text corpus. The chosen words were categorized into two groups:
Emotion words: Words that describe emotions directly (e.g., expectation, loneliness, stress). (Pavlenko, Reference Pavlenko2008).
Nonemotion words: This category includes all other words that do not describe emotions. These words could be general words (e.g., collaboration, feature, advantage) and could also be cognitive words which reflect thought processes but not emotions (e.g., curiosity, definition, idea). Our choice in this classification stemmed from our specific focus on emotion words. Thus, words categorized as nonemotion words did not convey emotional information, but were of neutral, general content.
From the initially identified words, 397 words (or 19%) appeared in more than one text. Of these, 85% were nonemotion words, and 15% were emotion words. These 397 words were then rated by 40 regional language instructors who are employed by the Ministry of Education, and who supervise and provide training for elementary school language arts teachers. The regional language instructors were asked to rank the words’ importance for teaching upper elementary school students, on a scale of 1 (not at all) to 5 (very much). For low-ranked words, instructors were asked: Is the word too simple or complex? Is the word uncommon or not important? The questionnaire had three versions, each including about 130 words. In total, each version of the questionnaire was rated by 13 language instructors.
Following these steps, each word was ranked using six parameters: explicit frequency (number of appearances), frequency of words that did not appear in the actual text but were deemed to be important for supporting understanding of the text (imported words), the number of resource types in which the words appeared (curriculum materials, assessment tasks, teaching units), the number of content subjects in which the word appeared (language arts, literature, science, geography, and social skills), the number of text types (narrative/expository) in which the word appeared, and the word’s average ranking by the language instructors. The average of each parameter was calculated across all words, and then each word that was above average received 1 point for the relevant parameter. Next, the scores were summed across these parameters for all words. The maximum score was 6 and the minimum was 0, with a mean of 2.72. Finally, the 100 words with the highest score were selected (scores of 4–6), 85 nonemotion words and 15 emotion words.
In the final stage, the selected emotion words were examined to determine whether they express the eight slices of emotion words based on the Semantic Space for Emotions model (Scherer, Reference Scherer2005). The emotion words identified in the previous steps were indeed represented in the model, but the list did not include equal representation for all eight slices. Therefore, additional words from the model were added to the test list. To do this, we identified six additional emotion words from the corpus-based list, which were not included following the initial selection process described above. These words received an average rating of 3 points. Finally, we added three additional words to fully reflect the categories of Scherer’s (Reference Scherer2005) model, which were not included in the corpus-based list. We selected words that were deemed appropriate for the target population in terms of content and frequency (e.g., words such as conscientious and gloomy were not included). In the final step, six basic emotion words were added based on Ekman’s model (Reference Ekman and Cole1972) (sadness, fear, disgust, joy, anger, and surprise) to avoid an overly complex level of difficulty. Thus, in order to remain faithful to Scherer’s theory (Scherer, Reference Scherer2005) and to avoid a high level of difficulty, the final test included more emotion words than nonemotion words. The nonemotion words included in Herut were also selected from the list described above. We made sure to choose both words that reflect a cognitive state, such as the words comparison and opinion, and words that are not cognitive, such as the words advantage and sharing. A two-step process was taken in order to select the words. First words were rated by three independent judges and 13 language experts on several categories listed below. Then four independent judges choose words there were high in frequency on the listed parameters. The parameters were as follows:
The word appeared in a higher average frequency at the explicit and imported level (7.28 vs. 5.58).
The word appeared in a higher average of content subjects (2.76 vs. 2.62). The word appeared in a higher average of resource types (curriculum materials, assessment tasks, teaching units) (3.09 vs. 2.87) and a higher average ranking by the language instructors (4.49 vs. 4.36). No difference was found in the average of text types (narrative/expository) since all the selected words appeared in both types of texts.
In order to neutralize the effect of the frequency of words in language on the findings of the analysis, we examined the frequency of the word types: emotion words versus nonemotion words. First, the frequency of words from students’ study materials in different disciplines was examined, as well as the frequency of words from a broader corpus, based on a database for word frequency in Hebrew (Frost & Plaut, Reference Frost and Plaut2005). Emotion and nonemotion words were well matched on frequency in the language and in the text corpus collected for this study, as well as on length in letters (see Table 1). In the t test for independent samples, no significant differences were found between the frequency of nonemotion words and the emotion words (p > .05).
Table 1. Frequency of words in the adult corpus and in study materials
From the final word list, two parallel versions (with different words) of the Herut measure were developed, but in the present study, data from only one of the Herut versions are analyzed. Each version contains 15 emotion words and 10 nonemotion words.
The Herut assessment had two parts. In the first part, students were presented with a target word and were asked to identify the correct definition from among four options. In the second part, students were presented with a definition and were asked to identify the correct word from among four options. The test was scored on accuracy – one point was given for each correct response. An accuracy percentage measure was calculated based on the number of correct items. Each student received a score for the entire task, and in addition a score for each scale separately – emotion, and nonemotion words.
In a sample of 1,333 Hebrew-speaking students, Cronbach’s alpha for the entire measure (23 items) was .809, after two (nonemotion) items were removed due to low reliability. Cronbach’s alpha of the emotion words (15 items) was .732 and of the nonemotion words (8 items) was .623.
Method
Participants
The sample in the present study was taken from a broad study (Katzir et al., Reference Katzir, Lipka, Prior and Shany2019), which was conducted over 3 years, with a sample of 114 teachers and 2,352 students from 33 schools. School administrators were sent explanatory letters about the study, and only schools that expressed consent participated in the study. Letters of consent approved by Office of the Chief Scientist in the Ministry of Education were sent to students’ parents. Only students whose parents signed a consent form, and who themselves gave consent, participated in data collection. Data were collected in school by trained research assistants who were studying for advanced degrees in education.
The sample in the present study was derived from the third year of the wider study. The research participants were 1,333 Hebrew-speaking students from 21 elementary schools (668 males and 665 females), 844 in fourth grade (63%), and 489 in fifth grade (37%). Eighty-five percent of participants were born in Israel. Participants varied in socioeconomic and sociodemographic characteristics: 236 students (18%) attended schools classified by the Ministry of Education as being of high socioeconomic status, 805 students (60%) attended schools classified as being of medium socioeconomic status, and 292 students (22%) attended schools classified as being of low socioeconomic status.
Measures
All the tools can be found at this link: https://osf.io/2XTRP/
Vocabulary
The Herut measure (Sabag-Shushan, Yosefi, Katzir & Prior). Described above.
The Elul Task (Shany, Prior & Blicher, Reference Shany, Prior and Blicher2017). A Hebrew vocabulary measure is based on a subset of items from the Elul Measure (Shatil et al., Reference Shatil, Nevo and Breznitz2007). The task includes 35 words organized by frequency (high to low) in the Hebrew language and does not include emotion or cognitive words. For each target word, participants were required to choose the picture that most accurately matches the meaning of the target word from among four options. An accuracy percentage measure was calculated based on the number of correct items. Cronbach’s alpha was .764.
Reading comprehension
Literal and inferential meaning (Shany & Blicher, Reference Shany and Blicher2017). Students read two texts, one narrative and one expository (416 words on average per text). After reading, students answered comprehension questions on two levels: 11 multiple choice questions that ask for explicit information in the text (simple comprehension) and 7 multiple choice questions examining inferences and integration with out-of-text information, including the student’s prior knowledge or information from another text (complex comprehension). The narrative text included 10 questions (α = .717) and the expository text 8 questions (α = .631). Cronbach’s alpha for the entire task (18 items) was .793.
Tamar Measure (Sabag-Shushan & Katzir, Reference Sabag-Shushan and Katzir2018). This is a RC measure to assess simple and complex comprehension abilities. Students read four short texts (two narrative and two expository, 145 words on average per text). Following each text, students answered five multiple choice questions: one question that evaluated explicit information (simple comprehension), three questions that assessed different types of mental inferences (inferences about the emotions of the characters in the text, the ability to evaluate the character’s thoughts, and finally about the character’s goal planning) and one question that evaluates the ability to identify a main idea in the text. The questions that assess inferences and main ideas are classified as complex comprehension. An accuracy percentage measure was calculated based on the number of correct items.
In order to illustrate the distinction between the types of questions, we present an example narrative text:
“A woman was shopping at the supermarket and because the weather was hot, she put her jacket in the shopping cart. After paying for the products, she loaded them into her car and then discovered that some additional products were hidden under the jacket. The woman quickly returned to the supermarket to pay for the missing products. The cashier told her: “Now I understand why the women behind you were whispering”. The woman asked: “Where are these women?” The cashier answered that they had just left for the parking lot. The woman quickly rushed to the parking lot.”
To assess emotional inference, we asked: What do you think the woman felt when she discovered the women were whispering about her? To assess the character’s thoughts, we asked what the cashier understood when the woman returned to the supermarket. And to assess goal and planning inference, we asked why the woman rushed to the parking lot. Or in other words, this question evaluates the purpose of this behavior.
Similarly, in expository texts, we applied mental inference questions. For example, one text described the life of Alfred Nobel and his invention of dynamite, which was originally invented for the purpose of blasting rocks but was soon used for military purposes. The Nobel Prize Foundation was his way of encouraging people’s contribution to humanity and to atone for the damage his invention led to. After reading the text, the students were asked what Alfred Nobel thought of the idea of using his invention for military purposes (inference about the character’s thoughts), Why Alfred Nobel founded the Prize Fund (goal and planning inference), How did Alfred Nobel feel about his idea of founding the Nobel Prize Fund (emotional inference)?
Cronbach’s alpha for the entire measure (20 items) was .833. Cronbach’s alpha for the narrative texts (10 items) was .738, and for the expository texts, it was .689. The emotion and nonemotion words examined in the Herut vocabulary measure did not appear in the texts so there was no overlap between the words in the vocabulary measure and in the RC measure. Since this is a novel RC measure, and it evaluates, among other things, emotional inferences, we examined its correlations with the previously established Literal and Inferential meaning test (Shany & Blicher, Reference Shany and Blicher2017) and found them to be positively correlated (r = .695, p < .001).
Word reading fluency
The Hebrew version of the TOWRE reading fluency measure (Torgrsen et al., Reference Torgrsen, Wagner and Rashotte1999), adapted by Katzir and her colleagues (Reference Katzir, Schiff and Kim2012), was administered individually to each student. This measure assesses decoding abilities that is, accuracy and reading fluency. Participants are asked to read aloud a list of Hebrew words, presented in the nonpointed script, as fast as they could in 45 s. The words were ordered by increasing the level of difficulty. The score was the total number of words read accurately in 45 s. Cronbach’s alpha was .769.
Procedure
Tasks were administered during two classroom sessions and one short session of individual testing. In each session, the students performed a RC task and a vocabulary task. In the individual session, the students completed the fluency task. All sessions were conducted within 3 weeks at most.
Results
Herut characteristics and performance
The first goal in this study was to validate the Herut test. We examined the normality of the distribution of scores, as well as Skewness and Kurtosis (Table 2).
Table 2. Descriptive statistics: range, mean, standard deviations, skewness, and kurtosis measures
Note. RC-simple and complex comprehension abilities, in narrative and expository text.
Since more than one session was required, there were students who did not complete all the required assignments, so there are missing data. 1,333 reflects students had at least one data measure. 1,035 students completed the full battery.
Next, children’s accuracy scores for the two types of vocabulary in the Herut test were compared. A paired-samples t test demonstrated that children were more accurate for emotion vocabulary (M = 72.92, SD = 23.64) than for nonemotion vocabulary (M = 56.94, SD = 24.93; t (1,211) = 25.66, p < .001).
Validating the Herut test
Convergent validity
Expression of the convergent validity is through high correlations of two different measures (Herut and Elul), which measure a common feature, here vocabulary knowledge. The Herut measure and the Elul vocabulary measure were significantly positively correlated (r = .592, p < .001). Positive correlations were also found between the Elul measure and each of the Herut subsets: emotion words (r =.553, p < .001) and nonemotion words (r =.550, p < .001). There was also a significant positive correlation between nonemotion words and emotion words within the Herut measure (r = .602, p < .001).
Construct validity
Expression of the construct validity is through correlation with related measures, such as reading fluency and RC indices. Pearson correlations between the Herut measure, reading fluency, and RC were calculated N = 1,204). Significant positive correlations were found between all research variables (all p < .001) (Table 3).
Table 3. Correlations between vocabulary, fluency, and reading comprehension (N = 1,204)
*p < .001.
Another expression of construct validity is through the assessment of differences between groups. Previous research has demonstrated that increasing age is associated with higher vocabulary knowledge (Seigneuric & Ehrlich, Reference Seigneuric and Ehrlich2005). Thus, differences between fourth- and fifth-grade students in vocabulary knowledge on the Herut measure are another measure of test validity. An independent samples t test showed significant differences between grade levels in the Herut vocabulary measure (t (1,212) = −7.49, p <.01), such that fifth graders showed higher accuracy (M = 69.18, SD = 18.39) than fourth graders (M = 60.61, SD = 19.63).
The contribution of different vocabulary measures to RC
Since the two RC tests (the Tamar test and the Literal and inferential meaning test) assess common aspects of RC ability, and because there is a high correlation between the tests (.695**), we calculated a combined RC measure based on both tests. In the combined measure, students read three texts of each type (narrative and informative) and answered 15 questions from the simple understanding (α = .788) as well as 23 questions from the complex understanding version (α = .825). Cronbach’s alpha for the narrative tasks (20 items) was .831, and .789 for the informative tasks (18 items). Cronbach’s alpha for the combined comprehension measure (38 items) was .894.
In order to examine the contribution of the Herut measure to predicting RC (the combined RC measure), a hierarchical regression analysis was performed. According to the simple view of reading model (SVR; Gough & Tunmer, Reference Gough and Tunmer1986), reading fluency was introduced into the model first, and the Herut measure was introduced into the model in the second step. The final model explained 44.2% of the variance in RC (F(2,1133 = 448.67, p < .001), and both variables contributed significantly to the model. Reading fluency explained 15.3% of the variance in RC, and the Herut measure explained an additional 28.9% of the variance (see Table 4 and Figure 1).
Table 4. The contribution of the Herut and Elul measures to reading comprehension (combined reading comprehension) (N = 1,133)
*p < .001.
Figure 1. The contribution of the Herut and Elul measure to variability in reading comprehension (combined reading comprehension).
Next, the same regression was conducted with the Elul vocabulary measure. The parallel model examining the contribution of the Elul test to predicting RC showed that both variables, reading fluency and Elul measure, contributed significantly to the model and jointly explained 32% of variance in RC (F(2,1076) = 266.894, p < .001). Reading fluency contributed 15.1% to RC, and the Elul measure contributed 16.9% to RC (Table 4 and Figure 1). Thus, across combined RC measure (Tamar and Literal and Inferential Meaning), the Herut vocabulary measure explained almost double the percent of variance in RC than did the Elul vocabulary measure.
The contribution of different types of words (Emotion vs. nonemotion) to RC
In order to examine the contribution of the two types of vocabulary words in the Herut test (emotion and nonemotion words) to predicting RC, two hierarchical regression analyses were performed. According to the simple view of reading model, reading fluency was entered in the first step of the regression. Because there is no previous research regarding the unique contribution of emotional vocabulary to RC, two regression models were performed; once when the nonemotion words were entered in the second step, followed by the emotion words, and once when the emotion words were entered in the second step, followed by the nonemotion words.
The findings showed that in both models, all variables, reading fluency, nonemotion words, and emotion words significantly explained 44.8% of the variance in RC (F(3,1130) = 305.983, p < .001). When the nonemotion words were entered in the second step of the model, they significantly explained 21.2% of the variance in RC. The emotion words, inserted in the third step, significantly added another 8.4% to the explained variance, as can be seen in Table 5 and in Figure 2. In the alternative model, when the emotion vocabulary was entered in the second step, it explained 23.9%, and nonemotion vocabulary was entered on the third step and significantly added 5.7% to the variance in RC, as can be seen in Figure 2 and Table 5.
Table 5. The contribution of different types of words (Emotion vs. nonemotion) to reading comprehension in combined reading comprehension (N = 1,133)
*p < .001.
Figure 2. The contribution of different types of words to reading comprehension (combined reading comprehension).
Analyses clearly demonstrate that nonemotion words and emotion vocabulary are indeed separable constructs and support the categorization of word types put forth in this research. This is because across all regression models, both types of vocabulary knowledge made significant independent contributions to RC. Further, emotion vocabulary makes a slightly greater contribution to explaining variance in RC than does nonemotion vocabulary in integrated RC measure (the Herut and Literal and Inferential measure).
Discussion
Validation of Herut measure
The current study aimed to validate the Herut measure, which evaluates the vocabulary knowledge of fourth- and fifth-grade Hebrew-speaking students. For this purpose, we examined convergent and construct validity. The findings confirm the validity of the Herut measure and thus support the notion regarding different types of vocabulary knowledge emotion and nonemotion words each add unique variance to the prediction of RC. A vocabulary measure that includes both types of words in this study was found to be more valid and useful for predicting RC. In addition, we suggest that assessment on a combined vocabulary measure that includes emotion words will provide a more nuanced approach for developing tailored intervention based on word type (Katzir, 2022, Reference Katzirin press).
The current findings also illustrate the strong connection between language and emotion in RC, corroborating and extending Beck’s previous findings (Beck et al., Reference Beck, Kumschick, Eid and Klann-Delius2012). In their study, strong positive correlations were found between linguistic measures, including receptive vocabulary, and emotional abilities among 7- to 9-year-old children. A plausible explanation of these results is that language promotes emotional ability by a process of lexical conceptualization. These findings also reinforce the role of emotional components in RC (Katzir & Primor, Reference Katzirin press).
The contribution of different vocabulary measures to RC
After assessing the correlations between the measures and word types, we assessed the contribution of the different vocabulary measures to RC. The findings showed that reading fluency, as well as vocabulary measures (Herut and Elul), together significantly contributed to RC, across two different measures. The overall percent of variability explained in this study is relatively low compared to what is typically observed in predicting RC in other languages (Primor et al., Reference Primor, Pierce and Katzir2011). However, it is similar to previous studies in the Hebrew language. For example, a study that evaluated the contribution of different cognitive and linguistic variables (vocabulary, morphological processing, word reading fluency) to RC among 190 Hebrew-speaking readers in fourth grade, only 25–34% of variance was explained (Primor et al., Reference Primor, Pierce and Katzir2011)
Importantly, because we included reading fluency in the regression models, the analysis controls for its covariability with vocabulary. Nonetheless, the unique contribution of the Herut measure, which includes emotion words and curriculum-based vocabulary, was found to be higher than the unique contribution of the Elul measure across both RC measures.
The greater contribution of the Herut measure compared to the Elul measure may be due to a number of reasons. First, the items included in the Herut measure were selected from students’ study materials according to the age group. It seems reasonable to assume that children’s mastery of vocabulary from their curriculum may contribute to RC success more than mastery of rare, lower frequency vocabulary. Beyond the fact that the words in the Herut measure were curriculum-based, they also reflected a wider range of words, including emotion words.
Secondly, most of the words (82%) in the Herut measure are categorized as mental words (emotion or cognitive), which are abstract words describing internal processes and situations (Hall & Nagy, Reference Hall, Nagy, Gopnik and Gopnik1986; Ravid & Egoz-Liebstein, Reference Ravid and Egoz-Liebstein2012) and which are acquired at later stages of development (Ravid & Egoz-Liebstein, Reference Ravid and Egoz-Liebstein2012). Thus, the mastery of mental and abstract words, unlike concrete words, might be more strongly related to the ability to achieve RC in upper elementary school.
Another explanation for the greater contribution of the Herut measure may be related to the type of assignment: The Herut measure assesses vocabulary depth, while the Elul measure is an assessment focused on vocabulary breadth. Thus, a study by Ouellette (Reference Ouellette2006) demonstrated that for fourth-grade students, vocabulary depth (a definition and synonym task) explained unique variance in RC than vocabulary breadth (the PPVT). Thus, depth of vocabulary knowledge seems to play a stronger role in RC than does vocabulary breadth.
Finally, the greater contribution of the Herut measure may be related to the nature of the task – success in the Herut required being able to understand definitions, which in itself requires RC, whereas the Elul does not require this.
The contribution of vocabulary types to RC
The final aim of the study was to examine the contribution of different types of words from the Herut measure (emotion vs. nonemotion words) to predicting RC. The results showed a unique contribution of each type to explaining variance in RC. Further, the contribution of emotion words was slightly larger than that of nonemotion words. We offer several explanations for this finding.
First, accuracy rates were significantly higher for emotion words compared to nonemotion words. Students tend to use existing knowledge resources. They have more knowledge about emotion vocabulary and therefore this knowledge is a resource that they can use to support greater RC.
The second explanation is related to the nature of words and their importance. Emotion words convey information about inner states, attitudes, beliefs, social contexts, motives, values, and behaviors (Shiota & Kelther, Reference Shiota and Kelther2005), thus mediating the inner world of the individual and their environment (Hobbs & Gordon, Reference Hobbs, Gordon and Khurshid2011). During reading, the reader acquires information from the text, and at the same time collects evidence of mental information, such as the emotional states of characters and events. Sentences that describe characters’ mental states are fundamental for the interpretation and comprehension of the characters’ aims and actions (Graesser et al., Reference Graesser, Singer and Trabasso1994) and are central for constructing meaning (Miall, Reference Miall1988). Identifying and processing emotional states in the text contribute to creating coherence between items, understanding causality, as well as understanding the chain of events and creating a mental model (Graesser et al., Reference Graesser, Singer and Trabasso1994; León et al., Reference León, Dávalos, Escudero, Olmos, Morera and Froufe2015). Therefore, emotion words may serve as an anchor for making connections between details and events in the text, thus contributing to a better understanding of what is being read, especially in narrative texts.
Another explanation is related to word processing and availability. Emotion words, unlike neutral words, have a value (positive/negative) dimension and are associated with arousal and physiological responses (Hsu et al., Reference Hsu, Jacobs, Citron and Conrad2015; Pavlenko, Reference Pavlenko2008). Differences between emotion words and neutral words are also evident in information processes. For example, emotion words were found to be identified more quickly and recalled more accurately than neutral words (Kissler et al., Reference Kissler, Herbert, Winkler and Junghofer2009; Kissler et al., Reference Kissler, Herbert, Peyk and Junghofer2007). Furthermore, eye movement during reading emotion words differs from those observed during reading neutral words (Hinojosa et al., Reference Hinojosa, Méndez-Bértolo and Pozo2010; Scott et al., Reference Scott, O’Donnell and Sereno2012). It is quite possible that the physiological arousal, memory processes, identification, and processing of emotion words also contribute to RC processes.
Limitations and future directions
There are several limitations to this study. First, the Herut test includes more items classified as emotion words compared to nonemotional words. It is possible that this gap in favor of emotion words may have affected the results.
Second, the data were collected in a single cultural context (Israel), and thus it is currently unknown whether the distinction between emotion and nonemotion words characterizes other societies as well. Thus, it will be interesting to examine whether and to what extent emotional expression in different cultures contributes to RC.
In addition, the vocabulary measures used and compared in this study (Herut compared to Elul) differ along several dimensions. It is difficult to pinpoint the exact reason for the difference manifested in the higher contribution of the Herut measure. This may be due to the higher frequencies of the words in the Herut measure, and it may also be due to the differences in the task characteristics (definition vs. recognition).
Future avenues of research could include further research on possible differences across types of vocabulary knowledge, such as the distinction between positive and negative emotions. Finally, based on our findings, the contribution of intervention programs that include a rich variety of emotion words should be considered as a promising way of supporting children’s development and literacy acquisition.
Conclusions
The current study describes a meticulous procedure for formulating an innovative vocabulary evaluation measure, which includes emotion vocabulary alongside nonemotion words. The study confirms the validity of the measure based on a variety of rigorous analyses, demonstrates its effectiveness in predicting RC, and demonstrates the contribution of emotional vocabulary to RC.
Importantly, the results demonstrate that a measure that includes emotion words and is curriculum-based is more effective in predicting RC, and that emotion words make a unique significant contribution to RC. The study highlights the importance of incorporating emotion vocabulary in assessments as well as in instructional processes and emphasizes the contribution of different types of vocabulary to RC. Emotion words must be recognized as integral components of vocabulary assessment and intervention.
Replication package
Replication data and materials for this article can be found at https://osf.io/2XTRP/
Financial support
This work was supported by a grant from the Israel Department of Education awarded to Tami Katzir, Orly Lipka, Anat Prior, and Michal Shany. This work was also partially supported by a PhD fellowship to TSS from the Israel Scholarship Education Foundation (ISEF).
Conflict of interest
The authors declare that they have no competing interests related to this manuscript.
