A multimodal approach to polysemy: the senses of touch

Abstract

This study investigated whether speakers use multimodal information (speech and gesture) to differentiate the physical and emotional meanings of the polysemous verb touch. We analyzed 302 hand gestures that co-occurred with this perception verb. For each case, we annotated (1) the meaning of touch (physical vs. emotional), (2) the gesture referent speakers physically touched (other-touch vs. self-touch), (3) the personal pronoun following the verb and (4) if they used intensifiers and negation. There were three main findings. First, we have seen that when speakers express the physical meaning, they are likely to reach an external referent (other-touch), but when they imply the emotional meaning, they tend to touch their own body (self-touch). Second, the most frequent co-speech gesture (chest-touching gesture) was associated with the emotional meaning, uncovering the metaphor the heart is container for emotions. Third, this study showed that the physical meaning of touch usually coexists with a wide variety of personal pronouns and negation words; in contrast, the emotional meaning of touch occurs primarily with the pronoun me and it is usually modified by intensifiers. Thus, speakers use both speech and gesture to differentiate the meanings of the polysemous verb touch.

1. Introduction

Polysemy can be succinctly defined as the synchronic association of multiple related meanings with a single form (Sweetser, 1990). Polysemous words typically have a prototypical meaning (i.e., their most frequent semantic interpretation) to which other senses are related in a motivated way. The complex phenomenon of polysemy has been extensively researched from a variety of disciplines such as lexicography, cognitive semantics and psychology, among others (see Falkum & Vicente, 2015 for an overview of the main theories regarding polysemy). Regardless of the research focus, most authors agree that the fundamental characteristic of polysemous constructions is the natural and systematic relation of its multiple meanings. In other words, the main point of polysemy is not the multiplicity of meanings but the underlying mechanisms in charge of creating and connecting its different senses.

If the multiple readings of a particular lexical item exhibit no systematic semantic relation, that word is considered homonymous rather than polysemous. Thus, the main difference between homonymy and polysemy is the absence or presence of a systematic relation between meanings. For example, even though the word ball can be used to designate at least two different meanings, (1) a round object used in games and (2) a formal dance, it must be considered homonymous since these meanings are unrelated, that is, they show no type of connection, neither semantic nor etymological (i.e., each meaning originated from different word forms). In contrast, an example of a polysemous lexical item is the noun taste. It is mostly used with adjectives to describe the flavor of a food product (e.g., it is a basic cake, but it has a good taste). However, it can also be used to refer to someone’s discernment (e.g., he has bad taste in clothing). These senses are naturally related as they both denote likes and dislikes. In this case, the latter sense reflects the metaphorical mapping of an already existing meaning from the domain of perception to the domain of personal judgment.

One of the theories which has examined more closely the potential motivations behind the semantic extensions of polysemous lexical items is cognitive semantics. This discipline has paid a great deal of attention to the role of metaphor as a connecting mechanism between senses. Many of the senses of a polysemous word, as is the case of the discernment sense of taste, are often considered figurative interpretations of the ‘prototypical’ sense of a lexical item. These ‘non-prototypical’ related figurative meanings (e.g., metaphorical, metonymic, pejorative) are generally conceived as semantic extensions of the polysemous lexical item.

Thus, metaphor is one of the key notions behind the motivation of semantic extensions. For a substantial number of scholars in the field, metaphor is the mechanism that allows us to ground abstract concepts on embodied experience. This means that the way we perceive our surroundings shapes our conceptual system (Lakoff & Johnson, 1999). In many cases, there is a bodily based motivation behind the senses of a polysemous item which structures its meanings, and the most frequent structuring mechanisms are metaphor and metonymy (Ibarretxe-Antuñano, 1999). An illustration of this would be the sentence I see your point, where we use the verb see metaphorically to convey comprehension instead of perception (understanding is seeing). This case appears to be motivated by the fact that most cultures conceive sight as the most reliable source of information that eventually leads us to the understanding of events. Therefore, this example shows how metaphor produces nonarbitrary associations between embodied experience and semantic extensions (Lakoff, 1987).

One of the areas in which the role of metaphor in semantic extension has been more thoroughly researched is the vocabulary of perception. The main reason for this is that, even though the vocabulary of perception is used to describe how we perceive the world through our senses, it is extremely common to find perception words used metaphorically. In fact, Sweetser (1990), who pioneered research on perception verbs by diachronically exploring their multiple meanings in English, concluded that the systematic connections between our vocabularies of mind and body are essentially metaphorical in nature. For example, when we say I smell trouble there is no physical act of perception; instead, we use this verb to express our intuition that something bad is going to happen. Multiple works have focused on the analysis of perception words (especially, verbs) and their semantic extensions in many languages such as English (Kövecses, 2019; Selmistraitis & Boikova, 2020; Sweetser, 1990; Winter, 2019), Spanish (Ibarretxe-Antuñano, 1999; Jansegers et al., 2015), Italian (Monachini & Roventini, 1994) and other languages (Burenhult & Majid, 2011; Evans & Wilkins, 2000; Krishna et al., 2022; Majid & Levinson, 2011; van Putten, 2020), amply showing the highly polysemous nature of this conceptual domain.

One of the fundamental works on the polysemy of perception verbs is the cross-linguistic analysis carried out by Ibarretxe-Antuñano (1999, 2002, 2006, 2019). In her work, she provides a cognitive semantic account of polysemy in English, Basque and Spanish perception verbs. The author describes the extended meanings that each of the five senses (i.e., vision, audition, touching, tasting and smelling) has in these three languages and summarizes the rich structure of the perception metaphors underlying them. For example, she explains that, besides its basic, physical sense of coming into contact with something, touch can be understood as ‘to affect’ both physically (my boots were here, who touched them?) and metaphorically (her story touched me deeply). Whereas the first sentence could be rephrased as who changed the position of my boots?, in the second one the verb touch specifies that what is affected is the emotional side of the speaker. Moreover, Ibarretxe-Antuñano (1999, 2006) shows how polysemy can sometimes be activated by the combination of a lexical item with other elements of the sentence which are necessary to build up that meaning. In a sentence such as ‘John hardly touched the food’ (Ibarretxe-Antuñano, 1999, p. 69), the property of briefness is being highlighted by the adverb hardly (which here refers to a ‘small degree of something’). Without this adverb, however, the extended meaning of ‘partake’ disappears completely and the resulting sentence (John touched the food) comes to trigger the physical meaning of touch. In spite of this, we may also encounter utterances, such as X touched Y, without any additional linguistic elements activating the correct meaning of the polysemous lexical item. When the competition between the multiple meanings of a lexical item cannot be resolved, people cannot determine which of the distinct potential representations of that word should be activated. Polysemous words may lead to this situation which is known as language ambiguity. Thus, our ability to process ambiguous constructions is also of relevance when studying polysemy.

Taking this into consideration, language ambiguity has been of great interest for polysemy studies since the phenomenon of polysemy is believed to be one of the major sources of ambiguity (Nerlich & Clarke, 2003; Norrick, 1981; Taylor, 2006). The issue of ambiguity often emerges because the different interpretations of polysemous words are manifestations of the same core meaning; however, in most polysemous expressions there is one sense that seems more appropriate in the given context (Klepousniotou, 2007). We tend to consider that a polysemous word is ambiguous when we find it hard to choose between two or more senses that can be potentially associated with that specific form. Although people normally determine the meaning of ambiguous items by appealing to context (e.g., previous speakers’ utterances, common sociocultural background, visual input), it is not always possible to access those additional pieces of information. A classic example that illustrates this situation is the sentence ‘John touched Mary’ – X touched Y – (Ibarretxe-Antuñano, 1999, p. 124). With no additional context, at least two of the most frequent senses of the verb touch could be considered: (1) establishing physical contact, or (2) affecting someone in an emotional manner. In short, when there is insufficient contextual information when uttering polysemous expressions, listeners cannot decide their most suitable meaning interpretation (Ibarretxe-Antuñano, 1999).

On the whole, previous works have mostly examined the phenomenon of polysemy from a linguistic perspective (Falkum & Vicente, 2015). However, the study of polysemy including ‘non-verbal’ communication modalities has not been sufficiently explored (Ibarretxe-Antuñano, 2019). Considering that meaning can be conveyed through multiple modalities such as gesture or voice pitch, it makes sense to extend the analysis of polysemy from the purely verbal level to a multimodal perspective. In contrast to the traditional pairings of specific word forms and discrete meanings, the notion of multimodality asserts that semantic distinctions generally arise from the interplay of multiple communication modalities which can be of a different nature (i.e., ‘verbal’ and ‘non-verbal’). Hence, introducing the factor of multimodality to study the polysemy of perception verbs could be a way of expanding our previous knowledge and improving our existing models of polysemy.

One of the areas in the study of polysemy that multimodality could contribute to is the disambiguation of meaning. As mentioned before, this area has mainly been addressed by looking at the linguistic elements accompanying polysemous constructions (Hirst, 1987). For this reason, redirecting the research focus toward distinct communication modalities and methods to study language ambiguity could prove to be especially useful, since it adds another layer of analysis to complement verbal information. So far, how (or indeed, whether) people disambiguate the meanings of polysemous words in spontaneous communication using multimodal cues has received little attention. There are, nonetheless, some works that have attempted to combine distinct modalities when studying language ambiguity. Holler and Beattie (2003) report that speakers do use gesture to clarify verbal ambiguity for the listener. They found that when speakers are asked to clarify the meaning of an ambiguous word used in the course of an extended discourse (in their work, homonyms), they are more likely to use a gesture in association with the problematic word; similar results were found by Holle and Gunter (2007). Similarly, when investigating the resolution of ambiguous statements related to neutral and moral preferences, Hinnell and Parrill (2020) showed that participants rely on gesture, when available, to understand the speaker’s ambiguous statements. Particularly, gesture appeared to be used by observers to understand the speaker’s opinion, since they were more likely to choose the idea accompanied by a gesture as the speaker’s preference, regardless of whether the linguistic cue expressed a neutral idea or a moral view that might be considered socially unacceptable. On the whole, these three studies sustain that, apart from listening to the oral message, interlocutors tend to look for meaning in the hand movements that accompany ambiguous speech (Maricchiolo et al., 2014).

Therefore, it appears that focusing on gesture together with speech could be beneficial for the analysis of polysemy since these modalities are semantically co-expressive and may transmit a different aspect of the same idea (Goldin-Meadow, 2003; McNeill, 2013). This means that gesture can express information either on its own or when co-occurring with oral production. Co-speech gestures (i.e., gestures co-occurring with speech) may express task-relevant information not conveyed in speech; for example, sometimes children make a gesture depicting width while saying ‘this cup is bigger’ (Goldin-Meadow & Alibali, 2013). Co-speech gestures have also become a key aspect in language, cognition and communication studies because they can convey both concrete and abstract ideas. We can point to the restroom while saying Ana is in the toilet, but we can also draw circles in the air while saying the song is on repeat. Besides, co-speech gestures may be especially useful when dealing with words that have literal and figurative meanings, such as polysemous words, since they can reveal some of our covert cognitive representations (cf. Alcaraz Carrión & Valenzuela, 2021; de la Fuente et al., 2014; Núñez & Sweetser, 2006). This is mainly because while gestures convey meaning ‘globally, relying on visual and mimetic imagery’, speech does it ‘discretely, relying on codified words and grammatical devices’ (Goldin-Meadow, 2003, p. 25). If someone said she is wearing a hat while moving his hand as though grasping the bill of a cap, this gesture might encourage listeners to infer that the hat referred to was particularly a baseball cap (Goldin-Meadow & Alibali, 2013). Therefore, in contrast to the traditional verbal approach, a multimodal approach focused on gesture could potentially extend our understanding of the mental processes behind polysemy.

Taking into account the aforementioned examples of polysemous perception verbs, the verb touch seems to be especially interesting, given the fact that it is the only perception verb closely tied to the domain of emotion (see Sweetser, 1990 for a more detailed explanation). In fact, several works (Ibarretxe-Antuñano, 2006; Jansegers et al., 2015; Lacey et al., 2012) have already explored how the semantic content of tactile expressions and its arguments contribute to the creation of meanings such as to affect, to deal with, and so on.

While previous studies aimed at introducing different modalities (through videos and images) in language disambiguation research, no work has paid attention to the role of specific gestures in the expression of polysemous constructions in spontaneous communication. Thus, the present article explores to which extent co-speech gestures can be used to disambiguate the meaning of polysemous words. Our hypothesis, thus, is that gesture, together with linguistic cues, could also play a relevant part in the phenomenon of polysemy.

This research adopts a multimodal perspective to explore two specific senses of the perception verb touch: its physical meaning and its emotional meaning. We address the following research question: do speakers use multimodal information (namely, speech and gesture) to differentiate the physical and emotional meanings of touch? With a view to discover whether gestures are used to disambiguate polysemous words, three specific objectives emerge: (1) to compare the co-speech gestures associated with each selected meaning of touch, (2) to determine the motivation behind potential frequent co-speech gestures associated with touch and (3) to describe the relation between the linguistic context and the physical and emotional meanings of touch. In this investigation, we evidence that speakers can manifest different semantic extensions of the same lexical item through verbal (speech) and non-verbal (gesture) modalities. Based on these findings, we conclude that both speech and gesture are indeed used to distinguish the meanings of polysemous words.

2. Methodology

2.1. Dataset and tools

The audiovisual and textual information was obtained from the NewsScape Library of TV News, a multimodal repository of television news hosted in the UCLA and CWRU libraries (which gathers data from March 2004 until today). This dataset is managed by The International Distributed Little Red Hen Lab™ which is a big data science consortium for research into multimodal communication. It contains more than 500,000 hours of television news programs as well as a 4-billion-word multilingual dataset formed by television subtitles of TV news. Speech and subtitles are synchronized through force-alignment tools, which enables researchers to access the exact moment in which specific linguistic expressions were uttered. The communicative exchanges found in NewsScape are set in different communicative situations like interviews, open discussions or debates.

We also employed the corpus software CQPWeb (Hardie, 2012) with a subset of the NewsScape dataset (Uhrig, 2018) in order to perform more accurate linguistic searches. Specifically, we employed the NewsScape 2016 sub-corpus (269,269,133 words) which gathers data and recordings from the year 2016.

2.2. Linguistic search

We collected all the cases in which the verb touch was followed by a personal pronoun or by the words someone or somebody in NewsScape 2016. In addition, we complemented this search in the 2016 sub-corpus with a search for the specific combination touched me in the whole NewsScape Library dataset (time span: March 1, 2004 to May 1, 2021). This second search was made to balance the number of emotional and physical senses and to obtain a representative sample of gestures used in an emotional context. The expression searched was selected from a preliminary analysis which revealed that touched me seemed to be the most frequently associated construction with the emotional sense of touch.

2.3. Analysis

2.3.1. Data filtering

Considering that the aforementioned corpora are built with TV recordings, the textual and visual data stored in them tend to be quite ‘noisy’ (e.g., hits may also include voice-over cases). After the searches were done, it was necessary to discard those cases which were not suitable for our goal. So, data were manually filtered in different steps to obtain an adequate final sample. First, we removed all the cases that presented a technical issue (e.g., audio/video not available) or presented a text-speech mismatch. We also removed tagging errors (i.e., when parts of speech were not correctly identified by the system). Second, we annotated whether the query item had a physical, an emotional or an ambiguous sense, as well as cases with other senses; here, only cases with physical and emotional meanings were kept. In order to check if the meaning of touch in the speakers’ oral message had been accurately classified, a second coder analyzed a subset of the sample obtained (N = 100 gestures, 33.33% of the final dataset); there was almost perfect agreement (κ = 0.94) between coders in identifying the meaning of touch in each case (Cohen, 1960). Third, we noted down whether the hands of the speakers were visible or not. In particular, we classified the cases into (1) barely or nonvisible gesture, (2) partly visible gesture and (3) completely visible gesture. Cases in which the speakers’ hands were not visible (category 1) were discarded, as it could not be determined whether the speaker performed a hand gesture or not. Considering that partly visible gestures can be reconstructed by the viewer, no distinction is made between categories 2 and 3 in the analysis. Fourth, we removed repeated clips (i.e., when the exact same utterance appeared in a duplicate video) to obtain a list of unique cases. Lastly, we annotated whether the speakers performed a co-speech hand gesture or not. The end product of this filtering process was the creation of two lists of cases: one containing the gestures made with a physical sense of touch and another one containing the gestures made with an emotional sense of touch.

2.3.2. Gesture analysis

For each of the cases that contained a co-speech gesture, we annotated the following features.

Firstly, we annotated whether the gesture was related to the semantics of the linguistic expression or not, as well as those gestures whose meaning was unclear. Specifically, we considered to be semantically related co-speech gestures those which were coherent with the speakers’ utterance by adding a parallel or complementary nuance (McNeill, 1992). An illustration of this category would be any case in which speakers point at an element mentioned in their verbal message, such as when somebody says do not touch it and simultaneously points to the object s/he is referring to. A gesture was considered unrelated when it was not directly associated with the speakers’ words, not depicting any semantic information related to the speech. For example, in this category, we include cases such as beats which are gestures that move along with the rhythm of speech (McNeill, 1992) and self-adaptors which are the incidental movements individuals make when manipulating their own body or clothes, such as when someone adjusts his glasses (Ekman & Friesen, 1969). There were eight problematic co-speech gestures that were labeled as unclear since they caused special controversy among the coders over whether they were related or unrelated cases. These cases were discarded from the dataset. A second coder annotated whether the co-speech gestures were related to the speaker’s utterance or not, showing a substantial agreement (κ = 0.71; Cohen, 1960) with the first coder. Unrelated co-speech gestures were thus discarded from the final dataset. Then, seven co-speech gestures initially labeled as related were also eliminated from the final dataset since they were ‘acted’ cases (e.g., Frankie Valli appears on KABC Jimmy Kimmel Live and while singing the words touched me, he extends his free arm forward and then changes his open palm hand shape into a fist shape); in other words, these gestures were part of a performance and not produced in a natural communicative situation.

Secondly, we annotated whether speakers touched the referent of the gesture (e.g., body parts, other people). With regards to what speakers physically touched, two categories were distinguished: other-touch (whether speakers touched an object or another person) and self-touch (whether speakers touched their own arm, chest, leg or belly). A second coder analyzed a subset of the data (N = 100 gestures, 33.33% of the final dataset) in terms of what speakers touched, showing a substantial agreement (κ = 0.73; Cohen, 1960) with the first coder.

2.3.3. Linguistic analysis

In relation to the linguistic elements, we examined the distribution of the pronouns following the verb touch and, taking into account that certain linguistic modifiers have been previously associated with emotional attributes in language, two additional verbal components were also annotated: the presence of intensifiers (i.e., linguistic markers which have ‘a heightening or lowering effect’ on a lexical unit; Quirk et al., 1972, p. 376) and the use of negation (i.e., syntactic process through which a word or particle can change a positive construction into a negative one; Quirk et al., 1972). In particular, the reason for the annotation of intensifiers was motivated by the fact that these linguistic markers have been shown to be very frequently associated with the expression of emotional experience (Argaman, 2009). Additionally, negation words were also taken into account because in a Sentiment Analysis study, Carrillo-de-Albornoz and Plaza (2013) showed that negation was the most influential phenomenon when predicting the emotional meaning of the text. Since one of the two meanings of touch examined in this article is directly linked to the expression of emotion, both intensifiers and negation words could be fundamental elements that help identify the sense of the verb.

These characteristics were noted down when they modified the verb and were fewer than five words away from the query item (i.e., when they were included within the range −5:+5). The list of intensifiers and negation signals found can be seen in Supplementary Table A.1. The final dataset is publicly available at https://osf.io/v8gp3/.

3. Results

3.1. Gesture frequency

The touch + personal pronouns search (in NewsScape 2016) returned 2571 matches (60.2%) and the touched me search (in the whole NewsScape dataset) gave 1698 matches (39.8%), amounting to a total of 4269 cases (100%). From the overall amount, we removed 417 cases that presented either a technical issue or a text-speech mismatch and, subsequently, we removed 59 tagging errors. When examining closely the meaning of touch in the remaining 3793 cases, we found 30 cases with an ambiguous sense and 214 cases with other senses (i.e., neither physical nor emotional), which were then excluded. After discarding the cases that were not associated with the physical or emotional meaning of touch, the remaining dataset (3549 cases) resulted in 2194 cases expressing the physical meaning of touch and 1355 cases expressing an emotional meaning. Then, 2562 cases (1528 physical and 1034 emotional) in which the speaker’s hands were not visible were eliminated. After this step, 175 physical cases and 71 emotional cases (246 in total) were removed as they were repeated. As this work focus was on co-speech gestures, 265 cases (178 physical and 87 emotional) where speakers did not make any hand gesture were not included in our gesture sample. In order to obtain the definitive sample of co-speech gestures, 8 unclear gestures, 159 unrelated gestures and 7 acted gestures were eliminated. After that, 186 physical (4.4%) and 116 emotional (2.7%) semantically related gestures were kept in the final dataset. Hence, the total number of cases was reduced from 4269 cases to 302 co-speech gestures which amounts to 7.1% of the cases obtained from both linguistic searches (for more data curation details see Supplementary Table A.2).

Taking into account only the cases associated with the physical or emotional meaning of touch (see Figure 1), a chi-square test of independence was performed to examine the relation between the presence and absence of gesture and the meanings of touch. The relation between these variables was not significant, χ ²(1, N = 567) = 1.9124, p = .167. Hence, there was no difference in the distribution of cases with related gestures and cases without gesture when co-occurring with the meanings of touch.

Figure 1. Distribution of cases associated with the physical or emotional meaning of touch.

3.2. Comparing the physical and emotional senses of touch

3.2.1. Gesture referent

In the case of co-speech gestures performed with the physical sense of touch, speakers touch the referent of the gesture in 21.5% of the cases (N = 40, out of 186). Concerning the co-speech gestures performed with the emotional sense of touch, speakers identify the referent by touching it 38.79% of the times (N = 45, out of 116). With a view to discovering whether there was a significant association in the final sample between touching the referent, or not, and the type of verb sense (i.e., physical and emotional), we calculated a chi-square test and the result, χ ²(1, N = 302) = 10.558, p = .001, indicates a significant association between both variables. This means that speakers are more likely to touch the referent of the gesture when the sense of the verb is emotional than when it is physical.

Additionally, when speakers physically touch the referent of the gesture, two categories can be distinguished: other-touch and self-touch (see Table 1). Self-touch is found with both verb meanings (16 times, 18.8%, in physical cases and 45 times, 52.9%, in emotional cases), but the former, other-touch, can only be found when the verb touch implies its literal meaning (24 times, 28.2%, in physical cases). Since there were no cases in some of the cells (Table 1) we could not use the chi-square test. This is why we employed Fisher’s exact test to determine if there was a significant association between the type of the meaning (i.e., physical and emotional) and the referent that was touched (i.e., other-touch and self-touch). There was a statistically significant association between the two variables (p < .001).

Table 1. Distribution of other-touch and self-touch in the final gesture dataset

Specifically, the group of gestures associated with the physical meaning shows that speakers tend to touch an object or another person (12 times, 14.1%, in each case, see Figure 2 for an example). Referring to the group of gestures associated with the emotional meaning, speakers mostly physically touch their chests (41 times, 48.2%, see Figure 3 for an example). The data collected suggests that the latter case frequently occurs together with the emotional meaning of the touch. Hereafter, this specific type of co-speech gesture is referred to as chest-touching gesture.

A chi-square test was carried out to examine the relation between the distribution of gestures where speakers touch their chest and gestures where speakers touch other things and the meanings of touch. The relation between these variables was significant, χ ²(1, N = 85) = 52.7032, p < .001. This result shows that speakers are more likely to physically touch their chest when the verb has an emotional meaning than when it has a physical meaning.

Concerning the cases of other-touch, they are noteworthy as they only seem to appear with the physical meaning of touch. The category other-touch can be easily distinguished as it involves physically touching an external element (in the following case, a different person). The other-touch example below (see Figure 2) is made while the speaker says:

1. 1) ‘he came over and touched me ’ (2010-12-07 KABC The View)

   In this case, Whoopi Goldberg talks about a little boy who was behaving badly and annoying her on a train trip. The most upsetting part of that situation was that the child touched her with his sticky hands. Simultaneously with this utterance, she touches her colleague’s arm with her left hand to illustrate and emphasize the fact that the naughty child dared to touch her.

   

   Figure 2. Example 1: other-touch gesture made with a physical meaning.

Regarding chest-touching gestures, the next case is a very clear example (see Figure 3):

1. 2) ‘initially it [the project] touched me ’ (2019-06-16 KCBS Entertainment Tonight)

   In this clip, Halle Berry explains how she got involved in a documentary of patients with AIDS. When talking about the phone call she received to participate in that project, she makes reference to a personal experience in order to explain why she accepted the offer immediately. Then, while saying the sentence above, she raises her left hand from a resting position and opens the palm to touch her chest area.

   

   Figure 3. Example 2: chest-touching gesture made with an emotional meaning.

3.2.2. Linguistic analysis

With regard to the linguistic utterances, we checked whether the meanings of touch could be associated with specific personal pronouns, the use of intensifiers and the use of negation words.

First, we counted the cases of each pronoun in our sample of gestures, confirming that the physical and emotional senses of touch might also be distinguished just by looking at the pronouns that appear after the verb (see Table 2). Taking into account only the touch + personal pronouns search in CQPweb (where pronouns were not specified), it can be seen that, when speakers express the physical meaning of touch, they mostly address an external referent (126 times, 79.8%) rather than refer to themselves (32 times, 20.3%). Concerning the cases associated with the emotional meaning of touch, speakers refer to themselves 11 times which amounts to 50% of the cases. That is, the frequency of the pronoun me increases when touch has an emotional meaning. The difference between the type of meaning (i.e., physical and emotional) and the type of pronoun that accompanied the linguistic expression (i.e., me and the other pronouns) is significant, χ ²(1, N = 180) = 9.3982, p = .002. Thus, the utterances conveying the emotional meaning of touch tend to be produced with the personal pronoun me.

Table 2. Distribution of pronouns that appear in the touch + personal pronouns search in CQPweb

As far as the use of intensifiers is concerned, it is infrequent with the physical sense of touch and pervasive with the emotional one (Table 3). Within the group of related gestures when touch had a physical meaning, only one out of 186 cases included an intensifier, a mere 0.5%. In contrast, speakers used intensifiers in 56 of the cases in which they did a related gesture when touch had an emotional meaning; this is 48.3% of the cases.

Table 3. Frequency of intensifiers and negation markers in physical and emotional meanings that co-occurred with a related gesture

Concerning negation words, they are commonly found when touch has a physical meaning but extremely rare in its emotional sense (Table 3). Negation words were present in 62 (33.3%) of the cases associated with the prototypical meaning of touch where speakers made a related gesture. In comparison, only two negation words (1.7%) were found in the list of cases linked to the emotional meaning of touch where speakers made a related hand gesture.

Taking into account the information above (see Table 4), a chi-square test was calculated to examine the relation between the distribution of linguistic markers (intensifiers and negation words) and the meanings of touch among all valid cases (i.e., non-repeated cases with visible hands). The result, χ ²(1, N = 278) = 239.1518, p < .001, shows a significant association between both variables. Thus, speakers are more likely to use negation words when the sense of touch is physical and they appear to use intensifiers when the sense of the verb is emotional.

Table 4. Frequency of intensifiers and negation markers in physical and emotional meanings among valid cases

After verifying that there was a statistically significant association between these two variables, we wondered if the meaning of touch could be predicted based on the use of intensifiers and negation. For this reason, a binary logistic regression (see Table 5) was calculated to predict whether the meaning was physical or emotional based on the use of intensifiers and negations among the valid cases. The effect of intensifiers and negation words on the meaning of touch was statistically significant (χ ²₂ = 329; p < .001; R ²_McF = 0. 347, R ²_N = 0.497). In particular, the presence of intensifiers – and absence of negation – was associated with expressions with an emotional meaning and the presence of negation – and absence of intensifiers – was associated with expressions with a physical meaning.

Table 5. Binary logistic regression model including Standard Estimate and 95% Confidence Interval for odds ratio

4. Discussion

4.1. Gesture frequency

Despite the fact that the distribution of the categories related gesture and no gesture among the physical and emotional meanings of touch was not statistically significant, these data also uncover some interesting points. In this work, the physical meaning and the emotional meaning of touch appear to be equally distributed in both categories (related gesture and no gesture). Thus, the difference in meaning (physical vs. emotional) does not influence the number of gestures speakers make. A potential explanation for this could be that speakers do not vary the frequency of gesture they make when they use either the physical meaning or the emotional meaning, instead they rather change the type of gesture (i.e., its formal features) they use with each verb sense. For example, when speakers express the emotional meaning of touch, they tend to physically touch their chest and they never reach out to external objects – as happens when speakers convey its physical meaning.

4.2. Gesture referent

Concerning the proportion of cases in which speakers physically touch the gesture referent, it varies depending on the meaning of touch. People touch the gesture referent more when the meaning of this perception verb is emotional (38.79%) than when it is physical (21.5%). Thus, speakers tend to visually indicate and specify the referent of their gesture – which might be crucial for determining the verb sense as it is typically the same as the ‘receiver of the action’ – when they are expressing the emotional meaning of touch. In addition to this, speakers tend to physically touch different types of gesture referents depending on the sense of touch expressed in speech. In fact, the gesture category other-touch (i.e., speakers touch an object or another person) was only found when speakers expressed the physical meaning of the verb. Even though the category self-touch (i.e., speakers touched their arm, chest, leg or belly) could be found with both meanings, it shows greater proportion in the emotional meaning. This is due to the high frequency of chest-touching gestures in emotional cases (48.2%), which is not only higher than the frequency of chest-touching gestures in physical cases (5.9%) but also higher than the overall frequency of self-touch in physical cases (18.8%). This would suggest that, while physical experience is connected to the external elements in our surroundings, emotions are conceived as an internal phenomenon developed and contained within our bodies.

This bond between chest-touching gestures and the emotional meaning of touch certainly is the most interesting result obtained from the gesture analysis. The relevance of this gesture goes beyond its frequent use since it seems to represent the Western conceptualization of hearts as the seat of emotions. Chest-touching gestures could be considered a way of depicting that something or somebody reached the speakers’ heart. The conceptual metaphor that seems to motivate the production of these gestures is the heart is a container for emotions (Kövecses, 1986; Lakoff & Johnson, 1980) which is at the same time included in the broader metaphor the body is a container for the emotions (Lakoff, 1987) that can be found in many languages (see Ibarretxe-Antuñano & Valenzuela, 2021 for an overview of this topic). For example, in English people can brim with excitement and in Spanish people can be full of rage (llenarse de rabia). In a few words, it seems that the emotional meaning of touch is expressed through the embodiment of emotions by means of chest-touching gestures.

Considering that mappings can sometimes vary depending on the speakers’ cultural background, this gesture is not only metaphoric but it is also likely to be culture-dependent. Due to the fact that the language explored in this research is English, it is coherent to find a conceptualization belonging to the Western world. However, as claimed in Ibarretxe-Antuñano and Valenzuela (2021), other cultures conceive the seat of emotions to be located in different organs such as the belly as is the case in Kuuk Thaayorre (Gaby, 2008), the liver in Indonesian (Sharifian et al., 2008), or even different organs as happens in Chinese, where anger is situated in the liver, fear in the kidneys, sadness in the lungs and joy in the heart (Yu, 2009). Taking this information into account, we cannot state that chest-touching gestures are associated with the expression of emotions in non-Western cultures since this path has not been researched yet. Hence, the gestures co-occurring with the narration of emotional events in languages such as the aforementioned needs to be explored in order to determine which are the gestures speakers use and whether these also imply the embodiment of emotions.

Another explanation for the propensity to use chest-touching gestures when the verb touch transmits an emotional meaning could be due to speakers seeking to reinforce this less frequent meaning of the verb by adding a more salient multimodal element. Considering the apparent lower frequency of use of the emotional meaning of touch, people appear to draw upon a more noticeable gesture – which does not only involve the hands but also a part of the speaker’s body – to facilitate the recognition of the ‘non-prototypical’ verb sense (i.e., emotional meaning). Furthermore, this explanation does not conflict with the potential use of co-speech gestures as elements helping to disambiguate the meaning of polysemous words. The results of this work suggest that gesture may be a useful supplementary level of analysis in the field of polysemy. Considering that, when there is no access to additional linguistic information, meaning disambiguation seems an impossible goal, the modality of gesture could be valuable to solve those cases occurring in situations with insufficient context or potential ambiguity.

In fact, Ibarretxe-Antuñano (2006) declared that the emotional meaning of touch is easy to identify when context is provided, but quite challenging to detect when it appears ‘isolated’ since it could have several interpretations. This situation was illustrated in Ibarretxe-Antuñano (2006) with the sentence John touched Mary (as discussed in the introduction). This is the reason why similar cases are called unpredictable cases of polysemy and their potential meanings argument-driven extensions (see Ibarretxe-Antuñano, 2006, for the detailed explanation of these concepts). Expressed differently, the meaning of this type of sentences cannot be predicted only by the arguments that the verb takes.

4.3. Pronoun frequency

The meanings of touch can also be distinguished by looking at the personal pronouns used with each type of meaning. Focusing on the touch + personal pronouns search, when speakers made a hand gesture related to the physical meaning of touch, they mostly addressed an external referent (79.8%). However, when the hand gestures were associated with the emotional meaning of the verb, speakers referred to themselves in half of the cases. Expressed differently, when speakers convey the physical meaning of touch, they make use of a wider variety of personal pronouns (where it is the most repeated one). The reason why physical cases occur with a diverse range of pronouns is simple: this meaning is more likely to include more elements (i.e., linguistic referents) because the description of external events depends on multiple environmental stimuli rather than on one, as happens when we refer to emotional experiences. Considering that the cases related to the emotional meaning of touch tend to be produced with the personal pronoun me, it appears that speakers do not usually use the verb touch to describe other people’s internal processes. The high proportion of the emotional meaning of touch being used to express our own emotions might occur due to the potential difficulty that entails reading people’s genuine feelings.

On the whole, this correlation between the personal pronoun accompanying touch and the meaning expressed appears to support the aforementioned gesture differences (self-touch vs. other-touch). This means that the wider the variety of pronouns used, the greater the diversity of gesture referents touched by speakers. The only issue concerning the distribution of personal pronouns in our sample of gestures is the fact that sometimes it might be challenging to determine their role; expressed differently, to decide whether chest-touching gestures were motivated by the emotional component of touch or originated as an ‘pronoun emphasizing strategy’. As chest-touching gestures used with the emotional meaning of touch are linked to the pronoun me, it could be argued that speakers touch their chest because they want to make the referent (who/what was touched) stand out. However, this is normally done through pointing gestures without the need of physically touching the referent – in this case, themselves (Cooperrider & Mesh, 2022). Nevertheless, another possibility is that these potential roles of chest-touching gestures might coexist together instead of excluding each other.

4.4. Linguistic markers

The linguistic analysis showed that there is an interaction between the meaning of touch activated and the type of linguistic marker used with the verb. When the co-speech gestures are associated with the physical meaning of touch, speakers often include a negation word while omitting intensifiers. In contrast, when the gestures are related to the emotional meaning of the verb, speakers do not use negation particles and they usually strengthen the meaning expressed by using intensifiers. This interaction can be almost conceived of an inverse relation since when a type of linguistic marker appears with the physical meaning, it is rarely manifested in the emotional meaning – and the other way round.

In general terms, linguistic modifiers interact with each other and can be used jointly since their appearance does not exclude the possibility of using more than one type of modifier within a construction (Carrillo-de-Albornoz & Plaza, 2013). In this research, there is also ‘the exception that proves the rule’; that is, one intensifier was found in physical cases and two negative particles in emotional cases. It is worth mentioning that the two negation words found in the emotional cases precede an intensifier and they are used before a comparative structure (e.g., the health care debate has not really touched me as much as it has Kelly). In this sentence, we find a great illustration of the use of negation as a diminisher of the strength of the polarity of the statement. Thus, the actual function of this negation is to emphasize that the speaker’s partner was deeply affected by the health care debate. In a few words, it seems that the negation word in this example is not directly used to ‘cancel out’ the emotional meaning implied by touch, but to be specific about the speakers’ level of affliction in comparison with his partner.

The association of the physical sense of touch with the use of negation might be due to the fact that it is felt as more ‘invasive’ in comparison to the emotional sense. In other words, speakers mostly use negation in order to warn people that they should not attempt to physically touch them or an external element. On the contrary, using negation words while conveying the emotional meaning of touch seems quite futile as individual emotions can be neither controlled nor ‘restricted’ by interlocutors themselves (Fernández Jaén, 2012, 2016; Jansegers, 2017). Also, physical contact could be potentially conceptualized as a binary feature [±physical contact], something either touches you or it does not.

As previously mentioned, the binary conceptualization of the physical meaning of touch does seem to not manifest in the emotional meaning of the verb. Actually, the fact that the emotional sense is associated with the use of intensifiers (e.g., very, really) supports the widely accepted idea that emotions are conceptualized along a gradable scale (Fontaine et al., 2013). Thus, it makes sense that intensifiers modify only emotional meaning (rather than physical meaning) because emotions are usually conceived as something which can have different degrees of intensity. Furthermore, in relation to the lack of intensifiers found when touch implies physical contact, it is quite unlikely that speakers would express the intensity of the touch (perception of pressure, temperature, etc.) through modifiers expressing degree rather than using manner adjuncts (e.g., he touched me softly so that he would not scare me). Concerning the degree of emotional involvement that someone might feel in a particular situation, it can also change from person to person and, as a result, it can be ‘graded’. For example, an event might ‘touch’ someone (i.e., they found the situation moving), but that same event might ‘really touch’ someone (i.e., they were deeply emotionally involved). In contrast, this specificity in the level of the action expressed cannot be done with the physical sense of touch.

Argaman (2009) stated that there is a connection between the intensity of emotions and the lexical modalities that speakers use in their expression of the emotional experience. She concluded that linguistic markers, such as intensifiers, are objective tools for determining people’s emotions since we normally give priority to a correct delivery of the content of the message rather than to a finely determined word choice, which is usually a conscious process. Our findings would confirm that speakers often use intensifiers when sharing their emotions, in our case, in oral productions. All things considered, the correlation between the use of intensifiers and the expression of emotional meaning could potentially apply generally and not only in relation the perception verb touch.

Analyzing the linguistic markers modifying polysemous verbs could be crucial when trying to determine the meaning conveyed. ‘In some cases, the polysemy is due both to one of the words of the sentence, in this case the verb, and to the meaning of the other arguments that the verb takes’ (Ibarretxe-Antuñano, 1999, p. 69). The connections found between linguistic modifiers and the meanings of touch could be much more than mere associations, as they also reveal specific ways in which speakers conceptualize these meanings. Thus, the results in this article would confirm Ibarretxe-Antuñano’s claim (1999, 2006) that polysemy is not always localized on a single word (touch), but also on other elements of the sentence (in our case, negation and intensifiers) which may contribute to trigger one specific meaning.

5. Conclusion

This research has adopted a multimodal perspective to explore the expression of the physical and emotional meanings of the perception verb touch. Our initial hypothesis has been confirmed: speakers use multimodal information (speech and gesture) to differentiate the meanings of the polysemous verb analyzed. With regards to our first objective, we have seen that while the gestures associated with the physical meaning tend to reach an external referent (other-touch), the gestures associated with the emotional meaning only involve touching the speaker’s body (self-touch). Concerning our second objective, the most frequent co-speech gesture, the chest-touching gesture, was mainly used when touch had an emotional meaning. Its motivation appears to be the conceptual metaphor the heart is A container for emotions (Kövecses, 1986; Lakoff & Johnson, 1980); therefore, this is the structuring mechanism by which chest-touching gestures embody the emotional meaning of touch. In relation to our third objective, linguistic elements distinguishing both meanings (pronouns, negation and intensifiers), this work showed that the physical meaning of touch is characterized by its use of a wide variety of personal pronouns and of negation words; in contrast, the emotional meaning of the verb touch occurs mostly with the personal pronoun me and it is usually modified by intensifiers. These marked differences reveal two issues: the conceptualizations of each sense and the way polysemy works. While the physical meaning appears to be considered a binary feature, the emotional meaning seems to be conceived along a gradable scale. Thus, this relation between the linguistic context and the meaning of touch corroborates that polysemy can be distributed among several elements of the sentence and not only on the polysemous item itself.

Redirecting the research focus of polysemy studies toward additional communication modalities could certainly contribute to expand our knowledge about this phenomenon and improve future works in this field. The present investigation could generate future experimental studies, which could test whether gestures are indeed elements helping to disambiguate meanings of touch combining the gesture patterns detected in this work with examples of unpredictable polysemy. Apart from this, there are of course questions that we are not able to answer just yet. First, more exploratory research is necessary to firmly establish whether this multimodal dimension can be found in other body-bound metaphors and in different languages. It would also be very useful to check whether the distinction of meanings through gesture is only related to the metaphors motivating the emotional meaning of touch or if this also happens with other figurative meanings, regardless of their emotional component.