Figurative meaning in multimodal work by an autistic artist: a cognitive semantic approach

Abstract

Research on figurative meaning in Autism Spectrum Disorder (ASD) would benefit from considering a greater variety of data types and using more diverse methods. Previous studies have predominantly applied experimental methods to investigate processing of figurative language (mostly metaphor) and have for the most part concluded that individuals with ASD have deficits in figurative language comprehension. In this study, we focus on the creation and communication of figurative meaning in discursively situated and thematically organized verbal, gestural, and pictorial data published by an autistic artist in the form of videos and comics. Across three prominent experiential themes in the data, we isolate types of conceptualizations and generalize over mappings between target and source experiences. We find that the data are rich in figurative meaning expression (e.g., metaphor and metonymy) conveyed through language, co-speech gestures, and pictures in ways that are clearly embodied (experientially based) and that reflect affordances and constraints of these modes of communication. While our case study of meaning production does not contradict previous research on figurative meaning comprehension in populations, it does indicate benefits of taking a broader and multimodal approach to figurative meaning in research on ASD.

1. Introduction

Research on figurative meaning in Autism Spectrum Disorder (ASD) has predominantly investigated the processing and comprehension of figurative language through experimental methods, and a great number of studies have concluded that individuals with ASD have difficulties understanding figurative meaning (e.g., Chahboun, Vulchanov, Saldaña, Eshuis, & Vulchanova, 2016, 2017; MacKay & Shaw, 2004; Mashal & Kasirer, 2011; Melogno, D’Ardia, Pinto, & Levi, 2012; Olofson et al., 2014; Rundblad & Annaz, 2010a). The main focus in this line of research has been verbal metaphor. A more limited number of studies have considered other forms of figurative language, such as idioms, irony, and metonymy. We propose that it is not sufficient to determine whether or not, or to what extent, individuals with ASD comprehend figurative language; there is also a need to explore the ways in which they convey figurative meaning in actual language use and through additional modes of communication such as gestures and pictures.

In this paper, we explore experiential meaning conveyed by an autistic artist through a body of work published online (videos and comics). The paper is intended as a contribution to research on figurative meaning in ASD, but also to research on figurative meaning more generally. As Littlemore points out, research on metaphor has emphasized “homogeneity over heterogeneity” and has focused largely on “prototypical” and “normal” experiences and people (2019, pp. 48–49, in a discussion of Lakoff & Johnson’s work). She argues that variation should be a focus of metaphor research, because it can tell us more about the variety of “ways in which people perceive the world […], the reasons for these different world views and the mechanisms through which they develop”, while at the same time adding to our knowledge of “the nature of embodied metaphor itself” (2019, p. 49). In our study, data are not treated as representative of meaning making by people with ASD. Rather than generalizing from the data onto a very diverse group of individuals, we use the data to make a case for broadening the scope of research on figurative meaning in ASD. A cornerstone of Conceptual Metaphor Theory (CMT) is that metaphors and other kinds of figurative meaning are “primarily a matter of thought and action and only derivatively a matter of language” (Lakoff & Johnson, 1980, p. 153). This assumption requires that our explorations of figurative meaning also account for other forms of communication (Cienki & Müller, 2008; Forceville, 2002, 2016a; Müller & Cienki, 2009). Our approach, which investigates meaning production across communicative modes, has the potential to allow new and significant insights into these areas of research and can be of interest to researchers in the language sciences and psychology, as well as gesture and comics studies.

2. Research on figurative meaning and ASD

In this section, we briefly report on previous research on figurative meaning in ASD. Some researchers have identified dilemmas in this line of autism research (e.g., Giora, Gazal, Goldstein, Fein, & Stringaris, 2012; Hobson, 2012; Olofson et al., 2014; Rundblad, 2017), and, much like these scholars, we perceive some theoretical and methodological matters in need of consideration.

2.1. theoretical scope

A central dichotomy in research on ASD is that between figurative and literal language, and in different ways, this distinction underpins the design of many studies. Some of these are designed to compare the comprehension of literal and metaphorical expressions (e.g., Chahboun et al., 2017; Kasirer & Mashal, 2014), while other studies incorporate literal expressions as alternative answers in tasks meant to test metaphor comprehension (e.g., Olofson et al., 2014). With some exceptions (e.g., Giora et al., 2012), the standing of these notions is not discussed at any length. The results of our study indicate that a separation between figurative and literal meaning needs to be considered in less absolute terms and from the perspective of diverse modes of communication.

Most studies of figurative language in ASD have focused on metaphor. Some studies have considered idioms (e.g., Chahboun et al., 2016; Mashal & Kasirer, 2011, 2012), a few have addressed metonymy (Rundblad & Annaz, 2010a, 2010b; Van Herwegen & Rundblad, 2018), the odd one a greater variety of figurative expressions (e.g., MacKay & Shaw, 2004). In light of the abundance of metaphor studies, some researchers have pointed out that there is little agreement in the field on the definition of metaphor (Olofson et al., 2014; Rundblad, 2017; Rundblad & Annaz, 2010b). The lack of a robust definition of metaphor is arguably at times a by-product of a more fundamental problem – the absence of a theoretical framework for meaning making more broadly (see Hobson, 2012, for some discussion) – something that directly impacts experimental design and hence results. In this paper, we take a broad approach to figurative meaning that comprises a range of processes, including but not limited to metaphorization.

2.2. data types

Research on figurative meaning in ASD has focused almost exclusively on language, despite the fact that a multimodal approach to figurative meaning may be especially appropriate in autism research. A limited number of studies have incorporated pictorial expression of figurative meaning in ASD, primarily as visual support for linguistic stimuli (Chahboun et al., 2016; MacKay & Shaw, 2004; Mashal & Kasirer, 2012; Olofson et al., 2014). Individuals with ASD have been described as having a visual “cognitive style” (Kamio & Toichi, 2000, p. 865) or as being prone to “visual thinking” (Kana, Keller, Cherkassky, Minshew, & Just, 2006, pp. 2488–2499). Dr Temple Grandin, an influential autistic author and scientist, writes, “I think in pictures. Words are like a second language to me” (1995, p. 19). The speaker in our data says that it is easier to convey her experiences visually than verbally (HB, Empathy). Against the background of such research claims and experiential accounts, it is of significance to consider the role of pictures in mediation of figurative meaning by individuals with ASD.

Gestures have also been overlooked in research on figurative meaning in ASD, even though difficulties using and understanding gestures have long been considered to be central to ASD. Such difficulties are, in fact, part of the diagnostic criteria for the disorder (DSM-5, American Psychiatric Association, 2013; de Marchena et al., 2019; Eigsti & de Marchena, 2017). Littlemore argues that more studies are needed of embodied metaphor production by individuals with ASD and, moreover, that such studies require “a strong focus on the use of gesture” (2019, p. 175). To our knowledge, no studies have focused specifically on the use of gestures for the expression of figurative meaning by individuals with ASD. There are also, more generally, gaps in research on the use of gestures in ASD; de Marchena et al. point out that “gestures that are used during the course of developed, fluent speech are relatively understudied in this population” and that “co-speech gestures have never been examined in verbally fluent autistic adults” (2019, p. 1439, italics in original).

A multimodal approach is also pertinent in research on ASD because it has been suggested that at the core of ASD are problems with integration across modalities (de Marchena & Eigsti, 2010; Eigsti, 2013; Kana et al., 2006). In fact, it has been proposed that “integrating across modalities in both production and comprehension may be one [of] the greatest communication challenges faced by autistic people” (de Marchena et al., 2019, p. 1451, italics in original). Multimodal studies that take a usage-based approach by considering authentic communication, in line with what we propose, can potentially offer valuable complementary insights to, for example, neuroimaging studies and cognitive testing.

2.3. methods

Research on figurative meaning in ASD has been primarily experimental with a focus on figurative language processing and has predominantly operationalized such processing as a function of response times and response accuracy, and most studies have concluded that individuals with ASD have deficits tied to figurative meaning comprehension (via language). Varied explanations for these deficits have been proposed: general language and/or communicative deficits, reduced ability to infer others’ intentions; Theory of Mind deficits; impaired executive functions; and differences in language processing (for overviews/reviews, see Chahboun et al., 2017; Eigsti, de Marchena, Schuh, & Kelley, 2011; Kalandadze, Norbury, Nærland, & Næss, 2018; Vulchanova, Saldaña, Chahboun, & Vulchanov, 2015).

While the research methods vary, what many experimental studies have in common is a focus on the processing and comprehension of isolated words or short sentences embedded in restricted contexts. Experiments have advantages in answering many types of research questions, but there are also problems associated with using artificial stimuli to assess language comprehension. Zwaan argues that using “decontextualized words and sentences as stimuli […] is problematic because the resulting patterns of brain and behavioral responses that are obtained in experiments cannot simply be extrapolated to discourse comprehension” (2014, p. 230). It is important to consider how the results of experimental studies further our understanding of actual figurative meaning use in ASD, which is contextually embedded (discursively, socially, and culturally), grounded in sensorimotor experience, and involves dynamic interplay between comprehension and production.

3. Analytical framework

In this study we focus on figurative meaning as it is achieved through what we refer to as comparative and salience construals (Paradis, 2005). Comparative construal is realized in communication as a variety of meaning phenomena, such as metaphor, analogy, simile, and literal comparison, and can be expressed via language, gestures, and pictures. The basis for any kind of comparative construal is a mapping between conceptual domains. As an example, through the conceptual metaphor life is a journey, a relatively abstract target, life, is understood in terms of a journey (source). The mapping between a life and a journey is characterized by construed similarities. Much like a journey, a life is understood as involving a beginning boundary, a spatio-temporally extended path, and an end boundary. As such, this mapping affords structure and meaning to expressions such as Follow your own path and His life was a flaming wreckage. In addition to structural correspondences (boundary, path), expressions that draw upon this mapping can also highlight other shared features such as sensory or emotive correspondences (Hartman & Paradis, 2018; Lakoff, 1990).

Salience construal, on the other hand, does not involve such cross-domain mapping, but instead entails foregrounding of elements within a single conceptual structure. Whereas comparative construal is based on construed similarity between domains, salience construal is a reference-point phenomenon whereby one element provides access to a conceptually contiguous element. Salience construal can be realized through metonymization, zone activation, or facetization (see Paradis, 2004, for a detailed account). In our discussion, we consider metonymization and zone activation. We use metonymization to describe expressions in which one meaning is used to evoke another meaning and these meanings “represent two distinct senses that, out of context, […] are associated with two different lexical items” (Paradis, 2004, p. 252). For instance, through metonymization a part can be used to stand for a whole, as in I need an extra pair of hands (hands for person/part for whole). Zone activation, on the other hand, operates within senses of expressions in communication. Zone activation is also significant to our discussion because different modes of communication – language, gestures, and pictures – come with affordances and constraints that influence the manner in which a meaning is expressed. For instance, we can use language to express the property generous as in She is a generous person. To express this meaning gesturally, we can hone in on a number of aspects of this property concept (generous). We can, for instance, use a Palm-Up-Open-Hand gesture, for which the “prototypical meaning […] can be described as giving, receiving, and showing something on the open hand” (Müller, 2017b, p. 292) (hand movement/action for property). These differences between metonymization and zone activation aside, in both cases, the evoked meanings are conceptually contingent to the expressed meanings through experience or convention.

Zone activation directs focus to contextually relevant aspects of a concept’s use potential. As such, this salience operation is also necessary in metaphorization (Paradis, 2011), so that salience and comparative construals can operate jointly. Language, co-speech gestures, and pictures can all serve to facilitate metaphorization by specifying and reinforcing appropriate aspects of an expression’s meaning potential. The metaphorical expression John is a bear can be interpreted in different ways depending on which aspects of bear are intended (e.g., strong, big, huggable, sleepy in the winter, or fond of berries). A gestural approximation of a hug concurrently with the verbal expression can be used to indicate that huggable is the most relevant meaning, in which case a metaphorical verbal expression evokes figurative meaning jointly with a zone activating gesture.

In line with the cognitive semantic model Lexical Meaning as Ontologies and Construal (LOC, Paradis, 2005), we distinguish between conceptually rich meaning domains drawn upon in meaning making (e.g., space or bear) and the schematic and conceptually simpler templates used to structure these domains (e.g., boundary, force). These templates (configurations in LOC) are akin to image schemas (Johnson, 1987; Lakoff, 1987; in gestures Cienki 2005; Mittelberg, 2018; in pictures, Forceville, 2016b) in that they can be characterized as embodied cognitive structures that capture fundamental perceptual experiences of the world, while also scaffolding new experiences and facilitating the creation of meaning.

4. Data and method

The data were drawn from published work by an autistic artist (HB) and consist of language (spoken and written), gestures, and pictures. Surveying all data types (Table 1), we isolated three central themes in HB’s work: (i) auditory processing, (ii) dissociative experiences, and (iii) echolalia and selective mutism. The expression of experiential meaning associated with these themes was then analyzed in detail.

table 1. Overview of data

Dissociation, depersonalization, lucid dreaming, echolalia, and selective mutism are terms used by HB. The same terminology is employed in research, but we use these terms because they appear in HB’s work. We give very brief descriptions of the phenomena in the respective sections, but avoid lengthy explanations because we are interested in HB’s descriptions rather than accounts of these phenomena from research or clinical perspectives.

4.1. analytical steps

We used the operational questions in Table 2 to guide our analyses of figurative meaning. In the discussion, we sometimes use standard descriptions of mappings (e.g., sounds are objects), but we would like to emphasize that such descriptions do not capture the full complexity of expressed meanings. We agree with Forceville and Paling, who argue that the “a is b formula downplays the dynamic nature of metaphors” (2018, p. 4), and that “[w]hat matters in metaphor is that things people do to, or experience through, the source domain are mapped onto the target domain, including the emotions, attitudes and values associated with these things” (2018, p. 4; also Gibbs, 2019; Müller & Cienki, 2009).

table 2. Questions for consideration of figurative meaning in verbal, gestural, and pictorial communication

4.2. data

Spoken language and co-speech gestures were accessed via a vlog published on Youtube.com (Table 1). Spoken accounts were transcribed and annotated for co-speech gestures. As mentioned, we approach figurative meaning from a production perspective, and consider how it is used in the mediation of the selected target experiences. Accordingly, our view of language and pictures, as well as gestures, is geared towards production. Naturally, co-speech gestures play a role in the comprehension of speech, but more significantly to the aims of this study, we assume that gestures “reflect internal cognitive processes” and that they serve not only “inter-cognitive or communicative functions,” but also “intra-cognitive functions” (Pouw, de Nooijer, van Gog, Zwaan, & Paas, 2014, p. 1; also Littlemore, 2019, pp. 65–69). As such, the study of gestures can yield insight into “deeply embodied aspects of the human mind” (Mittelberg, 2018, p. 2). In line with Müller and Cienki, we limit our consideration of co-speech gestures to visible “positions, orientations, and movements of the hands and forearms” (2009, p. 301). We do, however, at times comment on additional bodily expressions such as posture and facial expressions. Speech complemented by gestures is underlined in our examples, and speech accompanying the main strokes of gestures is placed in brackets. At times, we describe the placement of gestures in the speaker’s gesture space following McNeill (1992, p. 89) in terms of the speaker’s right/center/left, upper/center/lower, center/periphery/extreme periphery. In our examples, G, as in G1, G2, …, stands for gesture. We include arrows in the image stills of gestures to indicate their directions and present abbreviated titles of the videos and time codes so that readers can consult the original videos (e.g., Hearing, 1:43; see online supplementary materials, available at <http://doi.org/10.1017/langcog.2020.20> for full titles).

Written language and pictures in comics were accessed via a blog published online (Table 1). The comics have titles that help establish the relevant experiences (e.g., MELTDOWNS!!). Forceville points out that “there are ways of expressing metaphors available to comics […] that are not available to language” (2017, pp. 275–278; also El Refaie, 2009, 2019; Cienki, 2005, for gestures). He argues that comics make good data for the study of emotion metaphor because “unlike, for instance, realistic photographs and live-action films, which more or less ‘naturally’ mirror real-life manifestations of emotions, comics and cartoons make use of stereotypical exaggerations and of a rudimentary ‘sign-system’ very much like a language” (Forceville, 2005, p. 71). The comics in our data juxtapose written language and pictures, each medium bringing its own affordances and constraints to the mix.

4.3. ethical considerations and limitations

Using data published online has ethical implications, not least when the creator uses her real name and image. Even when data are published with unrestricted access in the public domain, there are grounds for caution. For this reason, we have secured HB’s permission to include her work in this paper. We would nonetheless like to briefly comment on ethics. Some researchers feel that the identities of bloggers should be protected, not least when they share personal information. Other researchers “argue that it is an issue of document copyright and intellectual property, and thus, bloggers and their material must be cited properly” (Kurtz et al., 2017, p. 8). As Kurtz et al. point out, “while it may be tempting for researchers to anonymize authors when sharing results based on the highly personal stories laid out in blogs, attempts to protect bloggers through anonymization also carry an ethical risk” (2017, p. 3) because ultimately use of such data without proper citation may infringe on the bloggers’ copyright to their materials. We uphold HB’s copyright and refer to her work.

One limitation of our study is that it considers work by a single individual. However, we do not claim that our data are representative of figurative meaning use by autistic persons. Generalization to all or most autistic persons, if/when possible, is not the only route to extending our knowledge of figurative meaning use in ASD; there is also value in testing the limits of variation by considering patterns and idiosyncrasies in actual meaning making. Another potential limitation of our study is the “‘monologue’ format” of our data, wherein the speaker expresses herself “uninterrupted” and not in conversation with others (de Marchena & Eigsti, 2010, p. 319), although there are asynchronous interactive features of the data, for instance when HB responds to listeners’ questions. There are additional facets of communication of potential interest to figurative meaning in ASD that we do not address, such as gestural timing (Eigsti & de Marchena, 2017), stress, and intonation (Cienki, 2008).

5. Autism experiences in videos and comics

The following sections account for HB’s verbal, gestural, and pictorial mediation of experiences associated with ASD. There is a wealth of data to consider in her work, but we focus on figurative meaning tied to the three prominent experiential themes: auditory processing (Section 5.1), dissociative experiences (Section 5.2), and echolalia and selective mutism (Section 5.3).

5.1. auditory processing

Auditory processing is a central theme in HB’s videos and comics. She describes difficulties in isolating and identifying speech sounds so that, even when the volume is high, she has trouble interpreting speech. Verbally and gesturally she establishes “active cross-domain mapping[s]” appealing to “the cognitive process of understanding something in terms of something else” (Cienki & Müller, 2008, p. 486), so that auditory control is accounted for by means of spatial and manual control, as in example (1), wherein she describes why she prefers to have subtitles on when she watches TV.

As demonstrated by example (1) and Figures 1a and 1b, HB verbally and gesturally portrays sounds as allowing physical manipulation, thereby realizing a metaphorical mapping from the spatial domain onto the auditory domain. In gestural expression, the resulting metaphor – sounds are objects – necessitates an initial contiguity relation whereby a “gesture is interpreted metonymically to mean that there is an object [in] the hand, and then, through metaphor, that object is interpreted as a word …” (Mittelberg & Waugh, 2009, p. 339). HB speaks of grabbing and anchoring sounds, speech sounds in particular, while gesturally enacting such grabbing and anchoring, as shown in Figures 1a and 1b. Concurrently with grabs the words out of the people’s mouths she makes three grabbing gestures with each hand in the extreme periphery of her gesture space at grabs (left hand), words (right hand), out of (both hands), and mouths (both hands). At anchors, she moves both hands (fingers slightly bent and palms first facing the camera and then facing down) from her upper left and right extreme periphery to the center of her gesture space (Figure 1b). She then holds the gesture through … them down for me. Anchors them down for me realizes a metaphor of control as being on the ground (Lakoff 1999, p. 275), and anchoring is used throughout HB’s work in language, gestures, and pictures to describe various forms of experiential control (e.g., auditory, tactile, emotional).

Fig. 1a. [grabs the words …]

Fig. 1b. … and [anchors] them down for me

In example (1), HB continues by pointing out that, without subtitles, the words sort of distort (she spreads all her fingers on both hands at distort to gesturally enact change) on the way to [her] ear. Verbally, she describes this auditory event by means of a “conduit” metaphor based on a source–path–goal schema (Reddy, 1979, for the conduit metaphor; Mittelberg, 2018, for source–path–goal in gesture; Cienki & Müller, 2008, for conduit in gesture). This schema is realized through language in example (1) by the people’s mouths (source), on the way to (path), and my ear (goal). The goal in this instance is only in part to be understood as HB’s ear; it is perhaps more significantly construed as the perception of sound, represented verbally and gesturally as spatial anchoring. The same metaphor is enacted gesturally, as shown in Figures 1c and 1d, when HB moves her open right hand from next to the camera (Figure 1c) to her ear (Figure 1d) along the sagittal axis.

Fig. 1c. on the way

Fig. 1d. [to my ear]

Through metaphorization, HB verbally and gesturally constructs an overriding mapping – sounds are objects – which in turn allows sounds to be construed as spatially positioned and subject to manipulation (e.g., contained and moved) as in expressions of grab and anchor. Auditory perception is metaphorically construed in terms of a conduit (source–path–goal), and auditory control is portrayed as spatial and manual containment.

HB also describes excessive volume as being problematic. She says that when she is out in public … like spinal tap everything is turned up to 11. This is Spinal Tap is a mockumentary from 1984 about a metal band wherein one of the band members proudly states that his amplifier goes to 11, and not 10 like most amps, and is therefore louder. The reference to Spinal Tap, signaled by like, evokes a humorous description of volume that HB uses on a number of occasions. While loud noises are difficult, HB mentions that she can tolerate such sounds better if she is able to control them. She likes toys that talk and make sounds and says that …

Figure 2a exemplifies a figurative gesture accompanying language used non-figuratively (metonymy in gesture, Mittelberg & Waugh, 2009; metonymy and embodiment, Littlemore, 2017).

Fig. 2a. they’re [slightly too] loud

In the gesture shown in Figure 2a, HB pinches her right thumb and index finger and makes a very slight turning movement, as if carefully turning a small volume dial in the air. The pinching reinforces slightly, and in tandem with the turning motion the combined gesture enacts slightly too loud. The gesture is an instance of metonymization because the action of turning a dial is used to refer to a resulting sound quality (action for property). Directly after the gesture shown in Figure 2a, HB makes another gesture concurrently with the spoken I’m controlling the noise. She stretches her right hand out in front of her at chest height and moves it back and forth in a swiveling motion while also repeatedly moving her thumb to and from the palm of her hand as if pressing a button to enact controlled noise. Like the previous gesture, the gesture shown in Figure 2b effects a salience construal, in this case zone activation – a gesture representing an action (moving the hand as if using a remote control) is used to evoke this action (manual gesture for concomitant action). Much like the gestures shown in Figures 1a and 1b that enact the grabbing and anchoring of sounds, the gestures in Figures 2a and 2b require an initial contiguity interpretation to establish (figuratively) “that there is an object [in] the hand” (Mittelberg & Waugh, 2009, p. 339). But, unlike the gestures in Figures 1a and 1b, the gestures in Figures 2a and 2b do not entail an additional metaphorical mapping between the imagined object and an auditory perception.

Fig. 2b. [I’m controlling the noise]

HB also accounts for auditory processing in her comics, as in No. 112, Auditory Processing Disorder (Figure 3). The title tells us what the comic is about, and a problematic discrepancy between a spoken message and how it is heard is likewise presented in writing: You speak butterflies. I hear bats.

Fig. 3. Comic No. 112: Auditory Processing Disorder

As in her spoken and gestural accounts of hearing, HB employs a conduit metaphor (source–path–goal) in Comic No. 112 to present her problems deciphering speech. The orientation of the schema differs between gestures and pictures due to the constraints and affordances of these modes of communication. The sagittal axis (front–back) does not exist in comic strips, which instead employ a path from left to right. The communicative event in Figure 3 is represented with the speaker to the left and the listener to the right (temporal progression is left to right orientation). The pictorial elements of Figure 3 offer much in addition to what is provided by language. Degree of detail in rendering is an affordance of drawing parallel to linguistic specificity (e.g., a person > a man > James). In Figure 3, selective salience is afforded to parts of the panel through relative degree of detail in visual representation. The speaker (source) is schematic, drawn as an outline to represent anyone, while the listener (goal) is specific (a drawing of HB) and represented with a perplexed facial expression in addition to a question mark in a thought bubble (top right-hand corner of the panel). The spoken message is portrayed as a metamorphosis of a butterfly into a bat along a left-to-right going path. A perplexing distortion of speech is conveyed using affordances of the comic, realizing a construal of sound as visible, animate, and moving from speaker to listener. Transformation of the spoken message is conveyed pictorially through animal metamorphosis and verbally through an inherent contrast between butterflies and bats.

In her comics, HB represents emotion through what Forceville refers to as “indexical signs”, which are aspects of drawings that represent prominent “symptoms” of an emotion, such as facial expressions and body posture or movement (2005, p. 77). Through zone activation, she thus expresses emotions and sensations via pictorial renditions of “physiological and expressive responses” (Kövecses, 2000, pp. 133–134) to subjective experiences. As we will see in upcoming sections, she also makes use of pictorial runes (as used by Forceville, 2005, based on Kennedy, 1982), such as jagged lines to represent loud noise or anger (see Shinohara & Matsunaka, 2009, for uses in Japanese manga), or curved lines around a body to represent movement or instability. Pictorial runes are considered metonymical (in our terminology zone activating) because they reflect salient aspects of embodied experience. In addition, “sound magnitude” is sometimes represented by “letter scale”, instantiating multimodal metaphor (Potsch & Williams, 2012, p. 18).

5.2. dissociative experiences

Dissociative experiences include feelings of derealization and depersonalization, including experienced detachment from one’s own emotions and sensations, as well as more “benign” experiences of being so absorbed in an activity that “the current surroundings are—temporarily—dissociated from awareness” (de Ruiter, Elzinga, & Phaf, 2006, p. 116). HB describes different forms of such reality distancing in her work. She primarily uses the terms dissociation, depersonalization, and lucid dreaming to describe these experiences, but her uses of these terms are not entirely consistent vis-à-vis different types of experiences. In the following, our use of lucid dreaming (in line with HB’s use of this term) corresponds roughly with the “benign” form of absorption described by de Ruiter et al. (2006), while our use of dissociation accounts for more severe and unpleasant sensations of depersonalization and/or derealization.

HB’s descriptions of dissociation feature verbal, gestural, and pictorial representations of unstable motion. Such motion construal foregrounds physical detachment from a source associated with veracity such as reality, the self, or the ground.

HB’s uses of drifting away from reality/myself in Comic No. 42, Anchor Points (Figure 4) evoke the construal of a split self (Lakoff, 1996), realizing a metaphor of a person out of control is a divided self (Kövecses, 2000, p. 44). Alongside language, the drawings in the comic likewise convey this kind of split. There are two figures in the first panel of Comic No. 42 (both are representations of HB) and one is shown to be drifting up and away from the other. Unstable motion is conveyed through language (wibbly wobbly) and pictorial runes in the form of curved lines around the body indicating movement and instability. The simile It’s like solidity disappears evokes a physical basis for understanding lack of experiential stability and permanence (solidity vs. separation). Ropes are visible at the bottom of the second panel under the drifting figure, and in the third panel these ropes attach HB’s legs to her anchor points. Like elsewhere in HB’s work, anchoring counteracts drifting, so that forces tied to physical control convey means of experiential control, which is metaphorically construed as being on the ground (Lakoff 1999, p. 275).

Fig. 4. Comic No. 42: Anchor Points

In one of her videos (Anxiety), HB describes the occasion when she first experienced depersonalization. She was in a dark pub with family …

HB moves both hands (palms toward her face) down from her eyes at my vision (Figure 5a), and then at and (G2 in example 3) she begins to move her bent arms up and down (alternating sides) as in Figure 5b to, concurrently with I feel all floaty, express the metaphor emotional instability is physical instability. Later in the same video, HB again describes anxiety and depersonalization as feeling floaty. She says she may need some weighted stuff to wear because when she is anxious her body seems to float (Anxiety, 12:04). floating thus represents a sensation that is physically experienced as unstable and that can be counteracted by weighing the body down,¹ suggesting that HB’s use of this metaphor is thoroughly grounded in her bodily experiences. This embodiment of construal also extends to other motion and force-based metaphors in her work, such as drifting and anchoring.

Fig. 5a. [my vision]

Fig. 5b. [and I feel all floaty]

In another video, HB describes dissociation as a startling experience of a mother figure leaving her alone and incomplete:

As in Comic No. 42 (Anchor Points, Figure 4), a split-self construal is evoked in example (4). HB uses a simile (it’s like there’s a mother figure inside me …) to introduce an extended comparison between dissociation and being left alone (see also the videos Innocent, 17:31 and Mental health, 13:43). One gesture accompanies example (4), in which HB strokes her head with the palms of both hands as she says somewhere in the brain, referring to the location of the mother figure. In reference to the mother figure, HB alternates between using they (they go away) and I (I come back), thus destabilizing the deictic center and reinforcing an ambivalent construal of the self. Additional instances of like extend the simile (it’s like … then I’m like …) and a number of opposites evoke experiential contrast (deictic motion: go away / come back; part–whole: on my own / complete).

Comic No. 120 (Little Miss ExistentialisT, Figure 6) likewise (verbally and pictorially) conveys a split-self construal. In the third panel of the comic, two representations of HB coexist – one observes the other through a magnifying glass. HB writes that she can look outside of [herself] and be objective. Similar to example (4), Comic No. 120 portrays ambivalence with regard to the self. Under the two drawings of HB in the third panel, she writes that she has two states equally – one more rational and objective and another more irrational and paranoid.

Fig. 6. Comic No. 120: Little Miss ExistentialisT

In Comic No. 120, HB verbally portrays the body as a container through expressions such as being inside my head, internalized all these thoughts and feelings, and my brain is full of magical thinking (body as container, Lakoff, 1987, p. 383; also Kövecses, 2000). She describes how in [her] teens, she fell inwards into [her] alien delusions and conspiracy theories. Pictorially, this experience of inward motion is shown as a spiral shape, representing a vortex, with a brain at the center and a drawing of HB falling towards it. This multimodal representation also shows words being sucked into the vortex.

HB at times makes use of specific affordances of comics to pictorially express such “inside–outside relations” (El Refaie, 2019, p. 9). In Comic No. 26, Squash (Figure 7), she describes the experience of bad feelings that [she doesn’t] understand. As in her accounts of dissociation, HB uses uncontrolled motion (drift about) to describe lack of experiential control.

Fig. 7. Comic No. 26: Squash

Emotive control is described in Comic No. 26 through language and pictures in terms of containment. In the first panel, a jagged shape protrudes out of the panel, and in the second panel, HB’s feelings are shown to drift about like clouds, which are not contained within the panel, but go outside of it. In the fourth and last panel, HB’s hands are represented as firmly gripping the clouds, now contained within the panel (emotive control is manual containment). As she gains control in the fourth panel, the pictorial perspective also changes to that of the first-person, thereby allowing the viewer to visually adopt HB’s perspective, potentially facilitating emotive/sensory simulation of her experience.

In her work, alongside descriptions of dissociation, HB describes lucid dreaming, a more pleasant form of reality distancing that involves total absorption in daily activities. In Comic No. 44, The fantasy zone is my Everyday (Figure 8), HB describes how she disconnects from reality and drift[s] away to imagination land,² especially when making coffee, taking [her] antidepressant and doing bathroom activities.

Fig. 8. Comic No. 44: The fantasy zone is my Everyday

Like dissociation, lucid dreaming is portrayed in HB’s work in terms of motion. However, the motion schema is realized differently for dissociation and lucid dreaming. Dissociation is described as movement away from a prominent reference point (drifting away from reality), while lucid dreaming is more often described as movement toward something: drifting off into my creative and fantasy world (Pegasus, 8:17) and drifting away to imagination land (Comic No. 44 / Figure 8). These two types of reality distancing thus foreground different parts of a source–path–goal schema: dissociation foregrounds source and path, and lucid dreaming path and goal. However, both forms of reality distancing are accounted for in terms of a split-self and a disconnect from reality (Figure 8).

Through language, gestures, and pictures, HB portrays dissociative experiences in terms of physical experiences. Reality is described in terms of solidity, permanence, and visibility. Dissociation is described as uncontrolled motion away from the ground or the self, and lucid dreaming is portrayed as movement toward an imaginary space. Experiential control, on the other hand, is described in terms of physical control, often as manual containment or forceful downward motion (anchoring). As in her descriptions of sensory and emotive experiences generally, contrast between experiential control and lack of control is mediated as tension between spatial motion / instability and rest/stability.

5.3. echolalia and selective mutism

Echolalia is characterized by repetitions of sounds, including but not limited to speech sounds, and can be considered “among the most recognizable characteristics of autism spectrum disorders” (Stiegler, 2015, p. 750; see also Bogdashina, 2005, pp. 174–195, for communicative and non-communicative echolalia). Echolalia is mediated in HB’s work through construal of the body as a container with an inside and an outside; the inside is private and the outside is public; thoughts are kept on the inside whereas movements and sounds are forcefully moved outwards (for a similar construal in Tourette Syndrome, see Hartman, 2017). In example (5), HB describes how her echolalia is projected outwards; it is not focused inwards. She uses a simile (a bit like) to compare aspects of her echolalia (talking to the invisible people) to one of her favorite TV-series, Dr Who. She explains how her echolalia involves something akin to breaking the fourth-wall. ³

Construed opposition between an inside and an outside is reinforced in example (5) by the verbs focus (constraining stabilizing action) and project (forceful outward motion). HB uses basically as a way of adjusting the comparative mapping between her echolalia and Dr Who’s fourth wall breaking. Such figurative “tuning” appears throughout HB’s work in conjunction with such mappings (for tuning devices, see Deignan et al., 2013). In example (5), co-speech gestures accompany both project outwards (G1, G3) and focus inwards (G2). She gesturally enacts project it outwards (twice as shown in Figures 9a and 9c) by indicating a forceful outward trajectory of her echolalia (force in gesture, Mittelberg, 2018). For both these gestures, HB moves her open hands in an arc from her shoulders to the left, away from her body. At focus inwards, on the other hand, she moves her slightly cupped hands close to the head front to back, as shown in Figure 9b.

Fig. 9a. … sort of [projects it outwards] …

Fig. 9b. … rather than [focus inwards] …

Fig. 9c. … I [project it outwards]

In other words, verbally and gesturally, construal of the body as a container is combined with forceful deictic motion. A boundary between inside and outside, described in example (5) as the fourth wall, can be traversed through echolalia.

HB describes her echolalia as being of two kinds. Alongside speech, her gestures indicate forceful and rhythmical qualities of these types of echolalia, making the descriptions accessible also to viewers who are unfamiliar with her spoken references (Family Guy, the Kool-Aid Man). The first type of echolalia is described as:

Along with example (6), HB looks to the left (oh no), to the right (oh no), and then she leans in close to the camera for an emphatic OH YEAH. She goes on to compare this to a second type of echolalia:

The speech in examples (6) and (7) and the concurrent gestures in Figures 10a and 10b illustrate both salience and comparative construal. The two types of echolalia are comparatively (metaphorically) construed as a steady train and the kool-aid man. However, salience construal – largely realized though gestures – activates force and rhythm as relevant zones for this comparative mapping. In example (7), the rhythmical motion is evoked by the steady train of echolalia in collaboration with a slow, wavy, up-and-down right-hand gesture (shown in Figure 10a). This is contrasted to the already introduced type of echolalia, the Kool-Aid Man, accompanied in example (7) by an explosive forward-moving gesture from the shoulder with an open hand (Figure 10b). As Mittelberg points out “speakers may readily use their heads, manual gestures, or postures to (re)enact force qualities they have experienced or imagine experiencing” (2018, p. 10), and for these gestures, HB uses not only her hand and arm, but also her posture to indicate the force of her echolalia. For the steady train, she is relaxed and leaned back, and for the Kool-Aid Man she leans forward. These examples demonstrate HB’s use of force contours associated with two entities (steady train and the Kool-Aid Man) to evoke comparative construal. The construed force qualities additionally prompt possible social responses associated with these entities. While a train chugs along in a predictable pattern, the Kool-Aid Man bursts through walls in an unexpected (and inappropriate) fashion.

Fig. 10a. that’s like [the steady train of echolalia]

Fig. 10b. that’s [the Kool-Aid Man]

HB also describes selective mutism. Selective mutism is characterized by the “absence of speech in almost all […] or only certain situations” and has been shown to be common in ASD (Steffenburg, Steffenburg, Gillberg, & Billstedt, 2018, p. 1163). Both echolalia and selective mutism thus involve the making (or not making) of sound, often speech. Unlike echolalia, which HB describes as forceful outward motion, selective mutism is portrayed as interrupted motion or lack of motion. In example (8) she uses an ice metaphor.

The gesture that coincides with you just freeze up in example (8) is shown in Figures 11a and 11b. HB holds her hands at her throat and lifts her head slightly, as if trying to get sound out. Jointly with the metaphorical verbal expression, the gesture’s position at the throat is suggestive of speech being stopped in the vocal tract and thus, through zone activation, the gesture reinforces relevant zones for the verbally expressed ice metaphor.

Fig. 11a. you just [freeze up]

Fig. 11b. you just [freeze up]

Through speech and gestures, HB describes echolalia in terms of forceful motion and selective mutism in terms of lack of motion or contained motion. Her body is portrayed as a container with an inside (hidden/invisible/inaudible) and an outside (public/visible/audible). A boundary separating these (e.g., a fourth wall) can be traversed through sounds and movements. Essential to her accounts of echolalia are motion, rhythm, and force.

6. Implications for research on figurative meaning in ASD

In this study, we considered discursively situated and thematically organized verbal, gestural, and pictorial data published with the expressed intention of mediating subjective autism experiences to other people. Our results indicate that figurative meaning in this work is malleable and multifunctional, conveys diverse experiences, and transcends single modes of communication. Metaphorization, for instance, is often realized through language and gestures or pictures jointly, as when HB talks of sounds that you grab while gesturally enacting grabbing, or when she writes You speak butterflies. I hear bats. alongside drawings of these animals. Language, gestures, and pictures contribute different nuances of meaning, so that even when their contributions reinforce the same figurative meaning, these are far from redundant (El Refaie, 2014, p. 152). Additionally, literally used meanings via one mode of communication can be complemented by figurative meanings via another mode, as when HB uses gestures metonymically to complement literal speech. It is well known from previous studies that distinguishing between literal and figurative language is not a simple matter (e.g., Hartman & Paradis, 2018; Paradis, 2004, 2015; Winter, 2019). When we consider multimodal communication, including language (spoken or written), gestures, and pictures, as in our data, these difficulties are compounded by the fact that meanings (literal and figurative) are communicated simultaneously via different modalities as “single integrated expression[s] of meaning” (McNeill, 1992, p. 79, on gesture and speech; El Refaie, 2019, for verbo-visual metaphor). In our view, a broader focus on figurative meaning in communication can be valuable to research on figurative meaning in ASD. If we study language alone (and comprehension only), we may miss significant means of figurative communication used by individuals with ASD (and also more generally).

To express her experiences, HB draws upon a number of metaphors that are arguably conventionalized in English, such as the conduit metaphor. In her work, these metaphors are instantiated verbally, gesturally, and pictorially. Her use of metaphor, not least in gestural expressions, strongly suggests that she employs such figurative mappings in a manner that is meaningfully embodied. Gestures reinforce mappings by providing “an embodied subjective experience of [a] metaphorical source domain” (Müller, 2017a, p. 299), and in HB’s work metaphoricity is not only foregrounded, but also perceptually anchored, in the sense that a “gesture is an embodied experience” (2017a, p. 299, italics in original). Conventional metaphors are at times regarded in ASD research as constituting rather stable and fixed form–meaning pairings that can simply be retrieved from memory. The embodied (experientially based) use of conventionalized metaphor in our data suggests that research on figurative meaning in ASD could benefit from viewing conventionality of figurative meaning as “an issue of discourse and interaction, of language in use” (2017a, p. 298), as opposed to a stable feature of figurative mappings. In addition, as El Refaie points out in her discussion of metaphor in graphic novels, conventional mappings can be used in creative ways for “discovery of new connections between two areas of experience and the imaginative reinterpretation of conventional metaphors” (2014, p. 152), so there is room for creativity in the use of conventional mappings as well. It is of significance to consider not only the presence or absence of figurative mappings, but also the manner and mode through which they are expressed in communication.

Some characteristics of HB’s figurative meaning expression are apparent only when we take a comprehensive view of her work. Certain concepts, such as anchoring, are employed over and over again to cover different experiences. Extensive use of single concepts for multiple experiences can be seen as limiting compared to using multiple concepts to describe the same experiences, but it can also be seen as highly creative because it optimizes the elasticity of these concepts. In order to capture this type of creative ability, it could be advantageous to complement experimental research on figurative meaning in ASD with qualitative studies of naturally occurring figurative meaning use by individuals with ASD.

Our study also indicates that research in this field could benefit from a focus on not only comprehension of isolated metaphors, but on patterns of figurative meanings in use. In our data, binary opposition (e.g., motion–rest, containment–release) expressed through language, gestures, and pictures facilitates mediation of a range of dynamic experiences, many of which are tied to control.

ASD constitutes a spectrum and capacity for figurative reasoning and expression likely varies to a great extent between individuals. In addition to successfully mediating her experiences verbally and gesturally, HB skillfully expresses nuances of these experiences pictorially. However, the study of drawing(s) is not limited to investigations of advanced artwork, like HB’s, but could be used more broadly in ASD research on figurative meaning.

7. Conclusion

To investigate the role of figurative meaning in ASD, researchers need to consider both production and comprehension of meaning, ideally through a range of methods and diverse data types. Such an expanded investigative scope should take figurative meaning as its focus as opposed to figurative language and include different modes of communication. In this study, we carefully consider one autistic artist’s use of language, co-speech gestures, and pictures and we couch our observations in a coherent analytical framework. We focus on three prominent themes in the published videos and comics and find that experiential meaning is expressed figuratively through varied and multimodal use of a range of construals of salience and comparison. The use of figurative meaning in our data paints a different picture than much previous research, which has presented primarily deficits in figurative meaning comprehension in ASD. As a case study of meaning production, our study does not contradict research on figurative meaning comprehension in populations, but it indicates benefits of an approach that takes a broader perspective on figurative meaning and that involves consideration of more than one mode of communication.