Leading voices: dialogue semantics, cognitive science and the polyphonic structure of multimodal interaction

Abstract

The neurocognition of multimodal interaction – the embedded, embodied, predictive processing of vocal and non-vocal communicative behaviour – has developed into an important subfield of cognitive science. It leaves a glaring lacuna, however, namely the dearth of a precise investigation of the meanings of the verbal and non-verbal communication signals that constitute multimodal interaction. Cognitively construable dialogue semantics provides a detailed and context-aware notion of meaning, and thereby contributes content-based identity conditions needed for distinguishing syntactically or form-based defined multimodal constituents. We exemplify this by means of two novel empirical examples: dissociated uses of negative polarity utterances and head shaking, and attentional clarification requests addressing speaker/hearer roles. On this view, interlocutors are described as co-active agents, thereby motivating a replacement of sequential turn organisation as a basic organising principle with notions of leading and accompanying voices. The Multimodal Serialisation Hypothesis is formulated: multimodal natural language processing is driven in part by a notion of vertical relevance – relevance of utterances occurring simultaneously – which we suggest supervenes on sequential (‘horizontal’) relevance – relevance of utterances succeeding each other temporally.

1. Introduction

Let us face it: it is all about meaning. A phoneme is the smallest meaning-distinguishing sound, a morpheme a meaning-carrying form. Most distinctions even in syntax – long regarded the core of linguistics – are based on semantic considerations. Now, investigating meanings poses a perplexing problem: we cannot directly encounter them or point at them or count them, and talking about meaning itself requires meaning. There are different ways to proceed in this situation. In psycholinguistics, for instance, experimental studies are used, where meaning is observed indirectly by observable features of language users’ processing of stimuli sentences. A quite different approach has been developed in philosophy and formal semantics: here, the act of interpretation is objectified in terms of mathematical models, that is, ‘small worlds’ which are used as items within which semantic representations of natural language expressions are evaluated. Both approaches exemplify research programmes that target distinct levels of meaning: This has recently been discussed in terms of Marrian (Marr, 1982) implementation versus computation (resp. neural activity vs. behaviour; Krakauer et al., 2017), and in terms of cognitive architectures complementing algorithmic representational models (Cooper & Peebles, 2015), among others. With regard to language, there has been a long-standing collaboration: answers to What? questions are provided by formal grammar and theoretical linguistics, How? questions are addressed in psycholinguistics. Yet, this cooperation cooled down for a while (Ferreira, 2005). There are several reasons for the disenchantment. With regard to meaning proper – that is, semantics – we think that theoretical linguistics ‘underaccomplishes’ the obligation to provide cognitively potent models of meaning given mainstream formal semantics’ sentence-oriented approach. The reason is this: consider a toy world that consists of three individuals, $ a $ (Aydın), $ n $ (Nuria), and $ x $ (Xinying). A mainstream model-theoretic approach to semantics maps natural language expressions onto terms of a formal language (mostly predicate logic), which in turn are interpreted in terms of the individuals of a world (denotation or reference). The meaning of a one-place predicate like sleep, for instance, is the set assigned to the formal translation $ slee{p}^{\prime } $ of the verb, and in our toy model (let us assume) is $ \left\{a,x\right\} $ (i.e., Aydın and Xinying sleep). The meaning of the sentence Aydın sleeps is compositionally derived as $ slee{p}^{\prime }(a) $ and is true iff (abbreviates if and only if) α ∈ . However, the formulae used in traditional formal semantics (e.g., $ slee{p}^{\prime }(a) $ ) are dispensable: they eventually get reduced to the basic notions of truth and reference ( $ slee{p}^{\prime }(a) $ , e.g., is true in our toy model) and therefore have no cognitive bearing. Hence, while being formally precise, it is unclear whether an approach of such a kind succeeds to ‘formulate the computational problem under consideration’, as Cooper and Peebles (2015, p. 245) put it.

Nonetheless, over the past 30 years, theoretical linguistics has developed a different sort of a formal model of meaning, namely dynamic update semantics – most notably Discourse Representation Theory (Kamp & Reyle, 1993) – where the construction of semantic representations is constitutive of meaning (Kamp, 1979, p. 409) and has cognitive (Hamm et al., 2006) and neuroscientific (Brogaard, 2019) interpretations (see also Garnham, 2010). The sentence Aydın sleeps is processed within a dynamic update semantics in such a way that a file ¹ for an object $ x $ (due to the proper name Aydın) is opened (if new) or continued (if known). We emphasise this detail since it reveals a dynamic shift in the notion of meaning: the meaning of an utterance updates a previous context and returns an updated context. Hence, reducing the meaning of an assertive sentence to truth and reference is replaced (or at least complemented) by its context change potential. The (new or continued) file is then populated with conditions that $ x $ is named Aydın (if not already known) and $ x $ is sleeping. ² A dynamic update semantics rooted in spoken language – also known as dialogue semantics – is KoS (Ginzburg, 1994, 2012). KoS [not an acronym] is formulated by means of types from a Type Theory with Records (TTR; Cooper, 2013; Cooper & Ginzburg, 2015) instead of terms and expressions from an interpreted language like predicate logic. There is a straightforward model-theoretic, denotational construal of types much in the spirit of classical formal semantics (Cooper, n.d.), but one can also think of types as symbolic but embodied structures which are rooted in perception (as Cooper, n.d., points out), which label instances of linguistic processing (Connell, 2019; Frankland & Greene, 2020), and are associated with motor and perception activation (Bickhard, 2008; Hummel, 2011; Meteyard et al., 2012). Indeed, types can also be construed neurally (Cooper, 2019).

Why promote dialogue semantics and to a cognitive science audience? Cognitive science has come to acknowledge that multimodal interaction is the ‘central ecological niche’ of sentence processing (Holler & Levinson, 2019, p. 639). The dominant view on interaction and coordination in cognitive science is a systemic view: interlocutors are observed and construed as a complex system – there is work on systemic coupling on neural, behavioural, and attentional, goal-predicting levels (Fusaroli et al., 2014; Hasson et al., 2012; Pickering & Garrod, 2004, 2013; Sebanz & Knoblich, 2009).

However, while a systemic view certainly provides important insights into the neural and cognitive underpinnings of alignment and communication within its ecological niche, we argue that significant lacunae remain unless this is complemented by analyses of the verbal and nonverbal signals (and their interactions) that constitute multimodal communication: the multimodal, interactive turn in cognitive science induces a renewed need for a precise formulation of the computational What? problem. Simplifying to a necessary degree, Fig. 1 summarises the semantic position within the multimodal discourse landscape. We focus on contents (cont) here and demonstrate throughout how contents depend to a very large extent on a fine-grained structured context (ctxt).

Fig. 1. A dialogue-semantics perspective for completing the systemic understanding of multimodal discourse.

In particular, we argue that a dialogue-semantics perspective makes at least three crucial contributions:

• • Dialogue semantics provides a formal notion of content that is needed in order to define different kinds of cross-modal signals. From gesture studies, we have the notion of multimodal ensembles (Kendon, 2004) – utterances including speech–gesture composites – and from psycholinguistics, multimodal gestalts (Holler & Levinson, 2019) – recurrent, statistically significant multimodal actions, signals or features which are interlinked by a (common) communicative intent or meaning. ³ However, recurrent ensembles or gestalts often occur with a simplification in form (Lücking et al., 2008). This raises the issue of how formally different gestalt or ensemble tokens are assigned to a common type instead of to different ones. Moreover, how to account for communicative signals, features or utterances which do not belong to a unique ensemble or gestalt? We argue mainly based on data from head shaking (Sections 2.1 and 4.2) that an explicit semantic analysis is needed to provide the required identity conditions and, among others, tell apart multimodal behaviour that, with respect to its perceptual forms, deceptively looks like a unified composite utterance.

• • In line with research on attentional mechanisms (Mundy & Newell, 2007), we discuss (non-)attending to interlocutors as new attentional data and argue that it can be used to explain – as far as we know – hitherto unstudied occurrences of specific types of other-repair in discourse targeting the speaker and hearer roles.

• • Timing and coherence within multimodal interaction is a subject sui generis for both cognitive science and dialogue semantics: dialogue agents are co-active during more or less the whole time of interaction – see also the analysis of Mondada (2016). ⁴ Accordingly, the notion of turn should be replaced by the notion of leading voice. Moreover, this applies even to spoken contributions, where despite the entrenched assumption of one speaker per turn, assumed in Conversation Analysis to be one of the essential and universal structuring notions of conversation (Levinson & Torreira, 2015), overlap is in certain situations and with inter-subject variation an acceptable option (Bennett, 1978; Falk, 1980; Hilton, 2018; Tannen, 1984; Yuan et al., 2006). In this respect, multimodal interaction is akin to a polyphonic musical piece. ⁵ Just like polyphonic music is organised by harmonic or contrapuntal composition techniques, polyphonic interaction is driven partly by dialogical relevance or coherence. ⁶ Note that the terms ‘leading’ and ‘accompanying voices’ also give rise to a subjective interpretation: a speaker or gesturer may have the impression to hold the leading voice in a conversation regardless of observational evidence.

Section 2 illustrates the above-mentioned challenges posed by multimodal phenomena. Section 3 then sketches a formal theory of multimodal interaction that involves: (i) semantic representations which can (and should) be construed as cognitive information state representations, (ii) partiturs (multimodal input representations) and (iii) lexical entries and conversational update rules that capture dialogical relevance enabling incremental and predictive processing. The machinery is applied to analyse the sample observations in Section 4. The formal theory may appear complex to those exposed to it for a first time or who do not endorse formal approaches, but its expressive granularity has been developed in light of many diverse dialogical phenomena, as explained in Section 3.2. In particular, it facilitates formulating our ultimate upshot in Section 5 in an explicit way: our claim, the multimodal serialisation hypothesis, is that vertical relevance – relevance of utterances occurring simultaneously – supervenes ⁷ on horizontal relevance – relevance of utterances succeeding each other temporally. Hence, multimodality compresses interaction temporally, but is not richer in terms of semantic expressivity. In other words, and with certain caveats we will spell out, simultaneous interaction, though more efficient and perhaps more emotionally engaging and aesthetically pleasing, can always be serialised without loss of semantic information. This is a rather strong claim, and it needs to be refined right away. On the one hand, there are multimodal signals which simply cannot be separated – for instance, you cannot separate spoken utterances from their intonation: they are coarticulated. This is, however, not just due to a common channel: speech–laughter is transmitted via the acoustic channel but can be separated into speech and laughter. On the other hand, serialising multimodal input gives rise to different possible orderings. We do not claim that every order of the elements from multimodal input when put into a sequence is equivalent, quite the contrary: we provide evidence for the opposite below. But in accordance with the claim, one of the possible orderings is semantically equivalent to the original multimodal input. Simultaneity and sequentiality in multimodal interaction can always become manifest in two ways: (i) across interlocutors and (ii) within one interlocutor. The multimodal serialisation hypothesis intentionally generalises over both manifestations (in fact, the empirical phenomena discussed in the following involve both kinds). Given these qualifications, the expressivity claim is a hypothesis that has to be explored in multimodal communication studies by cognitive science, theoretical linguistics, gestures studies and related disciplines.

2. Observations

2.1. Head shake

Eight uses of the head shake are documented by Kendon (2002). The most well-known (Kendon’s use I) is a non-verbal expression of the answer particle ‘No’. Thus, a head shake can be used in order to answer a polar question:

Depending on whether A produced a negative or a positive propositional kernel in the question, B’s head shake is either a denial of the positive proposition or a confirmation of the negative one (which is not discussed by Kendon, 2002). In uses such as those documented in (1), the head shake conveys a proposition. However, the proposition expressed by the head shake is in part determined by the context in which it occurs – in (1), it can be one of two contradictory dialogue moves: a denial or a confirmation one. Hence, what is needed for instances such as (1) is a notion of contextually aware content. We provide such a content in Section 4.2.

Having a context-aware semantic representation of denial at our disposal, it makes predictions for head shakes in other contexts as well. Consider (2):

While (2a) is fully coherent, (2b) (at least without additional context – examples of which we provide in Section 4.2) has a contradictory flavour: the head’s denial is not matched in speech. Hence, in order to discuss apparently simple uses of head shakes, one already has to draw on a precisely formulated, contextually aware notion of contents.

2.2. Co-activity and communicative breakdown

A well-known pattern of co-activity in spoken discourse is the interplay of monitoring and feedback signals. For instance, backchannelling signals such as nodding or vocalisations such as ‘mhm’ influence the development of discourse (Bavelas & Gerwing, 2011). The absence of monitoring or feedback signals leads to communicative breakdown since it raises the question whether one is still engaged in the joint interaction. Suppose $ A $ and $ B $ are sitting on a window seat in a café. $ A $ is telling $ B $ about a near-accident she witnessed on Main Street the day before. While $ A $ has been talking, $ B $ has been continuously staring out of the window. Thus, $ A $ lacks attentional gaze signals, which in turn raises doubts about $ B $ ’s conversational involvement. Accordingly, $ A $ will try to clarify $ B $ ’s addressee role:

$ A $ ’s clarification request or (other-initiated) repair ⁸ is a natural response in the sketched situation since it is triggered by a neurocognitive social attention mechanism (Nummenmaa & Calder, 2009) in response to a violation of a behavioural norm. However, seen from a turn-based view, (3) is not easy to explain: $ A $ is speaking and $ B $ is listening, so the all-important roles of hearer and speaker are clearly filled – and now it is $ B $ ’s turn. Crucially, (3) could equally be made by $ B $ if s/he gets the impression $ A $ is rambling incoherently.

2.3. Summary

The upshot of the few phenomena we have discussed above is that multimodal interaction is:

• • driven by a richly structured and fine-grained context,

• • which is distinct but aligned across the participants,

• • where the participants typically monitor each other’s co-activity.

In the following, we introduce a theoretical dialogue framework which can capture these observations.

3. Polyphonic interaction: cognitive–formal tools

3.1. Partiturs

A prerequisite for an analysis of multimodal interaction is a systematic means for telling apart the manifold verbal and non-verbal signals. We employ tiers in this respect, where a tier is built following the model of phonetics and phonology. Phonetics comprises the triple of articulatory phonetics, acoustic phonetics and auditory phonetics. Signalling of other communication means can be construed in an analogous way. For instance, facial muscles – facial display – vision defines the tier for facial expression. Tiers give rise to a uniform approach to linguistic analysis which ultimately rests on perceptual classification (cf. Cooper, n.d.), which we formulate in terms of TTR (a Type Theory with Records; Cooper, n.d.; Cooper & Ginzburg, 2015). Classification in TTR is expressed as a judgement: in general, object $ o $ is of type $ T $ . With regard to spoken utterances, a record $ r $ (situation) providing a sound event (construed as an Individual) – $ r= $ $ \left[\begin{array}{ll}{\mathrm{s}}_{\mathrm{event}}& =\mathrm{a}\\ {}\mathrm{c}& =\mathrm{s}1\end{array}\right] $ – is correctly classified by a record type $ T= $ $ \left[\begin{array}{ll}{\mathrm{s}}_{\mathrm{event}}& : Ind\\ {}\mathrm{c}& : Sign\left({\mathrm{s}}_{\mathrm{event}}\right)\end{array}\right] $ (i.e., $ r:T $ ), iff the object labelled ‘s_event’ (in this case, a soundwave) belongs to the phonological repertoire of the language in question. ⁹

Tiers can be likened to different instruments on a musical score: a partitur. ¹⁰ Building on Cooper (2015), we represent partiturs as strings of multimodal communication events $ e $ , which are temporally ordered sequences of types. One can think of strings in terms of a flip-book: a dynamic event is cut into slices, and each slice is modelled as a record type. Such string types (Cooper, n.d.; Fernando, 2007) are notated in round brackets and typed in an obvious manner, where RecType is the general type of a record type:

The progressive unfolding of subevents on the various tiers in time gives rise to incremental production and perception. Formally, this is indicated by the Kleene plus (‘ $ {}^{+} $ ’). (4) shows the type of multi-tier signalling, and it remains silent concerning potential inherent rhythms of the individual tiers. In fact, it has been argued that different kinds of gestures exhibit a specific ‘rhythmic pulse’ (Tuite, 1993), as does speech, which lead to tier-specific temporal production cycles that may jointly peak in synchronised intervals (Loehr, 2007). The temporal relationship between signals on different tiers is therefore specified in a relative way, following the example set by the Behaviour Markup Language (Vilhjálmsson et al., 2007). It should be noted that the subevents on partiturs can be made as detailed as needed – from phonetic features to complete sentences or turns. A reasonable fine-grained temporal resolution of partiturs seems to be the level of syllables. Arguably, syllables constitute coherent events as do tones in a melodic phrase and movement elements in locomotion, and to which attentional processes are rhythmically attuned in the sense of Jones and Boltz (1989). See Lücking and Ginzburg (2020) for more details on parsing on partiturs. We will make crucial use of record-type representations along these lines in the following.

3.2. Cognitive states in dialogue semantics

We model cognitive states by means of dialogue agent-specific Total Cognitive States (TCS) of KoS (Ginzburg, 1994, 2012; Larsson, 2002; Purver, 2006). A TCS has two partitions, namely a private and a public one. A TCS is formally represented in (5). In a dialogue between A and B, there are both, A.TCS and B.TCS. ¹¹

(The symbol “ $ := $ ” indicates a definition relation.)

Now, trivially, communication events take place in some context. The simplest model of context, going back to Montague (1974), is one which specifies the existence of a speaker, addressing an addressee at a particular time. This can be captured in terms of the type in (6), which classifies situations (records) that involve the obvious entities and actions.

However, over the last four decades it has become clearer how much more pervasive reference to context in interaction is. Indeed, arguably, this traditional formulation gets things backwards in that it seems to imply that ‘context’ is some distinct component one refers to. In fact, as will become clear, following Barwise and Perry (1983), we take utterances – multimodal events – to be the basic units interlocutors assign contents to given their current cognitive states and from this generalise to obtain utterance types, the meanings/characters semanticists postulate.

The visual situation is a key component in interaction from birth (see Tomasello, 1999, Ch. 3). ¹² Expectations arise due to illocutionary acts – one act (querying, assertion and greeting) giving rise to anticipation of an appropriate response (answer, acceptance and counter-greeting), also known as adjacency pairs (Schegloff, 2007). Extended interaction gives rise to shared assumptions or presuppositions (Stalnaker, 1978), whereas uncertainties about mutual understanding that remain to be resolved across participants – questions under discussion – are a key notion in explaining coherence and various anaphoric processes (Ginzburg, 1994, 2012; Roberts, 1996). These considerations among several additional significant ones lead to positing a significantly richer structure to represent each participant’s view of publicised context, the dialogue gameboard (DGB), whose basic make up is given in (7):

It should be emphasised (again) that there is not a single DGB covering a dialogical episode, but a DGB for each participant. Participants’ DGBs are usually coupled, that is, develop in parallel. Participant specific DGBs, however, allow to incorporate misunderstandings, negotiation, coordination and the like in a straightforward manner in KoS. In any case, facts represents the shared assumptions of the interlocutors – identified with a set of propositions. In line with TTR’s general conception of (linguistic) classification as type assignment – record types regiment records – propositions are construed as typing relationships between records (situations) and record types (situation types), that is, as Austinian propositions (Austin, 1950; Barwise & Etchemendy, 1987). More formally, propositions are records of type $ \left[\begin{array}{ll}\mathrm{sit}& : Rec\\ {}\mathrm{sit}\hbox{-} \mathrm{type}& : Rec Type\end{array}\right] $ . ¹³ The ontology of dialogue (Ginzburg, 2012) knows two special sorts of Austinian proposition: grammar types classifying phonetic events (Loc(utionary)Prop(ositions)) and speech acts classifying utterances (Illoc(utionary)Prop(ositions)). Both types are part and parcel of locutionary and illocutionary interaction: dialogue moves that are in the process of being grounded or under clarification are the elements of the pending list; already grounded moves (roughly, moves that are not contested, or agreed-upon moves) are moved to the moves list. Within moves, the first element has a special status given its use to capture adjacency pair coherence and it is referred to as LatestMove. The current question under discussion is tracked in the qud field, whose data type is a partially ordered set (poset). Vis-sit represents the visual situation of an agent, including his or her visual focus of attention (foa), which, if any (attention may be directed towards something non-visual, even non-perceptual ¹⁴), can be an object (Ind), or a situation or event (which in TTR are modelled as records, i.e., entities of type Rec). Mood tracks a participant’s public displays of emotion (i.e., externally observable appraisal indicators such as intonation or facial expressions, which often do but need not coincide with the participant’s internal emotional state), crucial for inter alia laughter, smiling and sighing (Ginzburg et al., 2020b), and, as we shall see, head shaking as well. The DGB structure in (7) might seem like an overly rich notion for interlocutors to keep track of. Ginzburg and Lücking (2020) show how the DGB type can be recast as a Baddeley-style (Baddeley, 2012) multicomponent working memory model interfacing with long-term memory.

Given that our signs (lexical entries/phrasal rules) are construed as types for interaction, they refer directly to the DGB via the field dgb-params. For instance, the linguistic meaning of the head shake from (1) in Section 2.1 patterns with the lexical entry for ‘No’ when used as an answer particle to a polar question (a.k.a. a ‘yes–no’ question) and, following Tian and Ginzburg (2016), is given in (8).

When used in the context of a polar question with content $ p $ (the current question under discussion – MaxQUD – is $ p $ ?), saying ‘No’ and/or shaking the head asserts a ‘No semantics’ applied to $ p $ . NoSem( $ p $ ) in turn is sensitive to the polarity of the proposition to which it applies (cf. the discussion of the head shake in Section 2.1). To this end, positive (PosProp) and negative (NegProp) propositions have to be distinguished. If a negative particle (not, no, n’t, never and nothing) is part of the constituents of a proposition $ \neg p $ , then $ \neg p $ is of type NegProp ( $ \neg p $ : NegProp). The corresponding positive proposition, the one with the negative particle removed, is $ p $ ( $ p $ : PosProp). With this distinction at hand, NoSem works as follows:

(Note that the result of ‘NoSem( $ p $ )’ is always of type NegProp – $ p $ : NegProp means that $ p=\neg q $ , which NoSem leaves unchanged according to the second condition in (9).) (8) and (9) provide a precise characterisation of answer particle uses of negation and head shake and therefore make testable predictions concerning meaning in context.

The evolution of context in interaction is described in terms of conversational rules, mappings between two cognitive states, the precond(ition)s and the effects. Two rules are given in (10): a DGB that satisfies preconds can be updated by effects.

Within the dialogue update model of KoS, following Ginzburg et al. (2020a), QUD gets modified incrementally, that is, at a word-by-word latency (or even higher). ¹⁵ Technically, this can be implemented by adopting the predictive principle of incremental interpretation in (11) on top of partitur parsing (see Section 3.1). This says that if one projects that the currently pending utterance (the preconditions in (11)) will continue in a certain way (pending.sit-type.proj in (11)), then one can actually use this prediction to update one’s DGB, concretely to update LatestMove with the projected move; this will, in turn, by application of the existing conversational rules, trigger an update of QUD: ¹⁶

We will make use of utterance projection in analysing head shakes synchronous to speech in Section 4.2 and in Section 5 when explicating vertical relevance. Such projective rules implement predictive processing in interactions and therefore provide a computational underpinning of a central cognitive mechanism (Litwin & Miłkowski, 2020).

4. Polyphonic interaction: cognitive–formal analyses

The formal tools from Section 3 are used to provide precise analyses of the observations from Section 2: attention and communicative breakdown (Section 4.1) and the semantics of head shake (Section 4.2).

4.1. Conversational engagement

In two-person conversation, the values of spkr and addr of a DGB are rarely in question, apart from initially (Who is speaking? Are you addressing me?), but the need to verify that the addressing condition holds is what we take to drive attention monitoring. We conceive the two states of being engaged or disengaged in conversation as two hypotheses in a probabilistic Bayesian framework. Relevant data for the (dis-)engagement hypotheses can be found in gaze, which is an excellent predictor of conversational attention (Nummenmaa & Calder, 2009; Vertegaal et al., 2001). The quoted sources as well as the discussion in the following concern unobstructed face-to-face dialogue, that is, dialogue where participants stand or sit opposing each other and can freely talk. The findings and the assumptions derived below do not carry over to ‘obstructed’ discourse situations simpliciter, for instance, when interlocutors are talking while carrying a piano.

Within cognitive DGB modelling, the Vis-Sit field already provides an appropriate data structure for gaze. Mutual gaze can be formulated as a perspectival default condition on partiturs. ¹⁷ Of course, there is no claim that mutual gazing occurs continuously. Indeed, continuous gaze is often viewed as being rude or encroaching. In fact, mutual gaze tends to be short, often less than a second (Kendon, 1967).

Gaze is not the only attentional signalling system, however. Dialogue agents regularly provide verbal and non-verbal feedback signals (Bavelas & Gerwing, 2011). Among the verbal reactive tokens (Clancy et al., 1996) the majority are backchannels. As with gaze, a lack of backchannelling will result in communicative breakdown. In sum, there is ample evidence that gazing and backchannelling provide important datapoints for tracking (mutual) attention. We combine both into a probabilistic framework along the following lines:

We assume that gazing provides slightly more attentional evidence than backchannelling by a proportion of 0.6 to 0.4. We derive the prior probabilities for gaze under $ {H}_1 $ from Argyle (1988, p. 159), and the priors for gaze under $ {H}_2 $ are stipulated, as are the backchannel probabilities. Furthermore, we assume that engagement is the probabilistic default case of interaction with a plausibility of 0.8 to 0.2:

If one of the kinds of gaze from $ \mathcal{D} $ is observed, the posterior probability can be calculated from the probability tree in (13) by means of a Bayesian update according to Bayes’ theorem ( $ P\left(H|D\right)=\frac{P\left(D|H\right)P(H)}{P(D)} $ ). Let us illustrate an update triggered by an observation of individual gaze, $ {D}_1 $ . Compared to the prior probabilities of the engagement and disengagement hypotheses, $ {D}_1 $ leads to an increase of the probability of $ {H}_1 $ at the expense of $ {H}_2 $ . The corresponding numerical values are collected in Table 1.

Table 1. Bayesian update table

The change of the posteriors in comparison to the priors show that the already more probable engagement hypothesis gains further plausibility (increasing from 0.8 to 0.9). Hence, observing individual gaze, $ {D}_1 $ , supports (the public display of) mutual attention. Bayesian updates apply iteratively: In this way, only a mixture of data observations of different kinds leads to an oscillation of $ {H}_1 $ within a certain probability interval. This leads us to a testable hypothesis, namely that the extrema of the oscillation interval constitute thresholds of mutual attention. If the engagement posterior takes a value below the minimum of the interval, it triggers attention clarification: Are you with me? Values that exceed the maximum lead to irritation: Why are you staring at me?

4.2. Head shake and noetics

In Section 2, the exchange re-given in (14) was introduced as an obstacle for the No-semantics of head shakes introduced in Section 3.

If we make the (rather consensual) assumption, that the outcomes of utterances are predicted as soon as possible (see Section 3, in particular example (11)), then an explanation of (14) is straightforward: A’s utterance in (14a) provides a negative proposition, $ \neg $ believe(A,B), which by NoSem the head shake affirms. On the other hand, (14b) provides a positive proposition, believe(A,B), which by the same lexical entry the head shake negates, hence a contradiction ensues.

The contradiction in (14b) can be ameliorated, however:

In this case, one can understand A as verbally expressing his belief in B’s protestation of innocence, whereas the head shake affirms the negative proposition B makes, $ \neg $ stole(B,500€) (when related to the second sentence uttered by B), or expresses that A is upset about what ‘they’ did (when related to B’s initial uttered sentence). In either case, this requires us to assume that the head shake can be disassociated from speech that is simultaneous with it, an assumption argued for in some detail with respect to speech laughter by Mazzocconi et al. (2020/22). Such observations are of great importance for a multimodal theory. This is because it has been claimed that multi-tier interpretation is guided by the heuristic ‘if multiple signs occur simultaneously, take them as one’ (Enfield, 2009, p. 9). Such heuristics have to be refined in consideration of the above evidence. ¹⁸

Examples like the head shake in (15b) – which can be glossed ‘I disapprove of $ p $ ’ – are therefore subsumed to a ‘negative appraisal use’ of negation (Tian & Ginzburg, 2016) by Lücking and Ginzburg (2021), and analysed as a noetic act expressing a speaker’s attitude towards the content of his or her speech via DGB’s Mood field. ¹⁹ Note, finally, that A’s response in (15) can be serialised as head shake followed by speech (i.e., head shake + ‘I believe you’). However, the sequence ‘I believe you’ + head shake seems to be a bit odd, illustrating a remark concerning the multimodal serialisation hypothesis we made in Section 1, namely that sequential orderings need not be equivalent. Such temporal effects need to be explored further in future studies.

5. Upshot: from ‘horizontal’ to ‘vertical’ relevance in multimodal dialogue

In uni-modal interaction (best exemplified perhaps by chat conducted sequentially between users across a network), conversation is constrained by relevance or coherence between successive participant moves (and ultimately across longer stretches). For reasons related to our metaphor with musical notion (cf. partiturs), we call this notion horizontal relevance.

Some examples for relevant (indicated by ‘✓’) responses to a query and to an assertion are given in (16a,b) and irrelevant ones (indicated by ‘#’) to both in (16c).

For conversation, the query/response relation is the one studied in greatest detail (Berninger & Garvey, 1981; Ginzburg et al., 2022; Stivers & Enfield, 2010). The basic characterisation of this relationship given in Ginzburg et al. (2022) is that the class of responses to a question $ {q}_1 $ can be partitioned into three classes.

A formal account of horizontal relevance in terms of conversational rules is given in Ginzburg (2012, Sects. 4.4.5 and 6.7.1). The basic idea is that an utterance $ u $ is relevant in the current context iff $ u $ can be integrated as the (situational component of the) LatestMove via some conversational rule.

But how does the sequential notion of horizontal relevance relate to simultaneous interaction on partiturs, that is, to vertical relevance (to stick to the basic metaphor)? We believe that vertical relevance is supervenient on horizontal relevance. To the best of our knowledge, a careful study, either experimental or corpus-based, of vertical dialogical relevance has yet to be undertaken, apart from one subclass of cases involving speech, known as overlaps and interruptions, to which we return in our discussion below. We offer an initial, partial and impressionistic characterisation of the notion of vertical relevance in Table 2.

Table 2. Vertical relevance: possible content relationships between overlapping utterances across two speakers

Table 2 offers a selection of signals/contents that a non-leading voice $ B $ can express simultaneously relative to a leading voice $ A $ (speaking in terms of turn-replacements, not in terms of subjectively assumed importance; cf. Section 1). Note that two cases can be distinguished. The first case involves a single speaker for whom certain signals from the multimodal utterances may take the leading voice over other ones. A natural leading voice is speech (de Ruiter, 2004). Co-leading or accompanying roles of non-verbal signals can be assigned in relation to speech. In this respect, at-issue ( $ \approx $ co-leading) and non-at-issue ( $ \approx $ accompanying) uses of co-verbal manual gestures have been distinguished (Ebert, 2014).

The second case concerns the distribution of voices among several interlocutors. Inhabiting a leading or an accompanying role is rooted in processes of utterance projection (11) and incremental QUD construction, as we discuss in more formal detail below. We assume that the interlocutor who is responsible for publicly constructing the initial QUD – a process which (by the first case above) can be multimodal or even nonverbal itself – has/is the leading voice. We think that the classic notion of turn holder dissolves into the notion of leading voice. Accompanying voices are characterised by monitoring the incremental QUD construction and commenting on it – in ways exemplified in Table 2. In the most trivial case, this consists in providing backchannelling, but it may also involve the joint production of an utterance (in which case, it could be argued that the accompanying voice becomes a co-leading voice).

The final class we mention is one that has been, in certain respects, much studied, namely simultaneous speech. This is a somewhat controversial area because whereas the ‘normativity’ of one speaker using speech and another producing a non-verbal signal is not in question, the normativity of the corresponding case where both participants use speech is very much in question. This is so given the notion of turn and the rule-based system which interlocutors are postulated to follow in the highly influential account of Sacks et al. (1974). This system is based on the assumption that normatively at any given time there should be a single speaker; deviations are ‘performance errors’, either unintentional overlaps or one interlocutor interrupting, attempting to gain the floor. The set-up we have provided does not predict any sharp contrast between non-speech/speech overlap and speech/speech overlap, although this could in principle be enforced by introducing conversational rules privileging the speech tier. Nonetheless, we do not think such a strategy is promising. Rather, there are other explanatory factors which conspire to suppress pervasive overlap. In a study of the multilingual CallHome corpus, Yuan et al. (2007) note that overlapping varies across languages, with significantly more (non-backchannel) overlaps in Japanese than in the other languages they study (Arabic, English, German, Mandarin and Spanish); they also find that males and females make more overlaps when talking to females than to males, and similarly find more overlaps when talking with familiars than with strangers. Tannen (1984) argues for the existence of distinct conversational styles, including a high-involvement style that favours a fast delivery pace, cooperative overlaps and minimal gaps contrasting with a dichotomous high-considerateness style. Hilton (2018) conducted a study which found statistically significant correlations between a subject’s conversational style preference and their assessment of the acceptability of overlaps. All this argues against viewing avoidance of overlapping as a fundamental, systematic organising principle.

Can we say anything systematic based on subject matter about cases where overlap seems to be acceptable? There is no dearth of evidence for such cases going back to Bennett (1978), Falk (1980), Goodwin and Goodwin (1992) and indeed Schegloff (2000), who while defending the basic intuition underlying Sacks et al. (1974) list various cases of acceptable overlaps. We mention several subclasses: the first involves what we dub, following Goodwin and Goodwin (1992), shared situation assessments. Examples of this are given in (18a,b); in all three cases, a single situation is being described. A second class, noted by Schegloff (2000), is symmetric moves like greetings, partings and congratulations (“we won!” “Yay!” etc.). A third class is exemplified by the attested (18c) – cases where the same question is being addressed; additional instances of this, noted by Schegloff (2000), are utterances involving self-addressed questions (Tian et al., 2017) and ‘split utterances’ – utterances started by A and completed by B (Goodwin, 1979; Gregoromichelaki et al., 2011; Lerner, 1988; Poesio & Rieser, 2010).

Our assumption throughout has been that vertical relevance supervenes on horizontal relevance – what we labelled earlier the multimodal serialisation hypothesis. We adopt this assumption since, at least on the basis of Table 2, all polyphonic utterances seem to have sequential manifestations which give rise to equivalent contents; such cases, nonetheless, do lead to distinct DGBs since the partiturs in the two cases are distinct. On the other hand, we believe that there exist sequential adjacency pairs that do not have polyphonic manifestations which give rise to equivalent contents: turn-assigning moves, such as those arising by using the assignee’s name or via gaze, do not have a polyphonic equivalent.

Assuming supervenience to hold, we derive vertical relevance from conversational rules by applying incrementalisation – in other words, given two conversational rules CR $ {}_1 $ and CR $ {}_2 $ that can apply in sequence where $ A $ holds the turn as a consequence of CR $ {}_1 $ and this is exchanged in CR $ {}_2 $ , if by means of incremental interpretation B finds herself in a DGB applicable to CR $ {}_2 $ before the move taking place CR $ {}_1 $ is complete, an overlap arises. To make this concrete, A asserting $ p $ and B discussing whether $ p $ is the case can be explicated in terms of the sequence of Assert QUD-incrementation and QSPEC (see (10)). Incrementalising this involves B using Assert QUD-incrementation before A completed their utterance, which then satisfies the preconditions of QSPEC. In such a case, as discussed above, A is the ‘leading voice’ and B is an ‘accompanying voice’.

All this means that to the extent that the conversational rules underlying horizontal relevance ensure the coherence of dialogue, the same applies to dialogue with polyphonic utterances. Given this, incrementalising conversational rules provides a detailed model for coherence-driven, predictive processing in natural language interaction. In particular, it makes the testable prediction that accompanying behaviour commenting on a leading voice (examples of which are collected in Table 2) is expected to occur before the leading voice finished its contribution on its own.

6. Conclusions

We have outlined a unified framework for describing multimodal dialogical interaction. We show how minor adjustments to an existing dialogue framework, KoS, which provides richly structured cognitive states and conversational rules along with (i) partiturs, representations of multimodal events and (ii) an incremental semantic framework are needed to analyse multimodal phenomena.

• • We demonstrate the existence of noetic head shakes whose contents are dissociated from simultaneous speech. Such dissociation has been demonstrated in previous work for laughter.

• • We offer a testable, quantitative account of mutual gaze repair and backchannelling driven by monitoring of participant roles – not enough leading to clarification requests, too much leading to complaints.

• • We have argued that no overlap is not a defensible norm in multimodal interaction, including in cases where the two tiers involve speech. The intrinsically sequential notion of turn should be replaced by a notion such as leading/accompanying voice, which is driven by vertical coherence.

On the more basic level of theory design, the observations we discussed all exemplify the need for analytic semantic tools within the systemic landscape of cognitive science. We argued that a dynamic dialogue semantics incarnates a cognitively potent, formally precise linguistic framework for fertilising cross-talk between the disciplines.

As is frequently pointed out but cannot be overemphasized, an important goal of formalization in linguistics is to enable subsequent researchers to see the defects of an analysis as clearly as its merits; only then can progress be made efficiently. (Dowty, 1979, p. 322)

The issues of timing and coherence as captured in terms such as leading voice and vertical relevance have been identified as specific topics within multimodal dialogue semantics.