Bimodal code-mixing: Dutch spoken language elements in NGT discourse

Abstract

Mouthings, the spoken language elements in sign language discourse, are typically analysed as having a redundant, one-on-one relationship with manual signs, both semantically and temporally. We explore exceptions to this presupposed semantic and temporal congruency in a corpus of spontaneous signed conversation by deaf users of Sign Language of the Netherlands (NGT). We identify specifying mouthings (words with a different meaning than the co-occurring sign), solo mouthings (uttered while the hands are inactive) and added mouthings (words added to a signing stream without their corresponding sign), and make a sentence-level analysis of their occurrences. These non-redundant mouthings occurred in 12% of all utterances, and were made by almost all signers. We argue for the presence of a code-blending continuum for NGT, where NGT is the matrix language and spoken Dutch is blended in, in various degrees. We suggest expansion of existing code-mixing models, to allow for description of bimodal mixing.

1. Introduction

When speakers use multiple spoken languages within utterances in ordinary conversation – so-called intra-sentential code-mixing – they may mix those languages effortlessly. Speakers are also able to keep the two languages separate in everyday use, even though there is ample evidence that both languages are activated simultaneously in any sentence uttered by a bilingual speaker (Van Heuven & Dijkstra, 2010). Mixing spoken languages is a sequential, linear phenomenon: a bilingual speaker articulates one language at a time, inhibiting the other until the next moment s/he switches.

When one of the languages is a signed language, there is no need to inhibit either of the articulatory channels (Emmorey, Petrich & Gollan, 2012; Morford, Wilkinson, Villwock, Piñar & Kroll, 2011; Giezen, Blumenfeld, Shook, Marian & Emmorey, 2015). Bimodal code-mixing may thus throw a better light on the nature and ease of mixing, as inhibition plays less of a role.

Deaf communities in western societies are de facto bimodal bilingual, having been subjected to oral education for several generations while at the same time using signed language as their primary and preferred language. The lack of a need to inhibit words from the spoken language results in the occurrence of those spoken language elements while signing. These are called ‘mouthings’ (see the contributions to Boyes Braem & Sutton-Spence, 2001). Mouthings are generally viewed as having a one-on-one relationship with the manual sign they co-occur with. This relationship comprises a temporal component and a semantic component. A typical mouthing is roughly time-aligned with the manual sign, and carries approximately the same meaning as the manual sign. As a consequence, it conveys mainly redundant information (e.g., Bank, Crasborn & Van Hout, 2011, for NGT). The grammar of NGT sentences usually leads to sign order patterns different from the word order patterns in spoken Dutch. The manual signs comprise the primary information stream including the order pattern, with mouthings following the sign order and accompanying the signs as a secondary information stream: a bimodal form of code-mixing, or ‘code-blending’ (Emmorey, Borinstein & Thompson, 2005; Emmorey, Borinstein, Thompson & Gollan, 2008).

For unimodal, sequential code-mixing in spoken languages, Muysken (2013), in an update of his (2000) account, distinguishes four types of underlying processes:

• Insertion: the insertion of lexical material from L2 into an L1 sentence.

• Alternation: the succession of structures from either language in a sentence.

• Congruent lexicalization: the use of elements from either language in a grammatical structure that is shared by L1 and L2.

• Backflagging: insertion of heritage language (L1) discourse markers in L2 discourse.

Given that manual information in sign language is the primary information stream, insertion seems to match code-blending best, as it is defined as the occurrence of lexical material from one language into the matrix or base language (Muysken, 2013, p. 714). Bank et al. (2011) suggested the term ‘lexical addition’ for redundant mouthings, since there is no real switch from one language to another. Van den Bogaerde and Baker (2005; Baker & Van den Bogaerde, 2008), in a longitudinal study on code-mixing between deaf mothers and their deaf or hearing children, found that the “type of code-mixing process that primarily occurred is congruent lexicalization with just some lexical insertion” (2005, p. 172). Muysken's description of code-mixing, however, does not take into account that signed and spoken languages have the intrinsic capability of being expressed simultaneously in their different modalities: signing on the hands, and spoken language on the mouth. Cognitive limitations aside, there is in principle nothing to stop a signer from expressing two distinct grammars simultaneously. However, also in cases of code-blended signed and spoken language, one of the two languages serves as the matrix language (Muysken, 2000; Myers-Scotton, 2006), embedding material from the other language in its structure. In the case of NGT combined with Dutch mouthings (the default manifestation of NGT in everyday use, see Bank, Crasborn & Van Hout, 2016), NGT can be considered the matrix language, the elements from spoken Dutch (i.e., mouthings) being the embedded language or guest language.

Another type of mixing between NGT and Dutch is found in ‘sign-supported speech’ (Nederlands met Gebaren, NmG (“Dutch with Signs”)). Although there is no formal system of manually coded Dutch such as has been described for Signing Exact English in North America (Gustason, Pfetzing & Zawolkow, 1975), the use of spoken Dutch with supporting signs is very common. It is characterised as a contact variant for specific situations, most typically where the primary audience is hearing but manual lexical items from the sign language are mixed in to ensure that the deaf audience gets more perceptual input for the recognition of the spoken language sentences. Terpstra and Schermer (2006) describe NmG as a continuum between NGT and Dutch where three main forms can be distinguished. In each of these, Dutch grammar forms the basis, combined with elements of NGT to various degrees. At the Dutch end of the continuum, spoken content words are combined with signs, while at the NGT end of the sign-supported speech continuum, both vocabulary and grammatical elements from NGT are combined with Dutch sentences, occasionally leading to grammatically incorrect Dutch sentences. The hypothesis we explore in this paper is that in communicating with each other, native and near-native deaf signers mix Dutch and NGT in the inverse way: NGT grammar forms the starting point, and Dutch is mixed in to various degrees. This kind of mixing has been explored before. Schermer (1990), for instance, looked at how elements from spoken Dutch go together with NGT signing, and Van den Bogaerde (2000) studied interaction in deaf families with hearing children. Both studies found various degrees of Dutch mixed into NGT based utterances. The point we want to make here is that what we expect to find, in a sense, is the mirror image of what Terpstra and Schermer (2006) have described for NmG, leading to a form of language contact variety that would merit a new abbreviation ‘GmN’ (Gebaren met Nederlands, ‘speech-supported signing’). Note that whether this mixed-in Dutch is either voiced or unvoiced is irrelevant to a deaf interlocutor.

As indicated above, mouthings are usually roughly time-aligned and semantically congruent with the manual sign. Because of this temporal and semantic overlap they are called redundant mouthings. However, there are exceptions to both temporal alignment and semantic congruency. One such exception, in temporal alignment, is the spreading of mouthings over adjacent signs, so that one mouthing co-occurs with multiple signs. Bank, Crasborn and Van Hout (2015) found this to be a frequent phenomenon in NGT, confirming the initial observations of Crasborn, Van der Kooij, Waters, Woll and Mesch (2008) that most mouthings spread rightward (progressively) and encompass one adjacent sign. However, leftward (regressive) spreading occurs as well, and the scope of spreading was found to extend over up to five signs (Bank et al., 2015).

Another such exception, in semantic overlap, is the variation in the Dutch lexical items that may accompany a manual sign (Schermer, 1990). Bank et al. (2011), looking at twenty highly frequent signs in the Corpus NGT, found that almost all mouthings either denote the same meaning as the manual signs they accompany, or denote something very close to that (like good with prima (‘excellent’) instead of goed (‘good’), for instance).

The current paper aims to establish what happens when there is no neat semantic overlap between an NGT sign and a Dutch mouthing, or when Dutch lexical items occur in the manual signing stream without any manual counterpart. An example of the latter is given in (1). To give an idea of the temporal alignment of the left and right hand glosses and mouth annotations, the appendix contains screenshots of the annotations to the examples throughout this paper; links to the locations of the examples in the online Corpus NGT are listed in the footnotes.

1. (1)

   

In (1), there are two extra mouthings added to an NGT-sentence, one just before and the other co-occurring with the onset of the first occurrence of the sign oraal (‘oral’). The mouthing meer (‘more’) occurs during the transitional movement that the hands make between the sign opgroeien (‘grow-up’) and ‘oral’, and the mouthing met (‘with’) is made while the hands have already begun the articulation of ‘oral’, accompanied by its own mouthing, mond (‘mouth’) later on. ² The thing to note here is that during the transitional movement between the two signs, linguistic material on the mouth is being inserted into the information stream.

It would be hard to classify this construction as NmG, since we don't see a Dutch sentence here. On the other hand, the use of met (‘with’) is not common in NGT and is influenced by how Dutch prepositional phrases are constructed – although the phrase met mond ‘with mouth’ in Dutch would require an article to be well-formed (met de mond ‘with the mouth’). The repetition of the adjective ouwer (‘older’) is not impossible in Dutch, but would require a conjunction (ouwer en ouwer, ‘older and older’), and would never occur by itself but only in a full clause such as naarmate ze ouwer en ouwer worden (‘as they grow older and older’). This example raises the question whether other sentences with added or specifying mouthings are comparable to this example: a mix of NGT and Dutch that is more speech-supported sign than sign-supported speech.

The occurrence of these insertions of added mouthings suggests that the lexicons of NGT and Dutch may be combined in more complex ways than by directly linking individual signs and words, calling for a sentence-level analysis. We will investigate to what extent and in which way NGT and Dutch are combined to convey complex messages by the signers in the semi-spontaneous interactions in the Corpus NGT.

We will use the same corpus as the studies above (Bank et al., 2011, 2015, 2016) and categorise the various occurrences of added mouthings and semantic mismatches. We will distinguish three varieties of special mouthings in this paper:

1. 1) Solo mouthings are isolated words or short phrases that occur while the hands are in rest position or otherwise engaged, and thus constitute code-switching rather than code-blending. Solo mouthings are commonplace in the Corpus NGT (Nedela, 2013). They are most often used as a backchannel, a short feedback cue where an interlocutor mouths a short ja (‘yes’) or okee (‘okay’) to signal s/he is following the flow of the conversation but does not claim the turn. Since we are interested in mouthings on the sentence level and how they relate to the manual signs, we will only briefly take these backchannels into account in the current study, focusing for the most part on short strings of solo mouthings, and solo mouthings embedded in a sentence when the hands may temporarily have moved to a rest position. We know from informal observations that such solo mouthings are not restricted to backchannels. They can also take the form of full Dutch sentences or even short conversations, when a signer has her/his hands full and wants to start or continue a conversation.

2. 2) Added mouthings occur in the stream of mouth actions that accompanies the manual signing stream, but an added mouthing does not accompany any specific sign; rather, they are squeezed in between two mouthings, occur during transitional movements of the hands, or overlap a manual sign that also comes with its own mouth action. There is a continuum between two extremes on which added mouthings can occur. On one end of that continuum, a mouthing occurs between two signs each having their own mouth action, and the added mouthing occurs during the transitional movement between those signs, not overlapping either sign. On the other end, a mouthing completely overlaps with a sign that also is accompanied by its own mouth action, so two mouth actions are articulated during one sign. Compared to solo mouthings, the defining difference is manual activity (i.e., when the hands are signing or in transition we label it an added mouthing; when the hands are in rest position we label it a solo mouthing). We have no reason to think they fulfil a different function in the discourse.

3. 3) Specifying mouthings are (roughly) time-aligned with a manual sign (and thus constitute code-blends) but do not denote the same semantic concept. Rather, they specify the meaning of the manual sign or they add a distinctive meaning to it.

For each of the three categories, we aim to establish what their semantic and/or morphosyntactic contribution is at the sentence level.

2. Methodology

We analysed parts of the Corpus NGT (Crasborn & Zwitserlood, 2008; Crasborn, Zwitserlood & Ros, 2008). This corpus contains video data of 92 prelingually deaf signers who retell video clips and picture stories, and discuss issues related to deafness, education and sign language. The great majority of signers have been educated at minimally secondary school level, with Dutch as the primary language of instruction and NGT playing a secondary role at best. NGT was not a subject language in school for any of the signers, but started to make its way into education for the younger signers as a language of instruction. In most cases, the language of instruction was spoken Dutch (for the older generations) and spoken Dutch alternated with sign-supported speech for children educated from 1980 onwards (people younger than 30 in our data set). The Corpus NGT is the best effort to date to record a representative sample of NGT use of the core of the deaf community, consisting of born-deaf or early-deafened signers, most of whom have not grown up in deaf families. All signers were recorded in dyads where people were matched in generation and regional background; the members of each dyad knew each other well and frequently interacted with each other in daily life.

Annotation of the corpus is on-going; the third public release of Corpus NGT annotations, published in June 2015, contains over 145,000 manual glosses (both left and right hands), and over 13,000 transcriptions of mouth actions. ^{3 , 4} For the current paper, we used a subset of 40 video clips from the corpus that were fully annotated for hands and mouth, spanning 94 minutes of signed conversation from 40 signers. Signers’ ages range from 17 to 82 (mean = 44.35, median = 41). All mouth annotations have a separate time-dependent annotation containing the type of mouth action: ‘M’ for mouthings (further specified with ‘M-add’ for added mouthing, ‘M-spec’ for specifying mouthing, or ‘M-solo’ for solo mouthing); ‘A’ for adjectival/adverbial mouth gestures that specify the sign; ‘W’ for whole face movement, a global facial expression; ‘4’ for mouth-for-mouth, where the mouth performs the action the sign depicts (as in kiss or chew); ‘E’ for semantically empty mouth gestures. See Crasborn et al. (2008) for a detailed description of mouth action classification, and Crasborn and Bank (2014) for a full annotation scheme for mouth actions. We went back to our annotations to check whether all solo mouthings, added mouthings and specifying mouthings were correctly annotated, including their alignment with the manual glosses, and created new sentence-level annotations containing both the manual glosses and orthographic mouth transcriptions for those sentences containing added or specifying mouthings. That is, all signs and mouthings in a sentence were included in a single annotation, allowing us a quick overview of all relevant information. This resulted in 266 new sentence-level annotations. Further, we asked two deaf native signers who are fluent in Dutch to translate these sentences, with the instruction to not only focus on the meaning of the manual signs, but to take all visible head, body and mouth movement into account. Looking at the semantics of the whole utterance enabled us to establish the additional value of mouthings in an utterance. The information on the gloss, mouth, and translation tiers formed the basis for our analyses.

The examples we will discuss below are supported by screenshots of our annotations, included in the appendix. These screenshots give an idea of the temporal alignment of the manual glosses and the mouth annotations. Since there is no orthography or an easily accessible writing system for sign languages, we use ID-glosses for our annotations: labels that serve as unique identifiers for sign types (see Johnston (2008, 2010) for the introduction of the term). For usability reasons, the word used for a gloss is semantically related to the sign it represents, but they are not meant as translations of meaning. In the examples, we will give the English equivalents of the Dutch ID-glosses used in the corpus. We also collapse the left and right hand tiers into one line, as differences in handedness or the use of one- vs. two-handed signs are not relevant to the present study.

The glosses pt and pt:1 refer to a general pointing sign and a pointing sign to self, respectively. Fingerspelling is indicated with a hash sign (#), false starts are indicated with a tilde (~).

In the mouth annotations, the Dutch orthographic representations of the visible speech elements are reproduced in case of mouthings; mouth gestures are indicated by their type, ‘A’, ‘E’, ‘4’, or ‘W’. Although providing screenshots of the gloss alignments may be helpful and concise, they do not contain all relevant linguistic events present in the interaction (leaving out non-manual cues, for instance, such as raised eyebrows to mark questions). The reader is encouraged to follow the links provided in the footnotes, pointing to the relevant sections of the online version of the Corpus NGT.

3. Results

There are an estimated 2,066 sentences in our 40-signer, 94-minute sample. ⁵ This includes the 266 utterances with a non-redundant (i.e., added, specifying or solo) mouthing occurring in that utterance, sometimes with multiple occurrences per utterance. These 266 utterances, about 12% of the total of 2,066, are produced by 36 signers and contain 359 added mouthings, 39 specifying mouthings and 38 solo mouthings.

We will first show examples of redundant mouthings (section 3.1) and solo mouthings used in backchanneling (3.2), followed by multiple solo mouthings used in short strings (3.3), mouthings specifying semantic information in content words (3.4), the use of Dutch function words in added mouthings (3.5), and Dutch word order and Dutch idiomatic expressions (3.6).

3.1. Redundant mouthings

As stated in the introduction of this paper, there is usually a close semantic and temporal relationship between a mouthing and the manual sign it co-occurs with. This is neatly illustrated in (2), in which most signs are accompanied by a mouth action, typically mouthings, and these mouthings are time-aligned with and have the same meaning as the signs they accompany. Consequently, we call these mouthings ‘redundant mouthings’.

1. (2)

   

The three mouthings in this phrase are all redundant mouthings: the sign glossed as 3 is aligned with the mouthing drie (‘three’), jaar (‘year’) is aligned with jaar (‘year’), and auto (‘car’) is aligned with auto (‘car’). There is no mouthing accompanying pt:1 (the pointing-to-self between vroeger (‘past’) and ‘car’, and there are adjectival mouth gestures together with the signs glossed as ‘past’ and ongeluk (‘accident’).

3.2. Solo mouthings used in backchanneling

There are 211 annotations labelled as solo mouthing in our 94 minute sample, including the 38 that are contained in (short) strings of mouth actions (to be discussed in section 3.3). The remaining 173 solo mouthings occur isolated from any other mouth actions or manual signs. In most cases the solo mouthings are used as backchannels, short feedback cues from the interlocutor. Most often this is a simple ja (‘yes’, 32%) or nee (‘no’, 12%), used by the interlocutor to signal that s/he is paying attention to the signer. About 6% of the tokens were unintelligible, and the remaining 50% of isolated solo mouthings consists of single tokens of all kinds of words, often merely repeating what the signer having the floor just signed. A few examples of these tokens are december (‘December’), nodig (‘necessary’) and woon (‘live-in’), to name just a few. This is quite like the backchanneling behaviour in spoken languages (Duncan, 1974; McCarthy, 2002; Wong & Peters, 2007), and also not unlike manual backchanneling in sign languages (Mesch, Nilsson & Wallin, 2011). We did no in-depth analysis of solo mouthings used in backchanneling, we focussed instead on solo mouthings occurring in the mouth action stream, to be discussed next.

3.3. Solo mouthings used in short strings

The 38 solo mouthings are contained in 18 utterances. There are a few utterances comprising a string of solo mouthings only, as in (3), where there is no manual activity at all (so strictly speaking there is no code-mixing here).

1. (3)

   

The signer agrees here with what the other signer says, not as a backchannel but confirming that indeed the weather was good in the period under discussion. One of the reasons why the signer decides to use mouthings only and no signs may be that she has her hands folded around her knee. There a few more cases in our sample where a signer's hands are not directly available for signing because of folded arms or hands. Sometimes, however, signers do have their hands available, but still choose to use only mouthings for short sentences. Occasionally, utterances start out with mouthings only, and signing starts only halfway the sentence. Example (4) illustrates this, where also the end of the utterance is only mouthed. Both at the start and at the end of the utterance, the signer has her hands in rest position, in her lap.

1. (4)

   

In the case of isolated strings of solo mouthings, the matrix language is clearly spoken Dutch (such as in (3)). In (4), there is alternation between NGT and spoken Dutch: a clear case of code-switching midway an utterance. But generally, the matrix language remains NGT, with only one or two mouthed words before or after a signed NGT sentence. Occasionally, a solo mouthing occurs in the middle of a (compound) sentence, such as the conjunction in (5). The signer brings her hand to her lap, mouths maar (‘but’), and then continues signing.

1. (5)

   

The matrix language remains NGT in ten cases out of 18, and is clearly Dutch in three. There is a switch in matrix languages in two sentences, the matrix language in the remaining three is unclear. There are no cases where strings of solo mouthings cross clause boundaries. There are fourteen signers in our sample who make at least one solo mouthing, not including backchannels. Solo mouthings thus clearly form a part of communication, and they give a touch of Dutchness to the message.

3.4. Specifying semantic information in content words (specifying mouthings)

There are 39 specifying mouthings in our sample, in 31 utterances by 14 signers. In (6), there is a mouthing that specifies the sign contact (‘contact’), a two-handed sign that is made twice here (the second gloss of ‘contact’ on the left hand is still part of the first occurrence of ‘contact’). During the first occurrence, there is initially no mouth activity; then, the left hand briefly interrupts signing ‘contact’ in order to sign makkelijk (‘easy’) – aligned with a (reduced) redundant mouthing makkelijk (‘easy’) – spreading over the last part of ‘contact’. The second occurrence of ‘contact’ has an accompanying mouthing, but not the redundant mouthing contact. Instead, there is the specifying mouthing moeilijk (‘difficult’), adding meaning to the sign. The repetition of the sign here, with a different mouthing than accompanying the first occurrence, makes it clear the signer is contrasting two types of contact, avoiding the need to sign ‘difficult’ as well.

1. (6)

   

Example (7) shows a more condensed form of contrasting. The sign glossed as regio (‘region’) is a one-handed sign that is articulated two-handed here, thus contrasting two different locations in signing space, one with the right hand and one with the left hand. The mouthing verschil (‘difference’) further intensifies the contrasting of locations by the two hands. As explained in section 2 above, the W on the mouth type tier stands for a mouth gesture where the whole face is involved.

1. (7)

   

In (8), there is nothing to contrast, but a sign is modified by its mouthing to indicate something is re-occurring. The sign glossed as discussieren (‘dispute’) is combined with the mouthing altijd (‘always’), to mean ‘always having an argument’. Like in the previous examples, there is a lexical sign for the specifying mouthing used here (in this case ‘always’), but the signer rather chooses to combine a sign and a specifying mouthing instead of two signs. The second instance of altijd slightly modifies the short articulation of the sign every, and then spreads over the second instance of dispute.

1. (8)

   

Interestingly, the sign zien (‘see’) is aligned with the mouthing gezien (‘have_seen’); this is an instance of temporal inflection of a mouthed verb. There is no system for temporal inflection in NGT; use of perfect participles in mouthings is not common in our data set.

There are various word class combinations in mouth/sign constructions. While the specifying mouthings are adjectival or adverbial in the examples above, specifying a nominal sign, many other combinations are found in the sample. There are combinations of a mouthed noun with a pointing sign (like Amsterdam, or haar ‘hair’, both specifying a pointing sign); combinations of a mouthed noun with a signed verb (like tolk ‘interpreter’ with the verb sign); combinations of two nouns (like buurvrouw ‘(female)_neighbour’, specifying woman); combinations of a mouthed adjective with a signed verb (like doof ‘deaf’, specifying the verb meet). One problem with trying to unequivocally classify the combinations is that it is often unclear what the signed word class is (see Schwager & Zeshan, 2008, for discussion).

3.5. Use of Dutch words in added mouthings

As stated above, added mouthings are defined as mouthings not accompanying a specific sign, but are inserted between two signs during transitional movements, or occur with a sign before or after a redundant mouthing accompanying that sign (in which case the sign can be said to spread over a second mouthing, the opposite of what Bank et al. (2015) studied). Added mouthings differ from solo mouthings in that the hands are not in rest position during the added mouthing, but active articulating another sign or in transition between signs. We found 359 added mouthings in our set of 266 utterances. All word classes are present, but they vary substantially in how many different tokens are present per word type in that class. Table 1 summarises.

Table 1. Word classes occurring as added mouthings, their token and type counts, and ratios.

The three words most frequently used as added mouthings are the conjunction maar (‘but’, 43 tokens), the copular verb zijn (‘to_be’, 30 tokens) and the negation niet (‘not’, 26 tokens). We will discuss some examples from the five groups with the largest token/type ratio: conjunctions, prepositions, negations, copular verbs and wh-questions.

Conjunctions

The 70 conjunctions in our collection of added mouthings are predominantly coordinating ones. We counted 43 tokens of maar (‘but’), 20 tokens of of (‘or’), 4 times als (‘if’), twice en (‘and’), and one token of omdat (‘because’). For each of these, there is an equivalent in NGT. Although we do not know of studies that have looked at conjunction in NGT in any detail, our impression is that NGT tends to prefer to realise conjunctions by use of space (for instance, articulating signs side by side) or nonmanuals (for instance, using a prosody for conditionals similar to that for topics). Example (9) illustrates how a signer deploys two strategies for contrasting deaf versus hearing. One is dominance reversal (deaf and the preceding signs are signed with the left hand, hearing and the following signs are signed with the right hand), the other is the addition of the mouthing maar (‘but’).

1. (9)

   

Another example of the use of a conjunction is given in (10). Here, all signs (except for the final palm-up) are made with only the right hand, and the signer wants to contrast a baby's being deaf with not being deaf. This contrast is made non-manually by first nodding and then shaking the head, and also by a combination of the added mouthing of (‘or’) and the sign niet (‘not’). Note that a manual negation is present here, which is not necessarily always the case (see section on negation, below). For the duration of the annotation ‘not visible’, the signer had his hand between the camera and his face; it is likely that he pronounced the full verb ontdekken (‘discover’) or a reduced version of it.

1. (10)

   

Prepositions

There are 52 added mouthings that classify as prepositions, in order of decreasing occurrence: voor (‘in front of’), om (‘around’, ‘in order to’), op (‘on’), in (‘in’), met (‘with’), bij (‘near’), van (‘of’), te (‘to’), door (‘through’), naar (‘to’), vanaf (‘from’), and volgens (‘according to’). In NGT, most spatial prepositions can be expressed lexically, although there may well be a preference for expression using spatial grammar. This is yet to be investigated. Other spatial relations can be expressed using classifier constructions, or positioning objects in signing space. There are no lexical signs for non-spatial prepositions. In example (11), om means ‘in order to’. A correct Dutch sentence, however, would include the adverb te to make om te leren (‘in order to learn’). Also, a correct Dutch sentence would have a different word order and not repeat the wh-question, indicating NGT is the matrix language here.

1. (11)

   

Negation

There are 32 cases of mouthed negation without the presence of manual negation: 26 tokens of niet (‘not’), and 6 tokens of nee (‘no’). In all cases except one, non-manual negation was also present in the form of shaking the head (cf. Coerts, 1992). In (12), the mouthed negation is combined with a shaking of the head and a general purpose palm-up sign that has no intrinsic meaning of negation (recall that the hash sign (#) denotes fingerspelling).

1. (12)

   

Copular verbs

There are 31 copular verbs among our added mouthings: 30 instances of zijn (‘to_be’), and one instance of worden (‘to_become’). There is no equivalent of a copular verb in NGT. In (13), the mouthing semantically associated with pt:1 is ik (‘I’), although that is articulated before the sign in both cases, so the signs align with the copular verb ben (‘am’, first person singular of ‘to be’). It could be argued that the signer associates the mouthed word pair ik ben (‘I am’) with the sign pt:1, but the appearance of the copular verb here strongly suggests Dutch influence. However, the signer does not use a conjunction to connect the two parts of the sentence, and she uses an NGT word order in the second part (Dutch word order would be ik ben 17 jaar oud ‘I am 17 years old’). This indicates NGT is the matrix language here.

1. (13)

   

Just as with gezien (‘have_seen’) in (8), there are occurrences of perfect participles in our sample, but these are rare cases. There is one instance of geweest (‘have_been’) as an added mouthing, and one instance of geworden (‘have_become’). All other instances are either ben(t) (‘am/are’, first or second person singular, 17 cases, such as in example (13)) or is (‘is’, third person singular, nine cases).

Wh-question words

Among the 25 wh-question words are 16 instances of hoe (‘how’), 5 times wat (‘what’), 3 times waar (‘where’) and one instance of waarom (‘why’). All these words have manual equivalences in NGT (see Coerts, 1992), although they are sometimes used in different ways. Example (14) shows a (short) occurrence of ‘how’, articulated while the hands are moving towards the start position of oud (‘old’). This sentence would appear to follow Dutch grammar, the more natural expression in NGT being old how-many, with ‘pt son’ (or son pt) as a sentence topic.

1. (14)

   

3.6. Dutch word order and Dutch idiomatic expressions

The examples shown so far, presenting solo, specifying and added mouthings, describe the multiple techniques that signers have available to blend some Dutch into their signing. We wanted to know to which extent this results in producing grammatically correct Dutch sentences together with signing. We found 18 such well-formed sentences in our sample of 266, so it is the exception rather than the rule. Please note that the sentences we found are a by-product from our analysis of non-redundant mouthings; there may be a few more in our 94-minute sample, if these mouthed sentences have the same grammatical structure as the NGT sentence they occur with. Given the observations on differences in grammar between Dutch and NGT (Bos, 1990, 1993, 1995; Crasborn, Van der Kooij, Ros & De Hoop, 2009), we predict there will not be many such cases of ‘accidental’ overlap.

In (15), a copular verb and an adverb are added mouthings, and the sequence makes a well-formed Dutch sentence.

1. (15)

   

Finally, we found a few cases where Dutch idiom is used in a signed sentence, such as in example (16): the Dutch idiom sterk in je schoenen staan (‘being confident, credible’) is mouthed together with a quite literal signed translation, Dutch being the matrix language here. Multiword idiomatic expressions are very rare in signed languages (Johnston & Ferrara, 2012, on Auslan), and NGT is no exception; this example clearly stands out as a Dutch expression.

1. (16)

   

Just as many compounds in NGT are loan translations of Dutch, idiomatic expressions like these are probably best analysed as instances of code mixing or loan translation. Although this is clearly an underexplored territory for sign languages, we have no reason as yet to see a direct link to mouthings, and leave these to future research.

4. Discussion

We summarise our findings (4.1), discuss implications for psycholinguistic models of bimodal language processing (4.2), discuss implications for models of code mixing (4.3), and conclude with a brief discussion of the limitations of the present study and suggestions for further research (4.4).

4.1. Summary of our findings

We have analysed the various forms in which non-redundant mouthings occur in NGT dialogue between native signers with no hearing people around. We found that non-redundant mouthings occur in roughly 12% of all utterances in our sample. Most of these non-redundant mouthings are added mouthings: they do not occur with their manual correlate but are inserted during manual activity. All word classes are present; conjunctions, verbs, adverbs and prepositions form the largest classes. Solo mouthings, also occurring without a manual correlate but now with the hands in rest position, and specifying mouthings, that specify the meaning of a co-temporal manual sign, take up smaller parts in our sample. The use of non-redundant mouthings is a pervasive communication strategy, used by the great majority of signers in the sample (36 out of our 40 signers). It is important to note these are not speech errors: no communication problems were observed, and interlocutors did not ask for clarification. In many cases (e.g., the use of conjunctions, spatial prepositions, wh-question words, negation), there are native (non-loan) lexical ways in NGT to convey the same concept using manual signs, but signers use added mouthings instead. Moreover, when we take all sources together, we find that manual, mouthed and other non-manual information leads to semantically coherent utterances where signers exploit the possibilities of combining resources from the two languages. We cannot say anything more substantial about the grammaticality of the utterances we analysed and about the impact of spoken Dutch on NGT morphosyntax. Perhaps studies on morphosyntax could have been helpful, but they are scarce and not informative in that respect (Bos, 1990, 1993; Kimmelman, 2014).

We can speculate about why deaf signers use non-redundant mouthings when signing among each other. All signers from deaf communities can be regarded as bilingual (Bank, Crasborn & Van Hout, 2016), and it is safe to assume, given the enormous amount of mouthing present in NGT, that both spoken and signed languages are active in the signer's brain. We offer three non-competing hypotheses as to why signers do use mouthings:

1. (1) Efficiency. With two manual articulators at their disposal, signers are already able to convey meaning in a condensed form through the use of simultaneous constructions (Vermeerbergen, Leeson & Crasborn, 2007). Adding the mouth as a third articulator would broaden the range of possibilities and deliver more information in a compact way, such as happens with the use of specifying mouthings. This fits with the overall structure of sign languages, where in fact many articulators that are independent of the hands (features of the face and hand, torso movements) are recruited in concert in the creation of utterances (Nespor & Sandler, 1999; Sandler, 1999; Brentari & Crossley, 2002).

2. (2) Ease of articulation. Oral articulators are smaller and therefore easier to move than the rather large articulators like hands and arms (Crasborn, 2012), and their use would thus contribute to the minimisation of articulatory effort. Once both languages are active in the brain of the signer, it is easy to slip in a word or two without hampering the information stream (Emmorey et al., 2012). This could promote the use of added and solo mouthings. On the other hand, the perception of mouthings (speechreading) is clearly harder than the perception of manual signs, which may contribute to an explanation of their relatively restricted use.

3. (3) Creativity. While it does not happen all the time during everyday signing, signers do make creative use of the possibilities that multiple articulators give them, in the sense of not only producing redundant information. Making use of a third articulator to produce code-blends (Emmorey et al., 2005, 2008), combining sign language with elements from spoken language, is another possibility for creative language use. It broadens the available repertoire for formulating a message, and thus enriches the use of sign language.

In section 3.6 we looked at the occurrence of Dutch word order, and found only a few sentences that could be classified as having Dutch as the matrix language. However, we only looked at the sentence level; further research may investigate ‘mouthings as Dutch sentences’ more thoroughly. Do they perhaps occur more often than we have been able to establish in the present study? Moreover, an investigation at the constituent level may perhaps reveal more Dutch structures in NGT constituents.

While we did not look at the use of added mouthings across generations, there is a potential age difference here. The shift in deaf education from purely oral-based (i.e., spoken language education) to a more sign-orientated one (i.e., both signed and spoken language in the curriculum) has happened relatively recently in the Netherlands. This could result in older signers being more Dutch-oriented (and possibly using more added mouthings) than younger signers. Bank et al. (2016) found no age differences in the general use of mouthings in a large sample, but they did not specifically look at added mouthings.

4.2. Implications for bimodal bilingual processing

From the point of view of bilingual processing, our findings suggest there is more complex bilingual processing going on than simply the lexical co-activation of Dutch word forms with NGT signs, such as is likely taking place in the omnipresent use of redundant mouthings (see also Bank et al., 2016). The use of added mouthings, and especially the use of Dutch function words, suggests that Dutch morphosyntactic constructions or representations are activated in the production of NGT sentences. This leads to the production of a chain of Dutch words that sometimes fits nicely with the word order and morphology of NGT, and sometimes produces mixed utterances with elements such as copulas or non-spatial prepositions unknown in the NGT lexicon.

Almost all signers use non-redundant mouthings, suggesting they expect their interlocutors to process it. A perceiving signer maintains a relatively steady gaze toward the producing signer (Siple, 1978). More precisely, the perceiver gazes at the other signer's face, where beginning (L2) signers tend to fixate on (the area around) the mouth, and native signers tend to fixate on (the area around) the eyes (in ASL, Emmorey, Thompson & Colvin, 2009). However, ASL has a longstanding reputation of hardly featuring any mouthings (but see Nadolske & Rosenstock, 2007, who argue otherwise). It would be interesting to see whether native NGT signers fix their gaze more upon the mouth than the eyes (as ASL signers do), as this could indicate a difference in how the two languages are processed. This has yet to be investigated by eye-tracking studies.

4.3. Implications for models of code mixing

If we compare the combinations of words and signs found in our data with Muysken's (2013) taxonomy of code-mixing, we see some similarities and some differences. We found a few instances that could be argued to be covered by Muysken's alternation, where NGT structure was temporarily replaced with spoken Dutch structure (e.g., examples (3) and (4)). There were also a few cases of congruent lexicalisation, where the grammatical structures of both languages are shared (e.g., example (15)). Backflagging – the insertion of L1 discourse markers in L2 discourse – is a problematic concept in sign language because of the simultaneous realisation of L1 and L2. When we try to place the use of redundant mouthings in Muysken's taxonomy, insertion would seem to be the best fit. There is however no replacement of items from the matrix language, as would be the case when mixing two spoken languages. A better fitting term may therefore be ‘addition’, since the simultaneous articulation of NGT signs and Dutch mouthings does not constitute a switch from one language to another, on account of the bimodality of the utterances (see also Bank et al., 2011). The information stream in this ‘addition’ type of code-blending thus consist of combining signing and redundant mouthings. When we try to include added mouthings and solo mouthings in the picture, it may be tempting to classify them as cases of insertion since it is ‘extra’ material. However, they are inserted into the mouthing part of the information stream – no real insertion in terms of Muysken's taxonomy, because the mouthings are already the L2. It is the same as with redundant mouthings: added mouthings and solo mouthings leave the matrix language structure intact, they are added to the information stream. We therefore argue that added and solo mouthings should be classified as addition as well, since the matrix language remains NGT ²¹ . Another thing that makes added and solo mouthings different from insertion in spoken languages is that in the latter, the inserted elements are overwhelmingly nouns (Myers-Scotton, 2006), where we typically found copulas, conjunctions, prepositions, and negations: functional elements.

Muysken (2000, 2013) uses code-mixing as a neutral term to cover concepts like code-switching and (nonce-)borrowing. Essentially, all categories described in his model (insertion, alternation, congruent lexicalisation, backflagging) assume a single articulatory channel, resulting in a model where, in any given string, there is only one active language observable at a time. While the term code-blending partly resolves this issue by highlighting the simultaneity of (bimodal) utterances, it is too generic to classify the variety of bimodal code-mixing strategies that we observed in our data. We argue that adding ‘addition’ as a fifth strategy to Muysken's model would serve to resolve this issue.

Terpstra and Schermer (2006) discuss sign-supported speech (NmG) in terms of a number of variants on a continuum between NGT and NmG. These variants are not seen as three different language varieties, but as illustrations of types of variation that occur when NGT elements are integrated in Dutch spoken utterances. They are non-categorical, and overlapping. Based on the results of the present study, we propose that even the ‘purest’ form of NGT has similar language contact characteristics. Dutch elements are integrated in various ways and to various extents into the matrix language NGT. At one end of the continuum, there are cases where redundant mouthings are used in sentences showing no other influence of spoken Dutch. This would be the ‘pure’ NGT side of the continuum. Our data here confirm the findings of Schermer's (1990) initial investigation of mouthings in NGT. Moving along the continuum, we find constructions where NGT word order is still maintained, but Dutch spoken elements are also inserted between manual signs (added and solo mouthings) and semantically non-congruent Dutch words (specifying mouthings) are used with manual signs. At the other end, we find constructions where Dutch word order influences the NGT word order, or where strings of Dutch words are articulated forming Dutch phrases or sentences, without any manual signs (sequences of multiple solo mouthings). Just as for the NmG continuum, these variants have no special ideolectal or variant status, but they are meant to illustrate the types of code-blending, code-mixing, and code-switching occurring along the continuum.

Although the two continua appear to be mirror images of each other, together forming one single continuum, there is one key property that differentiates the two: the nearly omnipresent use of voice in the NmG continuum, and the near-absent use of voice (or ideolectal variation in this regard) in the NGT continuum. For this reason, we suggest that the two are indeed better seen as two separate scales. In the first, Dutch is the matrix language; in the second, NGT is the matrix language.

5. Conclusion

Non-redundant mouthings are non-peripheral in our NGT data set. Our study confirms earlier findings by Schermer (1990) and Van den Bogaerde and Baker (2005; Baker & Van den Bogaerde, 2008). These non-redundant mouthings occur in 12% of all utterances, and were made by almost all signers. For most if not all of the examples we found, the use of non-redundant Dutch mouthings offers the signer ways of expressing something that could also be expressed by manual signs alone or by non-manual expressions alone. Non-redundant mouthings are not conventionalized, but rather individual choices among expressive possibilities in two articulatory channels, signing and mouthing.

As we found in earlier studies, redundant Dutch mouthings form an omnipresent complement running parallel to more ‘native’ manual-visual forms. From the perspective of minimal articulatory effort, one would expect that users do not articulate redundant information at all. Redundant mouthings illustrate how strong the co-activation is of the two articulatory channels in NGT signers.

In a substantial minority of sentences in our data set, elements of spoken Dutch are used in a way that cannot be interpreted as redundant. The mouthings in question are not strictly linked to individual manual signs, but contribute their own meaning through a parallel articulatory channel. They illustrate that spoken and signed language structures interact and that mouthings can be mixed with signs into mixed utterances.

Dutch lexical elements, both as redundant and non-redundant mouthings, are a paramount feature of everyday NGT communication. Intensive mixing in the form of code-blending apparently is a viable mode of communication.