Hostname: page-component-78c5997874-m6dg7 Total loading time: 0 Render date: 2024-11-10T15:21:17.533Z Has data issue: false hasContentIssue false

Cross-linguistic interactions across modalities: Effects of the oral language on sign production

Published online by Cambridge University Press:  21 April 2021

Marc Gimeno-Martínez*
Affiliation:
Center for Brain and Cognition (CBC), Universitat Pompeu Fabra, Barcelona, Spain
Andreas Mädebach
Affiliation:
Center for Brain and Cognition (CBC), Universitat Pompeu Fabra, Barcelona, Spain
Cristina Baus
Affiliation:
Center for Brain and Cognition (CBC), Universitat Pompeu Fabra, Barcelona, Spain Department of Cognition, Development and Educational Psychology, University of Barcelona, Barcelona, Spain
*
Address for correspondence: Marc Gimeno-Martínez, Universitat Pompeu Fabra, Ramon Trías Fargas, 25-27 08005, Barcelona. marc.gimenom@upf.edu
Rights & Permissions [Opens in a new window]

Abstract

To investigate cross-linguistic interactions in bimodal bilingual production, behavioural and electrophysiological measures (ERPs) were recorded from 24 deaf bimodal bilinguals while naming pictures in Catalan Sign Language (LSC). Two tasks were employed, a picture-word interference and a picture-picture interference task. Cross-linguistic effects were explored via distractors that were either semantically related to the target picture, to the phonology/orthography of the Spanish name of the target picture, or were unrelated. No semantic effects were observed in sign latencies, but ERPs differed between semantically related and unrelated distractors. For the form-related manipulation, a facilitation effect was observed both behaviourally and at the ERP level. Importantly, these effects were not influenced by the type of distractor (word/picture) presented providing the first piece of evidence that deaf bimodal bilinguals are sensitive to oral language in sign production. Implications for models of cross-linguistic interactions in bimodal bilinguals are discussed.

Type
Research Article
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.
Copyright
Copyright © The Author(s), 2021. Published by Cambridge University Press

1. Introduction

A well-established phenomenon in the literature is that bilinguals cannot restrict lexicalization to one of their languages. While speaking, listening, or reading, bilinguals’ two languages are simultaneously and automatically activated, revealing that lexical access in bilinguals is largely language non-selective (for discussion see, e.g., Kroll, Bogulski & McClain, Reference Kroll, Bogulski and McClain2012). Evidence of cross-linguistic interactions comes from studies on word comprehension (Marian & Spivey, Reference Marian and Spivey2003b, Reference Marian and Spivey2003a; Wu & Thierry, Reference Wu and Thierry2010) and word production (Colomé & Miozzo, Reference Colomé and Miozzo2010; Hermans, Bongaerts, De Bot & Schreuder, Reference Hermans, Bongaerts, De Bot and Schreuder1998), showing that activation of the non-intended language influences processing in bilingual's intended language.

Critically, the non-selective nature of bilingual lexical activation has also been shown in bilinguals with two languages of different modality (oral and sign), termed “bimodal bilinguals”. A number of experiments have showed that deaf and hearing bimodal bilinguals activate sign properties when processing words (Kubus, Villwock, Morford & Rathmann, Reference Kubus, Villwock, Morford and Rathmann2015; Morford, Kroll, Piñar & Wilkinson, Reference Morford, Kroll, Piñar and Wilkinson2014; Morford, Occhino-Kehoe, Piñar, Wilkinson & Kroll, Reference Morford, Occhino-Kehoe, Piñar, Wilkinson and Kroll2017; Morford, Wilkinson, Villwock, Piñar & Kroll, Reference Morford, Wilkinson, Villwock, Piñar and Kroll2011; Shook & Marian, Reference Shook and Marian2012; Villameriel, Dias, Costello & Carreiras, Reference Villameriel, Dias, Costello and Carreiras2016). For example, Morford et al. (Reference Morford, Wilkinson, Villwock, Piñar and Kroll2011) showed that phonological relationships in American Sign LanguageFootnote 1 (ASL) influenced semantic similarity judgements of written word pairs in English (see also Villameriel et al., Reference Villameriel, Dias, Costello and Carreiras2016, for similar results with hearing bimodal bilinguals and Morford et al., Reference Morford, Kroll, Piñar and Wilkinson2014, for a different result with hearing bimodal bilinguals).

Much scarcer is the evidence showing cross-linguistic influences of words on sign processing (Emmorey, Mott, Meade, Holcomb & Midgley, Reference Emmorey, Mott, Meade, Holcomb and Midgley2020; Giezen & Emmorey, Reference Giezen and Emmorey2016; Hosemann, Mani, Herrmann, Steinbach & Altvater-Mackensen, Reference Hosemann, Mani, Herrmann, Steinbach and Altvater-Mackensen2020; Lee, Meade, Midgley, Holcomb & Emmorey, Reference Lee, Meade, Midgley, Holcomb and Emmorey2019). Using the same paradigm as Morford et al. (Reference Morford, Wilkinson, Villwock, Piñar and Kroll2011), Lee et al. (Reference Lee, Meade, Midgley, Holcomb and Emmorey2019) showed that hearing bimodal bilinguals were sensitive to the phonological relationship of the English-translations (i.e., rhymed) while judging the semantic relationship of ASL sign pairs. Relevant here, results were not replicated in the deaf group, unless deaf individuals were aware of the English phonological manipulation.

While those results help demonstrate that cross-linguistic interactions in bilinguals are not modality-specific, they are also suggestive that, as in unimodal bilingualism, cross-linguistic interactions are not a ubiquitous phenomenon. At least two factors should be considered when exploring cross-linguistic interactions in bilinguals. The first relates to language dominance and proficiency. Cross-linguistic influences from L2 to L1 are weaker than the reverse and they only occur when sufficient proficiency in L2 has been attained (Van Hell & Tanner, Reference Van Hell and Tanner2012). Because most deaf signers are more dominant and proficient in sign language than in the oral language, this imbalance between languages could explain the lack of L2 (spoken) influence on L1 sign comprehension for deaf bilinguals in Lee et al. (Reference Lee, Meade, Midgley, Holcomb and Emmorey2019). The second factor, and more specific to bimodal bilingualism, relates to the mechanisms of phonological activation of the oral language, which might be different between hearing and deaf bilinguals. Because deaf bimodal bilinguals acquire the oral (L2) language via the written form (e.g., also referred as sign-print bilinguals; Piñar, Dussias & Morford, Reference Piñar, Dussias and Morford2011), phonological effects could be enlarged when spoken phonology is directly induced by the written language or, as showed in Lee et al. (Reference Lee, Meade, Midgley, Holcomb and Emmorey2019), in those deaf bilinguals with higher phonological awareness of the oral language.

Keeping these factors in mind, in the present study we tested deaf bimodal bilinguals during sign language production to further characterize cross-linguistic effects of the oral L2 on the sign L1 language. Before describing our study, cross-linguistic effects in language production and theoretical models of bilingual lexical selection are described.

1.1. Cross-linguistic effects in language production

In language production, most of the evidence on cross-linguistic interactions comes from studies using interference tasks. In these tasks, both a picture and a distractor are presented; participants are then asked to name the picture while ignoring the distractor. Experimental manipulations of the relationship between the distractor and the picture have been studied to inform models of bilingual language production (for a review see Hall, Reference Hall2011). In particular, semantic and phonological effectsFootnote 2 in picture-interference tasks have produced a fruitful debate concerning the role of competition in bilingual lexical selection.

Considering semantic effects, semantically related distractor words in the non-intended language (e.g., distractor: gato (cat) – target: DOG) elicit semantic interference (slower naming latencies and more errors in the semantically related condition than in the unrelated condition; Costa & Caramazza, Reference Costa and Caramazza1999; Costa, Colomé, Gómez & Sebastián-Gallés, Reference Costa, Colomé, Gómez and Sebastián-Gallés2003; Hermans et al., Reference Hermans, Bongaerts, De Bot and Schreuder1998). Models of bilingual language production have explained these semantic interference effects as a result of lexical competition between the two languages (Hermans, Reference Hermans2004; Hermans et al., Reference Hermans, Bongaerts, De Bot and Schreuder1998) or within the intended language (Costa et al., Reference Costa, Colomé, Gómez and Sebastián-Gallés2003; Roelofs, Piai, Garrido Rodriguez & Chwilla, Reference Roelofs, Piai, Garrido Rodriguez and Chwilla2016). Even though both views propose that interference is based on conflict at the lexical level, between-language competition assumes that lexical selection is accomplished through competition of all activated candidates regardless of language, and within-language competition assumes that distractors are automatically translated and then competition only occurs among lexical candidates in the target language. Alternatively, models assuming non-competitive lexical selection (Response Exclusion Hypothesis; Mahon, Costa, Peterson, Vargas & Caramazza, Reference Mahon, Costa, Peterson, Vargas and Caramazza2007) explain semantic interference effects as arising post-lexically, from control processes operating just prior to articulation.

At the ERP level, most of the picture-word interference studies have reported N400-like modulations of the semantic effect. Less negative ERP for pictures presented with semantically-related words (relative to semantically-unrelated words) have been taken as an index of semantic priming in competitive and non-competitive accounts of speech production alike (Blackford, Holcomb, Grainger & Kuperberg, Reference Blackford, Holcomb, Grainger and Kuperberg2012; Dell'Acqua et al., Reference Dell'Acqua, Sessa, Peressotti, Mulatti, Navarrete and Grainger2010; Koester & Schiller, Reference Koester and Schiller2008; Piai, Roelofs, Jensen, Schoffelen & Bonnefond, Reference Piai, Roelofs, Jensen, Schoffelen and Bonnefond2014; Roelofs et al., Reference Roelofs, Piai, Garrido Rodriguez and Chwilla2016; Zhu, Damian & Zhang, Reference Zhu, Damian and Zhang2015). Importantly here, differences between models arise in the predicted timing of the semantic interference effects. While lexical competition models predict a semantic effect during lexical selection, the response exclusion account predicts a later effect, much closer to the speech onset. Considering the time estimates of lexical selection in simple picture naming, which occurs at around 200 ms (e.g., Costa, Strijkers, Martin & Thierry, Reference Costa, Strijkers, Martin and Thierry2009), ERP modulations occurring at ~200 ms have been taken as competition occurring at the lexical level, therefore supporting lexical-competition models. Conversely, semantic effects starting at ~400 ms may imply a post-lexical locus, supporting non-competitive lexical selection models. However, it is not trivial to map time course estimates from simple picture naming to the picture-word interference task (in which naming latencies are prolonged). In addition, the electrophysiological literature does not show a consistent picture regarding the time course of semantic distractor effects, with studies showing earlier (200–500 ms; Aristei, Melinger & Rahman, Reference Aristei, Melinger and Rahman2011; Hoshino & Thierry, Reference Hoshino and Thierry2011) and later effects (325–600 ms; Blackford et al., Reference Blackford, Holcomb, Grainger and Kuperberg2012), making it difficult to localize the origin of semantic effects during speech production.

Relative to unimodal bilingualism, studies with bimodal bilinguals seem to favour predictions of the response exclusion hypothesis. Giezen and Emmorey (Reference Giezen and Emmorey2016) and Emmorey et al. (Reference Emmorey, Mott, Meade, Holcomb and Midgley2020) found no behavioural semantic interference effects in the picture-word interference task when hearing or deaf bimodal bilinguals were signing pictures in the presence of auditory or written distractors. According to the response exclusion account, semantic interference effects are not predicted in sign language because there should be no post-lexical conflict between signs and word responses (Emmorey et al., Reference Emmorey, Mott, Meade, Holcomb and Midgley2020; Giezen & Emmorey, Reference Giezen and Emmorey2016). Conversely, a semantic facilitatory effect is predicted as a result of activation of the semantic properties of the picture caused by a semantically related word prime. Indeed, Emmorey et al. (Reference Emmorey, Mott, Meade, Holcomb and Midgley2020), obtained a semantic facilitation effect supporting this prediction (cf. Giezen & Emmorey, Reference Giezen and Emmorey2016). In addition, consistent with a semantic priming effect, they observed an ERP modulation starting around 300 ms, with pictures paired with semantically related words showing an early N400-like attenuation. Interestingly, semantic effects in Emmorey et al. (Reference Emmorey, Mott, Meade, Holcomb and Midgley2020) matched the timing obtained in Baus and Costa (Reference Baus and Costa2015) when lexical variables (i.e., lexical frequency, iconicity) were manipulated in a picture signing task, which reinforces the idea that language interactivity is modality invariant and has a lexical origin (Shook & Marian, Reference Shook and Marian2012).

Here we further explored behavioural and ERP correlates of semantic processing in deaf bimodal bilinguals. If as described by the response exclusion account there is no competition at the articulatory buffer between signs and words (Emmorey et al., Reference Emmorey, Mott, Meade, Holcomb and Midgley2020; Giezen & Emmorey, Reference Giezen and Emmorey2016), then we should obtain a facilitatory effect of semantic relatedness behaviourally and a semantic priming effect at the ERP level. Note however that even if signs and words do not compete (at lexical or articulatory levels) semantic effects could be also expected as a result of within-(oral) language competition. Unlike unimodal bilinguals, bimodal bilinguals can produce signs and words at the same time (i.e., code-blending) because sign and oral languages use different motor systems. In this context, lexical competition might occur among candidates of the non-intended language (oral modality), which might end up affecting how words and signs are synchronized to produce a code-blend sign (Hosemann et al., Reference Hosemann, Mani, Herrmann, Steinbach and Altvater-Mackensen2020; Vinson, Thompson, Skinner, Fox & Vigliocco, Reference Vinson, Thompson, Skinner, Fox and Vigliocco2010). We return to this issue in the discussion.

Experiments with phonological distractor manipulations have broadly shown that distractors in the non-intended language which are phonologically related to the target facilitate picture naming (e.g., distractor: dos (two) – target: DOG; Colomé & Miozzo, Reference Colomé and Miozzo2010; Costa et al., Reference Costa, Colomé, Gómez and Sebastián-Gallés2003; Costa, Miozzo & Caramazza, Reference Costa, Miozzo and Caramazza1999; Hermans et al., Reference Hermans, Bongaerts, De Bot and Schreuder1998). Conversely, distractors in the non-intended language which are phonologically related to the target's translation slow down picture naming (e.g., distractor: pera (pear) - target: DOG (perro in Spanish); Boukadi, Davies & Wilson, Reference Boukadi, Davies and Wilson2015; Costa et al., Reference Costa, Colomé, Gómez and Sebastián-Gallés2003; Hermans et al., Reference Hermans, Bongaerts, De Bot and Schreuder1998; Hoshino & Thierry, Reference Hoshino and Thierry2011; Knupsky & Amrhein, Reference Knupsky and Amrhein2007; but see Costa, Albareda & Santesteban, Reference Costa, Albareda and Santesteban2008). Similar to semantic effects, phonological interference effects have been attributed to competition at the lexical level in models assuming between-language competition (Hermans, Reference Hermans2004) and to competition at the phonological level by models assuming within-language lexical competition (Costa et al., Reference Costa, Colomé, Gómez and Sebastián-Gallés2003; Roelofs et al., Reference Roelofs, Piai, Garrido Rodriguez and Chwilla2016).

At the ERP level, Hoshino and Thierry (Reference Hoshino and Thierry2011) showed similar semantic and phono-translation effects in a picture-word interference task. The behavioural interference occurred in the presence of reduced negativities for the semantic and phono-translation conditions relative to the unrelated condition. Both semantic and phono-translation effects elicited ERP modulations in two time windows (at around 200 ms and 350 ms respectively), which were interpreted as evidence of cross-language competition at the lexical level and beyond.

To the best of our knowledge, cross-modal phonological effects through the oral language in picture-word interference have not been tested in bimodal bilinguals. Different directions of the phonological effect could be expected depending on within or between-language competition views. Following within-language competition views (Roelofs et al., Reference Roelofs, Piai, Garrido Rodriguez and Chwilla2016) interference could only occur at the phonological level which is shared across oral languages. Because conflict at the phonological level should not exist between sign and oral languages, facilitation should be observed due to priming of the translation-equivalent in the non-intended language. Note that the same result would be predicted in code-blending production, when mouthing is activated and articulated together with the sign. In contrast, if phonological interference effects are observed, as have been found in unimodal picture-word interference studies, this finding would support between-language competition views, where interference occurs at the lexical level (Hall, Reference Hall2011). It should be noted that the response exclusion account has not been described to account for phonological effects in bilingual production. One tentative prediction for bimodal production could be that, since language membership is a response-relevant feature and no competition needs to be solved at the articulatory level, the phonology of the oral language is irrelevant and easily disregarded. In consequence, there should not be phonological influence from the oral language while signing.

1.2. The present study

In the present study, we explored cross-linguistic interactions in sign production by testing deaf bimodal bilinguals in two tasks, a picture-word and a picture-picture interference task. Comparing performance in two different interference tasks allowed us to examine whether cross-linguistic effects require the oral language to be directly activated by the (written) distractors in the task.

Semantic and phono-translation effects and their locus during sign production were evaluated, allowing us to test behavioural and electrophysiological traces of lexical selection processes in bimodal sign production. For example, the picture of a DOG (perro in Spanish) was presented with the distractor word or the distractor picture “gato” (cat in English; semantic condition), the word/picture “pera” (pear in English; form-related condition), or the word/picture “casa” (house in English; unrelated condition) superimposed on the target picture.

To explore cross-linguistic semantic effects (the contrast between the semantic and the unrelated condition) predictions necessarily must be put forward in the context of the picture-word inference task, given that results in the picture-picture interference task would not be informative regarding the involvement of the oral language in the task. A semantic effect in the picture-word interference task would demonstrate that activation of the oral lexicon (induced by the distractor word) influences sign production. If our results support the non-competitive nature of lexical selection, cross-modal co-activation should result in facilitation due to semantic priming because there should be no post-lexical conflict between sign and oral languages (for discussion see Emmorey et al., Reference Emmorey, Mott, Meade, Holcomb and Midgley2020; Giezen & Emmorey, Reference Giezen and Emmorey2016; Mahon et al., Reference Mahon, Costa, Peterson, Vargas and Caramazza2007).

The contrast between the form-related and the unrelated condition has the potential to reveal more about cross-linguistic interactions across modalities. Any differential effect of a distractor that is form-related to the Spanish name of the target picture would imply that Spanish was activated during LSC sign production and that the Spanish lexicon influenced sign production. In addition, if the phonological effects do not differ between tasks, it would suggest that these effects are not driven by the explicit presence of the Spanish language in the task.

2. Methods

2.1. Participants

Twenty-four deaf LSC-Spanish bilinguals (12 females, M age = 34.5 years, SD = 14.2 years) participated in the study. Twenty-two participants had profound hearing loss (91-120 dB), one participant had a severe hearing loss (71-90 dB), and one participant had a moderately-severe hearing loss (56-70 dB). Four participants reported using hearing aids, one reported the use of cochlear implants and nineteen participants did not use any type of hearing device. One additional participant was run, but excluded due to an excessive number of artefacts. All participants reported normal or corrected vision and no history of neurological problems. Self-ratings of LSC and Spanish proficiency were collected through a language background questionnaire (Table 1). All participants completed an informed consent form before the experiment and were paid for their participation.

Table 1. Demographic information of the participants. Mean ratings (M) and standard deviation (SD)

* Self-ratings from a language questionnaire; proficiency was rated on a 10-point scale ranging from ‘almost none’ to ‘very proficient’

2.2. Materials

A set of thirty pictures and a separate set of ninety picturable words were selected as targets and distractors, respectively, from different databases (Bates, D'Amico, Jacobsen, Székely, Andonova, Devescovi, Herron, Lu, Pechmann, Pléh, Wicha, Federmeier, Gerdjikova, Gutierrez, Hung, Hsu, Iyer, Kohnert, Mehotcheva, Orozco-Figueroa, Tzeng & Tzeng, Reference Bates, D'Amico, Jacobsen, Székely, Andonova, Devescovi, Herron, Lu, Pechmann, Pléh, Wicha, Federmeier, Gerdjikova, Gutierrez, Hung, Hsu, Iyer, Kohnert, Mehotcheva, Orozco-Figueroa, Tzeng and Tzeng2003; Snodgrass & Vanderwart, Reference Snodgrass and Vanderwart1980). In the picture-picture interference task, the stimuli consisted of two overlapping pictures, with targets in green and distractors in red. In the picture-word interference task, stimuli consisted of target pictures with a written superimposed word. In both tasks, each picture was paired with three different distractors (see table S1 in online supplementary material) and these distractor-set pairings were the same in both tasks. In the form-related condition, distractors were phonologically and orthographically similar to the Spanish name of the targets (e.g., CEREZA-cerebro; ‘cherry-brain’ in English), with phono-translation distractors and targets overlapping on 3 phonemes/letters on average (SD = 0.86; e.g., PINcel – PINgüino, ‘brush - penguin’ in English) and, in most cases, corresponding in their first syllables. Due to the nature of the Spanish language as a language with transparent orthography, materials selected based on phonological relations in Spanish were also mainly orthographically related. Thus, we refer to this condition as form-related condition. In the semantically related condition, distractors were from the same semantic field but were not form-related (e.g., CEREZA-manzana; ‘cherry-apple’ in English). A set of unrelated distractors were selected as the baseline condition (e.g., CEREZA-llave; ‘cherry-key’ in English). Targets and distractors were always phonologically unrelated in LSC and did not have obligatory mouth patterns as an intrinsic component of the sign. Furthermore, Spanish names for the distractor pictures were matched across conditions in number of phonemes/letters, lexical frequency, concreteness, and familiarity from the Spanish corpus B-Pal (Davis & Perea, Reference Davis and Perea2005) (see table S2 in online Supplementary Material).

2.3. Procedure

Participants were tested individually in an electrically shielded and dimly lit room. Instructions and other communication during the experiment were given in LSC by a hearing proficient signer. The order of the two tasks, picture-picture interference and picture-word interference, was counterbalanced across participantsFootnote 3. In each task, stimuli were presented in two blocks of 45 trials, and each task began with a practice block of three warm-up trials. E-Prime 2.0 ® was used to present the stimuli and record signing latencies. At the beginning of each trial, an instructional message asked participants to press and hold the spacebar to start the trial. Then, a 500 ms black screen was followed by a 500 ms central fixation cross and a 300 ms black screen. Target-distractor pairs were then displayed and maintained until participants released the spacebar in order to sign the name of each target picture. A final 500 ms black screen appeared at the end of each trial. Signing latencies were calculated from the onset of the stimuli display until the key release (see Baus & Costa, Reference Baus and Costa2015; Giezen & Emmorey, Reference Giezen and Emmorey2016; for the same method). Participants’ responses were recorded on video and checked for accuracy after the experiment ended. In addition to the signed responses, possible mouth movements elicited during sign production were checked by a hearing non-signer researcher.

2.4. Behavioural Analysis

Two target-distractor pairs were removed from all the analyses reported. One because participants reported a sign from the same semantic field instead of the desired sign (‘boat’ instead of ‘sailboat’), and the other because participants used the same signs adding mouthing to disambiguate between them instead of different signs (‘hair comb’ and ‘brush’).

We analysed the data by fitting linear mixed models, treating participants and items as crossed random factors (Baayen, Davidson & Bates, Reference Baayen, Davidson and Bates2008). Models were fitted in R (R Core Team, 2019) using the package lme4 (Bates, Mächler, Bolker & Walker, Reference Bates, Mächler, Bolker and Walker2015). Signing latencies where fitted with linear mixed models and error rates with generalized mixed models (binomial family). Models included fixed effects for task (sum coded), condition (treatment coded, unrelated condition as baseline), and their interaction. Significance of the fixed effects estimates was determined using the Satterthwaite approximation for degrees of freedom provided by the lmerTest package (Kuznetsova, Brockhoff & Christensen, Reference Kuznetsova, Brockhoff and Christensen2015). Additional analyses on log transformed latencies (to alleviate problems related to non-normality) as well as additional analyses including fixed effects for task sequence, mouthingFootnote 4, and its interactions with the other fixed effects lead to the same conclusions as the latency analyses reported here.

We aimed to fit models with the maximal possible random-effects structure (Barr, Levy, Scheepers & Tily, Reference Barr, Levy, Scheepers and Tily2013). We started out with a maximal model containing random slopes for distractor condition, task, and their interaction for both participants and items. In cases of non-convergence, we step-wise simplified the random structure, by dropping random correlations and the interaction terms before dropping main effect slopes from the model. In case of singular model fits, we first dropped the interaction terms before dropping condition or task slopes with an estimated variance (close to) zero.

2.5. EEG recording and analysis

EEG activity was continuously recorded from 30 Ag-AgCl electrodes, mounted on an elastic cap (ActiCap, Munich, Germany) and positioned according to the international 10-20 system. EEG data was recorded online to a common reference located at electrode site FCz. Eye movements and blinks were monitored with two electrodes placed below the right eye and at the outer canthus of the left eye. EEG data was sampled at 500 Hz with a bandpass of the hardware filter of 0.1–125 Hz.

Offline EEG data processing was carried out using the EEGLAB (Delorme & Makeig, Reference Delorme and Makeig2004) and ERPLAB (Lopez-Calderon & Luck, Reference Lopez-Calderon and Luck2014) MATLAB toolboxes. Signals were filtered offline with a bandpass filter of 0.1-30 Hz and re-referenced to the average activity of the two mastoids. Artefacts were corrected by means of an independent component analysis (Extended RunICA, 30 components). ERPs were computed offline for each participant in each condition, time-locked to the onset of the target stimuli presentation, relative to a 100 ms pre-stimulus baseline and until 750 ms post-stimulus onset. Epochs with amplitudes above or below 100μV or with a difference between the maximum and the minimum amplitude exceeding 75μV were considered artefacts and discarded from the analysis. One participant with an excessive number of artefacts (36% of trials) was discarded from the analysis.

Mean amplitudes for seven post-target onset latency windows were submitted to repeated-measures ANOVAs. ERPs analysis were analysed every 100 ms in order to cover early and late components: 50-150 ms, 150-250 ms, 250-350 ms, 350-450 ms, 450-550 ms, 550-650 ms, and 650-750 ms. The factors included in the analysis were: type of distractor (semantically related, form-related, and unrelated), electrode cluster (Anterior Left: F3, FC1; Anterior Right: F4, FC2, Central Left: FC5, C3, CP5; Central Right. FC6, C4, CP6, Centro-Posterior Left: CP1, P3; Centro-Posterior Right: CP2, P4; and Occipital: O1, Oz, O2; see Figure 1) and task (picture-picture and picture-word). Follow-up analyses were corrected using the Bonferroni correction and adjusted p-values are reported.

Fig. 1. Electrode montage used in the present study. Highlighted ROIs were used in the analysis.

3. Results

3.1. Behavioural results

Signing latencies were significantly slower in the picture-picture task than in the picture-word task, β = -110 ms, SE = 22.5, t(42.8) =4.88, p < .001. Compared to the unrelated condition, responses were faster in the form-related condition, β = -35 ms, SE = 9.9, t(29.9) = 3.48, p = .002. This phonological facilitation effect did not change significantly across tasks, β = -1 ms, SE = 22.7, t(30.1) = 0.06, p =.956. In contrast, there was no significant difference between the unrelated and the semantic condition, β = -4 ms, SE = 10.3, t(29.5) =0.36, p = .724 and no significant change of this contrast across tasks, β = 35 ms, SE = 19.2, t(32.7) = 1.85, p = .074.

Error rates did not differ significantly by task, β = -0.20, SE = 0.34, z = 0.58, p = .563. There was no significant difference between the unrelated and the form-related condition, β = -0.08, SE = 0.17, z = 0.49, p = .626, and no significant change of this contrast across tasks, β = 0.11, SE = 0.33, z = 0.32, p = .746. More errors were made in the semantic compared to the unrelated condition, β = 0.44, SE = 0.15, z = 2.83, p = .005. This semantic effect differed significantly across tasks, β = 1.32, SE = 0.31, z = 4.26, p < .001. In the picture-word task the semantic interference effect was significant, β = 1.09, SE = 0.21, z = 5.18, p < .001, whereas there was no significant semantic effect in the picture-picture task, β = -0.22, SE = 0.23, z = 0.98, p = .327. Figure 2 displays sign latencies and error probabilities as estimated in the model fits.

Fig. 2. Mean naming latencies and error probabilities for the picture-word (PWI) and picture-picture (PPI) interference tasks, as estimated in the model fits. Error bars represent the 95% CI. Small shapes and densities represent the individual means for each participant.

To summarize, behaviourally there was a facilitation effect for form-related distractors – which was of similar size across tasks. That is, participants named target pictures faster when distractors (either pictures or words) were related through the Spanish name of the target picture. There was a semantic interference effect in the picture-word interference tasks (on error rates only), whereas there was no semantic effect in the picture-picture interference task.

3.2. Electrophysiological results

Table 2 represents the main effects and interactions throughout the different time windows. Only significant results are discussed in this section.

Table 2. Significance table displaying the p-values on the repeated measures ANOVAs performed at 7 time windows. Significant effects are highlighted in bold with the corresponding F-statistics. Corrected values using the Greenhouse-Geisser correction are reported. TW: Time window (in ms) TD: Type of Distractor, T: Task, EC: Electrode Cluster

In the time window 150-250 ms after the onset of the stimuli presentation, ERP analyses revealed a main effect of type of distractor (F(1.88,43.13) = 3.31, p = 0.05). Post-hoc comparisons showed that there were significant differences between form-related and unrelated distractors (t(23) = 2.4, p = 0.02) and between semantically related and unrelated distractors (t(23) = 2.36, p = 0.03). Both related conditions elicited a larger positivity than the unrelated condition. In this time window, there was also a significant interaction between task and electrode cluster (F(2.14, 49.25) = 4.89, p = 0.01), but post-hoc comparisons did not reveal significant differences.

At 250-350 ms post-onset, there were significant interactions between task and electrode cluster (F(1.90, 43.63) = 10.73, p < 0.001); however, none of the post-hoc comparisons yielded significant results.

At the 350-450 ms time window, the analysis showed a main effect of type of distractor (F(1.93, 44.40) = 3.38, p = 0.04). Post-hoc comparisons showed that both form-related distractors (t(23) = 2.27, p = 0.03) and semantically related distractors (t(23) = 2.4, p = 0.02) elicited more positive-going waves compared to unrelated distractors. In addition, there was a significant interaction between task and electrode cluster (F(2.68, 61.55) = 5.24, p = 0.04). Post-hoc comparisons did not reveal significant differences.

At the 450-550 ms time window, there was a significant interaction between task and electrode cluster (F(2.83, 65.03) = 3.43, p = 0.02). No significant results were obtained in post-hoc comparisons.

To summarize the results reported above, in the early time window 150-250 ms post-onset and in the late time window 350-450 ms post-onset, there was a main effect of distractor type, and this factor did not interact with task or electrode cluster. Post-hoc comparisons revealed that form-related and semantically related distractors elicited more positive-going waves compared to unrelated distractors. Figure 3 depicts the ERP waves for each type of distractor across tasks, in the seven regions of interest.

Fig. 3. Event-related potentials from the semantic (sem), form-related (form) and unrelated (unr) conditions (Y axis: Mean amplitude in μ V) from the stimuli presentation (time 0) to 750 ms. Panel (a) depicts the ERP waves for the picture-word interference task (PWI) and panel (b) depicts the ERP waves for the picture-picture interference task (PPI). Nine regions of interest are represented: anterior left (AL), anterior right (AR), central left (CL), central right (CR), centro-posterior left (CPL), centro-posterior right (CPR) and occipital (O).

Figure 4 depicts the scalp map for semantic and phonological effects across tasks for the two critical time windows where significant differences were observed.

Fig. 4. Topographic maps depicting semantic (sem) and form-related (form) effects for the critical time windows. Effects were computed by subtracting the semantic and phonological distractor ERPs from the unrelated distractor ERPs. Voltage scale in microvolts. Panel (a) represents the picture-word interference task (PWI) and panel (b) represents the picture-picture interference task (PPI).

4. Discussion

In the present study, we explored cross-language, cross-modal interactions in deaf LSC-Spanish bilinguals. Participants named pictures in LSC while ignoring visual distractors in the form of Spanish words (picture-word interference task) or pictures (picture-picture interference task). Distractors were either semantically related to the target picture, form-related to the Spanish name of the target picture (phono-translation), or unrelated. For the semantic contrast (vs. unrelated), we observed no significant semantic effect on sign latencies in either task, but participants made more errors in the semantic condition than in the unrelated condition during the picture-word interference task. Electrophysiologically, semantically related distractors elicited less-negative-going waves than unrelated distractors in the time windows 150-250 ms and 350-450 ms post-stimulus onset. For the form-related contrast (vs. unrelated), we observed phonological facilitation across tasks; pictures presented with form-related distractors in Spanish were signed faster than those pictures presented with unrelated distractors. At the ERP level, this relationship elicited a reduced negativity in the same time windows where semantic effects were reported.

These results provide further evidence of cross-linguistic interactions in deaf bimodal bilinguals, both in language comprehension (Lee et al., Reference Lee, Meade, Midgley, Holcomb and Emmorey2019; Morford et al., Reference Morford, Wilkinson, Villwock, Piñar and Kroll2011) and language production (Emmorey et al., Reference Emmorey, Mott, Meade, Holcomb and Midgley2020; Giezen & Emmorey, Reference Giezen and Emmorey2016), and from the weaker L2 oral language onto the more dominant L1 sign language, a pattern seen in other studies within the oral modality (Bobb, Von Holzen, Mayor, Mani & Carreiras, Reference Bobb, Von Holzen, Mayor, Mani and Carreiras2020; Holzen & Mani, Reference Holzen and Mani2014). This indicates that deaf signers had attained sufficient proficiency in their L2 oral language to experience word influences during sign production (Van Hell & Tanner, Reference Van Hell and Tanner2012). Finally, we obtained very similar results in the picture-word interference task and in the picture-picture interference task. Despite differences between the two tasks in the input format of the distractors (written words vs. pictures), the magnitude of the phonological effect was very similar across these tasks. These results suggest that the oral language knowledge of deaf bilinguals influences sign production even when it is not directly involved in the task.

Regarding the semantic manipulation in our study, we only observed a semantic interference effect on error rates in the picture-word interference task, whereas we did not observe any significant semantic effect on sign latencies. The null result in the picture-picture interference task is in line with previous results in the oral domain. Multiple studies have shown no semantic effects from distractor pictures in picture naming (Damian & Bowers, Reference Damian and Bowers2003; Navarrete & Costa, Reference Navarrete and Costa2005; Roelofs, Reference Roelofs2008). Furthermore, there is evidence that semantic interference in this task may only be observed if the task specifically promotes attention to the distractor pictures (Jescheniak, Matushanskaya, Mädebach & Müller, Reference Jescheniak, Matushanskaya, Mädebach and Müller2014; Matushanskaya, Mädebach, Müller & Jescheniak, Reference Matushanskaya, Mädebach, Müller and Jescheniak2016), which was not the case in the present study.

For the picture-word interference task, the absence of a clear semantic interference on naming latencies is at odds with previous studies in the oral domain (Costa & Caramazza, Reference Costa and Caramazza1999; Hermans et al., Reference Hermans, Bongaerts, De Bot and Schreuder1998), but corresponds to other results in sign production of bimodal bilinguals. Similar to the present study, Giezen and Emmorey (Reference Giezen and Emmorey2016) had reported a null effect on naming latencies, whereas Emmorey et al. (Reference Emmorey, Mott, Meade, Holcomb and Midgley2020) even observed semantic facilitation. This suggests that semantically related distractor words may induce no semantic conflict in sign production in line with predictions of the response exclusion account. However, we believe that this conclusion may be premature for the following reasons. First, we did observe a semantic interference effect on error rates. This was also the case in the study by Emmorey et al. (Reference Emmorey, Mott, Meade, Holcomb and Midgley2020) and suggests that semantically related distractor words induce some level of conflict. Second, the null effect we observed on latencies is not in itself informative because we predicted either semantic interference (lexical competition) or facilitation (non-competitive selection) to occur. It is possible that the null effect in the present study (and in Giezen & Emmorey, Reference Giezen and Emmorey2016) reflects insufficient power of the design to reliably observe semantic interference effects (Brysbaert & Stevens, Reference Brysbaert and Stevens2018; Bürki, Elbuy, Madec & Vasishth, Reference Bürki, Elbuy, Madec and Vasishth2020). The results by Emmorey et al. (Reference Emmorey, Mott, Meade, Holcomb and Midgley2020) suggest that the true underlying effect may be one of facilitation. However, we believe that more evidence is needed to evaluate the direction and size of semantic distractor effects in bimodal sign production.

Our ERP results – in particular, the observation of less negative ERPs for semantically related distractors in the N400 time window – correspond to previous results in monolingual and bilingual picture-word interference studies (e.g., Blackford et al., Reference Blackford, Holcomb, Grainger and Kuperberg2012; Dell'Acqua et al., Reference Dell'Acqua, Sessa, Peressotti, Mulatti, Navarrete and Grainger2010; Roelofs et al., Reference Roelofs, Piai, Garrido Rodriguez and Chwilla2016; Rose, Aristei, Melinger & Rahman, Reference Rose, Aristei, Melinger and Rahman2019; Zhu et al., Reference Zhu, Damian and Zhang2015). In line with previous studies, we interpret this N400 attenuation to indicate semantic priming between the target picture and a related distractor. Some authors have argued that N400-attenuation reflects facilitatory priming of the target by the distractor and is therefore incompatible with lexical competition (Blackford et al., Reference Blackford, Holcomb, Grainger and Kuperberg2012; Emmorey et al., Reference Emmorey, Mott, Meade, Holcomb and Midgley2020). However, prominent lexical competition accounts assume that semantic context effects reflect a trade-off between facilitatory priming of the target picture by the distractor and interfering “reverse” priming of the distractor by the target picture. Following this argument, our result is compatible with both competitive and non-competitive production accounts (for discussion see Piai et al., Reference Piai, Roelofs, Jensen, Schoffelen and Bonnefond2014; Roelofs et al., Reference Roelofs, Piai, Garrido Rodriguez and Chwilla2016). Interestingly, we found an ERP-modulation not only in the N400 time window, but also earlier at around 200 ms. As we discuss in the introduction, earlier ERP-correlates of semantic distractor effects have also been found in some previous studies (Aristei et al., Reference Aristei, Melinger and Rahman2011; Dell'Acqua et al., Reference Dell'Acqua, Sessa, Peressotti, Mulatti, Navarrete and Grainger2010; Hoshino & Thierry, Reference Hoshino and Thierry2011; Rose et al., Reference Rose, Aristei, Melinger and Rahman2019). Such an early modulation fits with common time-course estimates for lexical access in picture naming. A recent study suggested that early ERP-modulations (at around 200ms) may reflect lexical competition, whereas later modulations (at around 400ms) may reflect ongoing semantic priming between target picture and distractor word (Rose et al., Reference Rose, Aristei, Melinger and Rahman2019).

In sum, we conclude that neither the behavioural nor the electrophysiological evidence unequivocally support the competitive or the non-competitive account. The behavioural results are inconclusive in this regard. The ERP-results correspond to similar findings in previous mono- and bilingual studies but appear compatible with both accounts. Moving forward, more evidence is needed to determine whether there is indeed semantic facilitation instead of semantic interference in bimodal picture-word interference tasks (as suggested by the result by Emmorey et al., Reference Emmorey, Mott, Meade, Holcomb and Midgley2020) and to clarify the functional relevance of the ERP-results for the behavioural effects.

Concerning phonological effects, the observation that picture names were signed faster in the presence of distractor words form-related to the oral name of the picture is, to our knowledge, the first piece of evidence showing that phonological properties of the oral language modulate how deaf individuals produce signs (see Lee et al., Reference Lee, Meade, Midgley, Holcomb and Emmorey2019; Hosemann et al., Reference Hosemann, Mani, Herrmann, Steinbach and Altvater-Mackensen2020, for similar results in comprehension). This was further validated by the ERPs showing sensitivity to the phonological manipulation. Form-related distractors elicited a reduced negativity compared to unrelated distractors between 150-250 ms, and 350-450 ms post onset. These modulations replicate phono-translation ERP effects reported in the oral modality (Hoshino & Thierry, Reference Hoshino and Thierry2011) and signed modality (e.g., Hosemann et al., Reference Hosemann, Mani, Herrmann, Steinbach and Altvater-Mackensen2020). Remarkably, phonological ERP effects were obtained in the same time windows and with the same direction as those obtained for the semantic contrast, although different results were obtained behaviourally for the two manipulations. This indicates that ERP polarities do not have a direct correspondence with behavioural effects (see also Dell'Acqua et al., Reference Dell'Acqua, Sessa, Peressotti, Mulatti, Navarrete and Grainger2010). In our study, similar ERP-modulations for the semantic and the phonological contrast might indicate priming between the distractor stimulus and the target picture, while not reflecting the functional consequence of such priming (in terms of facilitation or interference). Note that the polarity and timing (especially the early modulation) of the reported ERPs do not fit with the canonical N400 responses reported in picture-word interference tasks (Chauncey, Holcomb & Grainger, Reference Chauncey, Holcomb and Grainger2009). Acknowledging the differences, we followed N400 interpretations in picture-word interference studies and interpret our data as evidence of the priming effect for form-related distractors relative to unrelated targets (see also Hosemann et al., Reference Hosemann, Mani, Herrmann, Steinbach and Altvater-Mackensen2020 for a similar interpretation of N400-like effects in bimodal bilinguals).

Behaviourally, the phonological effect found here differed from previous studies with unimodal bilinguals using the picture-word interference paradigm (Costa et al., Reference Costa, Colomé, Gómez and Sebastián-Gallés2003; Hermans et al., Reference Hermans, Bongaerts, De Bot and Schreuder1998). The so-called phono-translation interference effect occurs when distractors are phonologically related to the translation of the target language (saying perro, ‘dog’ in English, presented with the distractor doll) and it has been interpreted as evidence of lexical competition in bilingual speech production.

Finding phonological facilitation in bimodal bilinguals and phonological interference in unimodal bilinguals could be reconciled by models which posit phonological interference effects arising at the phonological level (within-language competition models). According to this account, in the absence of phonological overlap between language modalities, the activation of the phonological properties of the oral lexicon could not interfere with the activation of the sign language phonology. Thus, these models would account for phonological facilitation effects in sign production arising at the lexical level. In particular, processing of the distractor, word or picture, would lead to lexical activation of the oral phonological neighbours of the distractor. For the form-related distractors this includes activation of the translation equivalents of the target sign. Activation of the target's translation in the oral language (due to phonological priming by the form-related distractor) may then facilitate target retrieval via automatic translation from oral to sign language. Under this assumption, phonological effects would be a consequence from both languages being activated during the task as a result of parallel activation processes.

As mentioned in the introduction, the obtained pattern of semantic and phonological effects, both behavioural and at the ERP level, could be attributed to the “mouthing” of words (or part-words) that co-occurs with sign articulation in code-blending production (Capek, Waters, Woll, MacSweeney, Brammer, McGuire & Campbell, Reference Capek, Waters, Woll, MacSweeney, Brammer, McGuire and Campbell2008; Giustolisi, Mereghetti & Cecchetto, Reference Giustolisi, Mereghetti and Cecchetto2017; Hosemann et al., Reference Hosemann, Mani, Herrmann, Steinbach and Altvater-Mackensen2020; Vinson et al., Reference Vinson, Thompson, Skinner, Fox and Vigliocco2010). In this context, it is conceivable that the articulatory buffer is shared for word-distractors and picture mouthings (i.e., the oral language). In this case, the response exclusion hypothesis may predict semantic interference to result from the same post-lexical conflict as for oral production: that is, due to slower exclusion of semantically-related distractor words (relative to an unrelated ones) from the articulatory buffer. This could delay availability of the mouthed phonemes of the picture, which would also delay sign onset if mouthings are produced in synchrony with the sign. More importantly, the phonological facilitation effect we observed could be explained by phonemes of the distractor overlapping (fully or partly) with those of the mouthed picture name, thus facilitating mouthing production of phonologically-related words. In other words, the phonological facilitation effect may reflect mouthing preparation rather than genuine cross-linguistic influence of the oral language on sign preparation. Following the suggestion of an anonymous reviewer we conducted a follow-up analysis to explore this possibility. As in previous reports (Vinson et al., Reference Vinson, Thompson, Skinner, Fox and Vigliocco2010), we observed that tendencies to produce mouthing widely varied between participants. In the present study, thirteen out of twenty-four participants were mouthing during most of the trials, and six participants did not produce mouthing while signing, indicating that mouthing and the manual components of signs could dissociate and were not obligatory mouth patterns of the signs selected. Further analysis suggested that mouthing was not a critical factor for the behavioural and the ERP results. That is, there were no substantial differences between those participants who were overtly mouthing and those participants who did not produce mouthing (or produced it only in a few trials). Thus, our results do not seem to be caused by mouthing productions while signing.

From the perspective of the parallel activation account it is surprising that the phonological facilitation effect appeared to be virtually identical with distractor pictures and distractor words. For distractor pictures, parallel activation of the phonological cohort should be much less direct, and thus weaker, than for distractor words, because phonological activation is necessarily mediated by visual and conceptual processing of the picture. In line with this argument, unimodal studies have found phonological facilitation to be more robust with distractor words than with distractor pictures (Bloem & La Heij, Reference Bloem and La Heij2003; Jescheniak et al., Reference Jescheniak, Oppermann, Hantsch, Wagner, Mädebach and Schriefers2009). For this reason, we prefer an alternative account under which the phonological effect is not a direct result of immediate co-activation of the L2 but reflects the reorganisation of the L1 as a result of L2 language learning processes (Costa, Pannunzi, Deco & Pickering, Reference Costa, Pannunzi, Deco and Pickering2017, Reference Costa, Pannunzi, Deco and Pickering2019). Within this framework, the present results would be reflecting the reorganisation of the sign language lexicon as a consequence of learning an oral language.

Under this account, lexical signs that were a priori not related in the sign language lexicon (e.g., CEREZA and CEREBRO, cherry and brain in English) would become related as a result of the phonological similarity of their corresponding translations in the oral language. It is possible that phonological properties of the oral language are linked to signs via mouthing production, considered to develop with bilingual experience of deaf individuals with the oral modality.

In this line, although not directly proposed for language production, the reading vocabulary acquisition model for deaf children (Hermans, Knoors, Ormel & Verhoeven, Reference Hermans, Knoors, Ormel and Verhoeven2008) could account for the learning hypothesis. The model describes how sign and oral languages interact in three developmental stages. In the first stage, deaf children only have access to the form of written words and the meaning is necessarily accessed throughout signs. The repeated co-activation of the sign and the written word translation equivalent results in the semantic and syntactic representations of the signs copied into the lexical representation of the written word. Finally, in the last stage, lexical entries contain all the semantic, morphological, and syntactic information. Considering the learning hypothesis, it is also possible that during the second stage, properties of the oral language are linked to sign forms via orthographic/phonological representations and, consequently, relationships in the oral language would ultimately map onto the sign language lexicon.

Either via reading (orthographic links) or mouthing processes (phonological links), or via the two representations combined, the sign lexicon of a native signer would be restructured when learning an oral language, resulting in a lexical network different from that of a deaf individual without such oral language experience. Thus, when processing a given sign not only the properties of the sign language would be activated, but also properties of the oral language which became linked to the sign. If this is the case, the effects observed in the present study may result from activation flow within the sign language lexicon instead of direct activation flow from the oral language lexicon to the sign lexicon. Note that this account is not arguing against oral language co-activation during sign production itself. The crucial difference is that under this account it is not the immediate co-activation of the oral language that causes the phonological facilitation effect but the reorganization of the sign lexicon following the repeated co-activation of both languages in the process of learning the oral language.

5. Summary and conclusion

The present study tested for cross-linguistic effects from an oral language (L2) on sign language production (L1) by deaf bimodal bilinguals. We found evidence for such cross-linguistic effects, most clearly in form of a phonological facilitation effect of distractors which were form-related to the oral language translation of the target signs. The ERP results suggest a lexical locus of the cross-linguistic interaction between sign and oral languages. The critical phonological effect appeared to be similar with distractor pictures and written distractor words. This suggests that cross-linguistic influences of the oral language are not restricted to task contexts involving oral language stimuli (i.e., distractor words). Most importantly, the present results provide the first piece of evidence that deaf bimodal bilinguals are sensitive to the properties of the oral language in sign production.

Acknowledgements

This work was supported by the BIAL Foundation [grant number 226/2016] the Spanish Government (RTI2018-096238-A-I00) and the Catalan government (2017 SGR 268). CB was supported by the Beatriu de Pinòs program [AGAUR grant number 2016 BP 00381]. AM was supported by the Beatriu de Pinòs program [AGAUR grant number 2017 BP 00180]. We would like to thank Gemma Barberà for her help in recruiting participants, Iris Milán for her assistance during the testing sessions, Nicholas Grunden for his suggestions on the present manuscript, and all the participants for taking part in the study. Author note: Data and analyses scripts are stored and publicly available at the Open Science Framework under https://osf.io/ymz9q/.

Supplementary Material

Supplementary material can be found online at https://doi.org/10.1017/S1366728921000171

Footnotes

1 Different than phonemes in oral languages, phonemes in sign languages are defined by structural units based on manual parameters such as handshape, place of articulation, movement, palm orientation (Brentari et al., Reference Brentari, Fenlon, Cormier, Brentari, Fenlon and Cormier2018) and non-manual behaviours of the face and the body (Pfau & Quer, Reference Pfau and Quer2010). Therefore, signs and words do not share any of its core components and the aforementioned parameters are not the translation of spoken phonemes, and vice versa.

2 Along the present manuscript we use the terms semantic and phonological effects referring to the semantic and phonological experimental manipulations, but it does not imply a semantic or a phonological locus of the effects.

3 Balancing was incomplete for the following reason. Three additional participants were scheduled but did ultimately not participate in the experiment. For this reason, fourteen participants performed the picture-picture interference task first and eleven participants performed the picture-word interference task first. It should be noted that we included task-sequence in the analysis and the critical results do not dependent on this factor.

4 Following a reviewer's suggestion, we explored the possibility of mouthing patterns accounting for some of the effects observed. During the experimental session, thirteen participants were overtly mouthing during most of the trials, five participants produced mouthing in some trials and six participants were not mouthing while signing. Post-hoc analysis showed no substantial differences between groups, so mouthing was not included as a factor in the final model.

References

Aristei, S, Melinger, A and Rahman, RA (2011) Electrophysiological chronometry of semantic context effects in language production. Journal of Cognitive Neuroscience 23(7), 15671586. https://doi.org/10.1162/jocn.2010.21474CrossRefGoogle ScholarPubMed
Baayen, RH, Davidson, DJ and Bates, DM (2008) Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59(4), 390412. https://doi.org/10.1016/j.jml.2007.12.005CrossRefGoogle Scholar
Barr, DJ, Levy, R, Scheepers, C and Tily, HJ (2013) Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68(3), 255278. https://doi.org/10.1016/j.jml.2012.11.001CrossRefGoogle ScholarPubMed
Bates, E, D'Amico, S, Jacobsen, T, Székely, A, Andonova, E, Devescovi, A, Herron, D, Lu, C, Pechmann, T, Pléh, C, Wicha, N, Federmeier, K, Gerdjikova, I, Gutierrez, G, Hung, D, Hsu, J, Iyer, G, Kohnert, K, Mehotcheva, T, Orozco-Figueroa, A, Tzeng, A and Tzeng, O (2003) Timed picture naming in seven languages. Psychonomic Bulletin and Review 10, 344380. https://doi.org/10.3758/BF03196494CrossRefGoogle ScholarPubMed
Bates, D, Mächler, M, Bolker, BM and Walker, SC (2015) Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1). https://doi.org/10.18637/jss.v067.i01CrossRefGoogle Scholar
Baus, C and Costa, A (2015) On the temporal dynamics of sign production: An ERP study in Catalan Sign Language (LSC). Brain Research 1609(1), 4053. https://doi.org/10.1016/j.brainres.2015.03.013CrossRefGoogle Scholar
Blackford, T, Holcomb, PJ, Grainger, J and Kuperberg, GR (2012) A funny thing happened on the way to articulation: N400 attenuation despite behavioral interference in picture naming. Cognition 123(1), 8499. https://doi.org/10.1016/j.cognition.2011.12.007CrossRefGoogle ScholarPubMed
Bloem, I and La Heij, W (2003) Semantic facilitation and semantic interference in word translation: Implications for models of lexical access in language production. Journal of Memory and Language 48, 468488. https://doi.org/10.1016/S0749-596X(02)00503-XCrossRefGoogle Scholar
Bobb, SC, Von Holzen, K, Mayor, J, Mani, N and Carreiras, M (2020) Co-activation of the L2 during L1 auditory processing: An ERP cross-modal priming study. Brain and Language 203. https://doi.org/10.1016/j.bandl.2019.104739CrossRefGoogle ScholarPubMed
Boukadi, M, Davies, R. A. I. and Wilson, MA (2015) Bilingual lexical selection as a dynamic process: Evidence from Arabic-French bilinguals. Canadian Journal of Experimental Psychology 69(4), 297313. https://doi.org/10.1037/cep0000063CrossRefGoogle ScholarPubMed
Brentari, D, Fenlon, J, Cormier, K, Brentari, D, Fenlon, J and Cormier, K (2018) Sign Language Phonology. In Oxford Research Encyclopedia of Linguistics. https://doi.org/10.1093/acrefore/9780199384655.013.117CrossRefGoogle Scholar
Brysbaert, M and Stevens, M (2018) Power Analysis and Effect Size in Mixed Effects Models: A Tutorial. Journal of Cognition 1(1), 120. https://doi.org/10.5334/joc.10CrossRefGoogle ScholarPubMed
Bürki, A, Elbuy, S, Madec, S and Vasishth, S (2020) What did we learn from forty years of research on semantic interference? A Bayesian meta-analysis. Journal of Memory and Language 114, 104125. https://doi.org/10.1016/j.jml.2020.104125CrossRefGoogle Scholar
Capek, CM, Waters, D, Woll, B, MacSweeney, M, Brammer, MJ, McGuire, PK and Campbell, R (2008) Hand and mouth: Cortical correlates of lexical processing in British sign language and speechreading english. Journal of Cognitive Neuroscience 20(7), 12201234. https://doi.org/10.1162/jocn.2008.20084CrossRefGoogle ScholarPubMed
Chauncey, K, Holcomb, PJ and Grainger, J (2009) Primed picture naming within and across languages: An ERP investigation. Cognitive, Affective and Behavioral Neuroscience 9(3), 286303. https://doi.org/10.3758/CABN.9.3.286CrossRefGoogle ScholarPubMed
Colomé, À. and Miozzo, M (2010) Which Words Are Activated During Bilingual Word Production? Journal of Experimental Psychology: Learning Memory and Cognition 36(1), 96109. https://doi.org/10.1037/a0017677Google ScholarPubMed
Costa, A, Albareda, B and Santesteban, M (2008) Assessing the presence of lexical competition across languages: Evidence from the Stroop task. Bilingualism 11(1), 121131. https://doi.org/10.1017/S1366728907003252CrossRefGoogle Scholar
Costa, A and Caramazza, A (1999) Is lexical selection in bilingual speech production language-specific? Further evidence from Spanish–English and English–Spanish bilinguals. Bilingualism: Language and Cognition 2(3), 231244. https://doi.org/10.1017/s1366728999000334CrossRefGoogle Scholar
Costa, A, Colomé, À., Gómez, O and Sebastián-Gallés, N (2003) Another look at cross-language competition in bilingual speech production: Lexical and phonological factors. Bilingualism: Language and Cognition 6(3), 167179. https://doi.org/10.1017/s1366728903001111CrossRefGoogle Scholar
Costa, A, Miozzo, M and Caramazza, A (1999) Lexical Selection in Bilinguals: Do Words in the Bilingual's Two Lexicons Compete for Selection? Journal of Memory and Language 41(3), 365397. https://doi.org/10.1006/jmla.1999.2651CrossRefGoogle Scholar
Costa, A, Pannunzi, M, Deco, G and Pickering, MJ (2017) Do Bilinguals Automatically Activate Their Native Language When They Are Not Using It? Cognitive Science 41(6), 16291644. https://doi.org/10.1111/cogs.12434CrossRefGoogle ScholarPubMed
Costa, A, Pannunzi, M, Deco, G and Pickering, MJ (2019) Does Bilingualism Alter Lexical Structure? Response to Oppenheim, Wu, and Thierry (2018). Cognitive Science 43(2), e12707. https://doi.org/10.1111/cogs.12707CrossRefGoogle Scholar
Costa, A, Strijkers, K, Martin, C and Thierry, G (2009) The time course of word retrieval revealed by event-related brain potentials during overt speech. Proceedings of the National Academy of Sciences 106(50), 2144221446. https://doi.org/10.1073/pnas.0908921106CrossRefGoogle ScholarPubMed
Damian, MF and Bowers, JS (2003) Locus of semantic interference in picture-word interference tasks. Psychonomic Bulletin and Review 10(1), 111117. https://doi.org/10.3758/BF03196474CrossRefGoogle ScholarPubMed
Davis, CJ and Perea, M (2005) BuscaPalabras: A program for deriving orthographic and phonological neighborhood statistics and other psycholinguistic indices in Spanish. Behavior Research Methods 37(4), 665671. https://doi.org/10.3758/BF03192738CrossRefGoogle ScholarPubMed
Delorme, A and Makeig, S (2004) EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. Journal of Neuroscience Methods, 134(1), 921. https://doi.org/10.1016/j.jneumeth.2003.10.009CrossRefGoogle ScholarPubMed
Dell'Acqua, R, Sessa, P, Peressotti, F, Mulatti, C, Navarrete, E and Grainger, J (2010) ERP evidence for ultra-fast semantic processing in the picture-word interference paradigm. Frontiers in Psychology (OCT). https://doi.org/10.3389/fpsyg.2010.00177CrossRefGoogle ScholarPubMed
Emmorey, K, Mott, M, Meade, G, Holcomb, PJ and Midgley, KJ (2020) Lexical selection in bimodal bilinguals: ERP evidence from picture-word interference. Language, Cognition and Neuroscience. https://doi.org/10.1080/23273798.2020.1821905Google ScholarPubMed
Giezen, MR and Emmorey, K (2016) Language co-activation and lexical selection in bimodal bilinguals: Evidence from picture-word interference. Bilingualism 19(2), 264276. https://doi.org/10.1017/S1366728915000097CrossRefGoogle ScholarPubMed
Giustolisi, B, Mereghetti, E and Cecchetto, C (2017) Phonological blending or code mixing? Why mouthing is not a core component of sign language grammar. Natural Language and Linguistic Theory 35(2), 347365. https://doi.org/10.1007/s11049-016-9353-9CrossRefGoogle Scholar
Hall, ML (2011) Bilingual picture-word studies constrain theories of lexical selection. Frontiers in Psychology 2(DEC). https://doi.org/10.3389/fpsyg.2011.00381CrossRefGoogle ScholarPubMed
Hermans, D (2004) Between-language identity effects in picture-word interference tasks: A challenge for language-nonspecific or language-specific models of lexical access? International Journal of Bilingualism 8(2), 115125. https://doi.org/10.1177/13670069040080020101CrossRefGoogle Scholar
Hermans, D, Bongaerts, T, De Bot, K and Schreuder, R (1998) Producing words in a foreign language: Can speakers prevent interference from their first language? Bilingualism: Language and Cognition 1(3), 213229. https://doi.org/10.1017/s1366728998000364CrossRefGoogle Scholar
Hermans, D, Knoors, H, Ormel, E and Verhoeven, L (2008) Modelling reading vocabulary learning in deaf children in bilingual education programs. Journal of Deaf Studies and Deaf Education 13(2), 155174. https://doi.org/10.1093/deafed/enm057CrossRefGoogle Scholar
Holzen, K. Von and Mani, N (2014) Bilinguals implicitly name objects in both their languages: An ERP study. Frontiers in Psychology 5(DEC), 1415. https://doi.org/10.3389/fpsyg.2014.01415Google Scholar
Hosemann, J, Mani, N, Herrmann, A, Steinbach, M and Altvater-Mackensen, N (2020) Signs activate their written word translation in deaf adults: An ERP study on cross-modal co-activation in German Sign Language. Glossa: A Journal of General Linguistics 5(1), 57. https://doi.org/10.5334/gjgl.1014CrossRefGoogle Scholar
Hoshino, N and Thierry, G (2011) Language selection in bilingual word production: Electrophysiological evidence for cross-language competition. Brain Research 1371, 100109. https://doi.org/10.1016/j.brainres.2010.11.053CrossRefGoogle ScholarPubMed
Jescheniak, JD, Matushanskaya, A, Mädebach, A and Müller, MM (2014) Semantic interference from distractor pictures in single-picture naming: evidence for competitive lexical selection. Psychonomic Bulletin and Review 21(5), 12941300. https://doi.org/10.3758/s13423-014-0606-5CrossRefGoogle ScholarPubMed
Jescheniak, JD, Oppermann, F, Hantsch, A, Wagner, V, Mädebach, A and Schriefers, H (2009) Do perceived context pictures automatically activate their phonological code? Experimental Psychology 56(1), 5665. https://doi.org/10.1027/1618-3169.56.1.56CrossRefGoogle ScholarPubMed
Knupsky, AC and Amrhein, PC (2007) Phonological facilitation through translation in a bilingual picture-naming task. Bilingualism 10(3), 211223. https://doi.org/10.1017/S1366728907003033CrossRefGoogle Scholar
Koester, D and Schiller, NO (2008) Morphological priming in overt language production: Electrophysiological evidence from Dutch. NeuroImage 42(4), 16221630. https://doi.org/10.1016/j.neuroimage.2008.06.043CrossRefGoogle ScholarPubMed
Kroll, JF, Bogulski, CA and McClain, R (2012) Psycholinguistic perspectives on second language learning and bilingualism. Linguistic Approaches to Bilingualism 2(1), 124. https://doi.org/10.1075/lab.2.1.01kroCrossRefGoogle Scholar
Kubus, O, Villwock, A, Morford, JP and Rathmann, C (2015) Word recognition in deaf readers: Cross-language activation of German Sign Language and German. Applied Psycholinguistics 36(4), 831854. https://doi.org/10.1017/S0142716413000520CrossRefGoogle Scholar
Kuznetsova, A, Brockhoff, PB and Christensen, RHB (2015) Package ‘lmertest’. R package.Google Scholar
Lee, B, Meade, G, Midgley, KJ, Holcomb, PJ and Emmorey, K (2019) ERP Evidence for Co-Activation of English Words during Recognition of American Sign Language Signs. Brain Sciences 9(6), 148. https://doi.org/10.3390/brainsci9060148CrossRefGoogle ScholarPubMed
Lopez-Calderon, J and Luck, SJ (2014) ERPLAB: An open-source toolbox for the analysis of event-related potentials. Frontiers in Human Neuroscience, 8(1 APR). https://doi.org/10.3389/fnhum.2014.00213CrossRefGoogle ScholarPubMed
Mahon, BZ, Costa, A, Peterson, R, Vargas, KA and Caramazza, A (2007) Lexical Selection Is Not by Competition: A Reinterpretation of Semantic Interference and Facilitation Effects in the Picture-Word Interference Paradigm. https://doi.org/10.1037/0278-7393.33.3.503.suppCrossRefGoogle Scholar
Marian, V and Spivey, M (2003a) Bilingual and monolingual processing of competing lexical items. Applied Psycholinguistics 24(2), 173193. https://doi.org/10.1017/S0142716403000092CrossRefGoogle Scholar
Marian, V and Spivey, M (2003b) Competing activation in bilingual language processing: Within- and between-language competition. Bilingualism: Language and Cognition 6(2), 97115. https://doi.org/10.1017/s1366728903001068CrossRefGoogle Scholar
Matushanskaya, A, Mädebach, A, Müller, MM and Jescheniak, JD (2016) When Sufficiently Processed, Semantically Related Distractor Pictures Hamper Picture Naming: Implications for Models of Lexical Access in Speech Production. Experimental Psychology 63(6), 307317. https://doi.org/10.1027/1618-3169/a000340CrossRefGoogle Scholar
Morford, JP, Kroll, JF, Piñar, P and Wilkinson, E (2014) Bilingual word recognition in deaf and hearing signers: Effects of proficiency and language dominance on cross-language activation. Second Language Research 30(2), 251271. https://doi.org/10.1177/0267658313503467CrossRefGoogle ScholarPubMed
Morford, JP, Occhino-Kehoe, C, Piñar, P, Wilkinson, E and Kroll, JF (2017) The time course of cross-language activation in deaf ASL-English bilinguals. Bilingualism 20(2), 337350. https://doi.org/10.1017/S136672891500067XCrossRefGoogle ScholarPubMed
Morford, JP, Wilkinson, E, Villwock, A, Piñar, P and Kroll, JF (2011) When deaf signers read English: Do written words activate their sign translations? Cognition 118(2), 286292. https://doi.org/10.1016/j.cognition.2010.11.006CrossRefGoogle ScholarPubMed
Navarrete, E and Costa, A (2005) Phonological activation of ignored pictures: Further evidence for a cascade model of lexical access. Journal of Memory and Language 53(3), 359377. https://doi.org/10.1016/j.jml.2005.05.001CrossRefGoogle Scholar
Pfau, R and Quer, J (2010) Nonmanuals: Their grammatical and prosodic roles. In Sign Languages (pp. 381–402). https://doi.org/10.1017/CBO9780511712203.018CrossRefGoogle Scholar
Piai, V, Roelofs, A, Jensen, O, Schoffelen, JM and Bonnefond, M (2014) Distinct patterns of brain activity characterise lexical activation and competition in spoken word production. PLoS ONE 9(2). https://doi.org/10.1371/journal.pone.0088674CrossRefGoogle ScholarPubMed
Piñar, P, Dussias, PE and Morford, JP (2011) deaf readers as bilinguals: An examination of deaf readers’ print comprehension in light of current advances in bilingualism and second language processing. Linguistics and Language Compass 5(10), 691704. https://doi.org/10.1111/j.1749-818X.2011.00307.xCrossRefGoogle Scholar
R Core Team (2019) R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.r-project.org/Google Scholar
Roelofs, A (2008) Tracing Attention and the Activation Flow in Spoken Word Planning Using Eye Movements. Journal of Experimental Psychology: Learning Memory and Cognition 34(2), 353368. https://doi.org/10.1037/0278-7393.34.2.353Google ScholarPubMed
Roelofs, A, Piai, V, Garrido Rodriguez, G and Chwilla, DJ (2016) Electrophysiology of cross-language interference and facilitation in picture naming. Cortex 76, 116. https://doi.org/10.1016/j.cortex.2015.12.003CrossRefGoogle ScholarPubMed
Rose, SB, Aristei, S, Melinger, A and Rahman, RA (2019) The closer they are, the more they interfere: Semantic similarity of word distractors increases competition in language production. Journal of Experimental Psychology: Learning Memory and Cognition 45(4), 753763. https://doi.org/10.1037/xlm0000592Google ScholarPubMed
Shook, A and Marian, V (2012) Bimodal bilinguals co-activate both languages during spoken comprehension. Cognition 124(3), 314324. https://doi.org/10.1016/j.cognition.2012.05.014CrossRefGoogle ScholarPubMed
Snodgrass, JG and Vanderwart, M (1980) A standardized set of 260 pictures: Norms for name agreement, image agreement, familiarity, and visual complexity. Journal of Experimental Psychology: Human Learning and Memory 6(2), 174215. https://doi.org/10.1037/0278-7393.6.2.174Google ScholarPubMed
Van Hell, JG and Tanner, D (2012) Second Language Proficiency and Cross-Language Lexical Activation. Language Learning 62(SUPPL. 2), 148171. https://doi.org/10.1111/j.1467-9922.2012.00710.xCrossRefGoogle Scholar
Villameriel, S, Dias, P, Costello, B and Carreiras, M (2016) Cross-language and cross-modal activation in hearing bimodal bilinguals. Journal of Memory and Language 87, 5970. https://doi.org/10.1016/j.jml.2015.11.005CrossRefGoogle Scholar
Vinson, DP, Thompson, RL, Skinner, R, Fox, N and Vigliocco, G (2010) The Hands And Mouth Do Not Always Slip Together in British Sign Language: Dissociating Articulatory Channels in the Lexicon. Psychological Science 21(8), 11581167. https://doi.org/10.1177/0956797610377340CrossRefGoogle Scholar
Wu, YJ and Thierry, G (2010) Chinese-english bilinguals reading english hear chinese. Journal of Neuroscience 30(22), 76467651. https://doi.org/10.1523/JNEUROSCI.1602-10.2010CrossRefGoogle ScholarPubMed
Zhu, X, Damian, MF and Zhang, Q (2015) Seriality of semantic and phonological processes during overt speech in Mandarin as revealed by event-related brain potentials. Brain and Language 144, 1625. https://doi.org/10.1016/j.bandl.2015.03.007CrossRefGoogle Scholar
Figure 0

Table 1. Demographic information of the participants. Mean ratings (M) and standard deviation (SD)

Figure 1

Fig. 1. Electrode montage used in the present study. Highlighted ROIs were used in the analysis.

Figure 2

Fig. 2. Mean naming latencies and error probabilities for the picture-word (PWI) and picture-picture (PPI) interference tasks, as estimated in the model fits. Error bars represent the 95% CI. Small shapes and densities represent the individual means for each participant.

Figure 3

Table 2. Significance table displaying the p-values on the repeated measures ANOVAs performed at 7 time windows. Significant effects are highlighted in bold with the corresponding F-statistics. Corrected values using the Greenhouse-Geisser correction are reported. TW: Time window (in ms) TD: Type of Distractor, T: Task, EC: Electrode Cluster

Figure 4

Fig. 3. Event-related potentials from the semantic (sem), form-related (form) and unrelated (unr) conditions (Y axis: Mean amplitude in μ V) from the stimuli presentation (time 0) to 750 ms. Panel (a) depicts the ERP waves for the picture-word interference task (PWI) and panel (b) depicts the ERP waves for the picture-picture interference task (PPI). Nine regions of interest are represented: anterior left (AL), anterior right (AR), central left (CL), central right (CR), centro-posterior left (CPL), centro-posterior right (CPR) and occipital (O).

Figure 5

Fig. 4. Topographic maps depicting semantic (sem) and form-related (form) effects for the critical time windows. Effects were computed by subtracting the semantic and phonological distractor ERPs from the unrelated distractor ERPs. Voltage scale in microvolts. Panel (a) represents the picture-word interference task (PWI) and panel (b) represents the picture-picture interference task (PPI).

Supplementary material: File

Gimeno-Martínez et al. supplementary material

Table S2

Download Gimeno-Martínez et al. supplementary material(File)
File 13.3 KB
Supplementary material: File

Gimeno-Martínez et al. supplementary material

Table S1

Download Gimeno-Martínez et al. supplementary material(File)
File 15.8 KB