Radical Creative Semantic Anchoring: Creative-action metaphors and timbral interaction

Abstract

This paper explores participatory and socially engaged practices in ubiquitous music (ubimus). We discuss recent advances that target timbre as their focus while incorporating semantic strategies for knowledge transfer among participants. Creative Semantic Anchoring (ASC from the original in Portuguese) is a creative-action metaphor that shows promising preliminary results in collaborative asynchronous activities. Given its grounding in local resources and its support for explicit knowledge, ASC features a good potential to boost socially distributed knowledge. We discuss three strategies that consolidate and expand this approach within ubiquitous music and propose the label Radical ASC. We investigate the implications of this framework through the analysis of two artistic projects: Atravessamentos and Ntrallazzu.

1. SECOND-WAVE UBIQUITOUS MUSIC

Ubiquitous music (ubimus) research strives to address the needs of ‘lay participants’ ¹ in creative musical activities and of musicians in everyday settings (Keller, Lazzarini and Pimenta 2014). While it is true that these issues have been partially addressed in the contemporary music literature, the strength of the ubimus approaches lies in their empirical grounding, ² allied to a self-critical perspective on the presence of technologies in everyday contexts. Except for a few isolated initiatives (Spiegel 1981), until the turn of the century most research in musical interaction revolved around hierarchical and individualistic notions of creativity, targeting concepts such as ‘virtuosity’, ‘self-expression’ (Wessel and Wright 2002) and ‘the orchestra’ (Trueman 2007). ³

Ubiquitous music research has opened new avenues for creative action ‘which in part seeks to increase the accessibility and spread of creative and artistic tools, including outside of traditional studio settings’ (Cullimore and Gerhard 2015: 5). Keller et al. (2014) mention four research trends that frame our discussion:

1. 1. A change of focus from creative products to processes, fostered by various initiatives in musicology that target the analysis of alternative creative methods (Keller and Ferneyhough 2004; Marsden, 2012; Coelho de Souza 2019). These approaches entail the usage of temporalities ⁴ as a replacement of centralised and hierarchical techniques. ⁵

2. 2. An increased reliance on information technology support (Vaggione 2001; Truax 2002). After 2020 there are new demands of usage of resources available in everyday settings and involving forms of musical interaction not based on physical contact (Keller, Costalonga and Messina 2020).

3. 3. An intensive usage of local resources ⁶ in creative activities ⁷ (Burtner 2005; Basanta 2010), fostering the emergence of everyday musical creativity (Keller and Lima 2016).

4. 4. A tendency to move from prescriptive models to descriptive and predictive models that signals a shift from genre-specific musical theories to multimodal creative frameworks (Ferraz and Keller 2014; Keller, Messina, Silva and Feichas 2020).

Feldman’s (Feldman and Friedman 2000) critical metaphor of the composerly hand hints at creative methods that avoid centralised decision-making through computational tools, environmental sounds, multimodal media and/or audience participation. This trend erodes the romantic image of the isolated composer creating music just in the head via quasi-metaphysical inspiration and encourages the search for creative perspectives that avoid the preconceptions of musical genres that characterise many twentieth-century music practices.

‘Creative semantic anchoring’ (or ASC, from the original in Portuguese) is a metaphor for creative action that uses verbal strategies to ground aesthetic decision-making. Creative semantic anchoring involves the usage of verbal tokens to facilitate aesthetic decisions and musical knowledge transfer. Keller et al. (2020) present a study that applies semantic strategies for the selection of sound sources. This study focuses on emulations of everyday sounds ⁸ using semantic descriptors. The sonic materials consist of excerpts of violin works, based on imitations of everyday events. To assess the relevance of the descriptors, the experimenters gathered responses from 58 university students with basic or intermediate musical training. The results indicate a good creative potential for sonic applications. These results also imply that the creative usage of the semantic approaches may be inversely related to the technical knowledge necessary to deal with the sonic resources.

The present paper provides examples of creativity support strategies that would not be considered ‘musical’ by twentieth-century hegemonic compositional narratives (cf. Schaeffer 1966; Boulez and Nattiez 1986), while expanding the available resources for professional and novice-oriented musical activities. Temporalities (Keller 2020) as opposed to hierarchical and centralised models of social interaction in music-making, massive deployments of potentially useful artistic resources in everyday settings, the emergence of everyday creative phenomena entailing an active engagement of non-musicians in decision-making and aesthetic experiences that feature multimodal elements, all point to the need to move beyond the genre-centric (see section 1.2) understanding of music that characterises acoustic-instrumental thinking. The methods presented in this paper can be gathered under a rubric of politics of doing, akin to Michel de Certeau’s ‘tactics’ or the ‘art of “making do”’ (Certeau 1988: 30). Instead of focusing on what each artistic project is about, we are interested in unveiling how the stakeholders manage to deploy and deal with the resources. We claim that the second wave of ubimus proposals can be characterised by very specific approaches to musical interaction design and that these methods have political consequences that have not been addressed by the music research literature.

The discussion is organised around three strategies. The first thread focuses on the relationship between the semantic aspects of music-making and their impact on timbre. The second thread involves the concept of place and how ubimus methods have incorporated local resources. The third targets the support of distributed decision-making procedures while avoiding the centralisation and the hierarchical relationships that characterise the acoustic-instrumental paradigm. These three aspects are materialised in the multimedia projects Atravessamentos and Ntrallazzu that serve as examples of the deployment of Radical ASC.

1.1. Beyond mappings and presets

Creative approaches to music-making based on timbre may demand tools and concepts that not only rely on parametric control but also need to take into account the semantic dimensions of creative practice. Ubimus provides strategies to deal with these issues. This section establishes a bridge between the ongoing research on timbre as a perceptual entity and the possibilities afforded by its creative usage in music-making. We point to shortcomings of simplistic mapping strategies and suggest alternative paths to enable access to non-musicians.

From a sound-perception outlook, timbre phenomena can be described through high-level tokens related to their perceptual attributes. These tokens may comprise semantic meanings associated with different aspects of sound (such as the profiles of dynamics, harmonics and transients and the spread of spectral energy). Early experiments on sound analysis and synthesis indicated the possibility of extracting and modifying key characteristics of instrumental sounds (Risset 1991). Grey and Gordon (1978) applied multidimensional models to extract spectral features that yielded spatial representations of timbre. A subsequent study carried out by Wessel (1979: 46) targeted the control of timbre parameters for compositional usage. Pitch sequences with instrumental timbre were generated by means of additive-synthesis techniques. The results indicated that melodic-fission and stream-segregation effects are related to the spectral differences between the sounds (Wessel 1979: 49).

Despite the widespread usage of timbre space as a two- or three-dimensional mapping tool for parametric control, recent discussions of the relationships between timbre and the design of metaphors for creative action have pointed to some limitations in this mapping strategy. For instance, Born (2013: 11) argues that regarding the ‘Euclidean and statistical model of timbral space … derived from subjective perceptions … to drive sound synthesis controls; we might say that the normative and technical enfold and order the subjective and perspectival.’ Although this criticism is valid, the historical context of the original proposal should be taken into account. Risset (1991), Grey (1978), Wessel (1979), Bregman (1999) and other researchers provided the initial insights into a phenomenon that was not fully understood and that remained beyond the grasp of audio synthesis and processing techniques for many years after the deployment of the acoustic compilers. ⁹ Mapping strategies have shown serious limitations because they enforce simplistic relationships between the interaction parameters and the timbral outcomes. Wessel’s mapping model may be limited when applied to broader conceptualisations of timbre. In short, we need more refined methods that take into account the recent advances in timbre research.

Other authors examine the link between the verbal attributes and the perceptual dimensions with promising contributions towards accessible strategies to timbre manipulation. Stepánek (2006: 126) correlated verbal attributes spontaneously pronounced by subjects with common words borrowed from hearing (such as ‘gloomy’, ‘harsh’ or ‘narrow’). Zacharakis, Pastiadis and Reiss (2012) conducted a multilingual experiment on the semantic dimensions of timbre. They found that verbal attributes, in both Greek and English, correlate to specific acoustic features. For example, texture is related to the energy distribution of harmonic partials, thickness and brilliance is linked with inharmonicity and the variations of spectral centroid, and mass or luminance is related to the fundamental of the harmonic content. Furthermore, Saitis, Weinzierl, von Kriegstein, Ystad and Cuskley (2020) focus on the implications of the perceptual studies to achieve a better grasp of human semantic processing for sound analysis and creative practice.

Thus, while the relevance of timbre research to music-making remains undisputed, current perspectives tend to rely on cross-modal and semantic aspects to explain perceptual phenomena. Simplistic mapping strategies, despite being useful at a time when interaction and audio-processing techniques were limited, suffer from various shortcomings related to the complex dynamics of timbre emergence. The adoption of semantics as a strategy to deal with timbre may provide a path to handle part of these issues. One approach involves understanding creativity as a product of ecologies or ecosystems (Keller and Lazzarini 2017). The next section explores this perspective.

1.2. Beyond acoustic-instrumental thinking

Ecologically grounded creative practices emerged in the late 1990s and encompassed two threads. One thread proposed a theoretical framework supported by embedded–embodied cognitive mechanisms (Keller 2000; Burtner 2005; Nance 2007). Another thread entailed the concurrent development of design techniques, featuring participation and emergence as two central creative driving forces (Basanta 2010; Keller et al. 2010; Gomes et al. 2014; Connors 2015). Eco-oriented creative practices applied the notion of embodiment to creative music-making before its popularisation among musicologists and anthropologists (for examples of late adopters of this approach, see Ingold 2013 and Manning and Massumi 2014). Embodiment in eco-grounded creative practice was not divorced from the related concepts of embeddedness and situated action, indicating an alignment with the recently labelled E4 perspective (Malinin 2016). ¹⁰

When restricted to acoustic-instrumental music-making, embodiment implies a narrow view on musical knowledge acquisition (see, for instance, Nijs, Lesaffre and Leman 2009). The interfaces and resources based on European orchestral instruments furnish a prime example of genre-specific knowledge. Rather than calling this knowledge ‘musical’, it should be labelled ‘orchestral’ or, even better, ‘piano-, clarinet- or guitar-based’ knowledge. It only targets the resources linked to the instrumental performance of acoustic or digitally emulated instruments. Its applicability does not encompass the rich experiences provided by a growing variety of multimodal artistic formats (Mesz, Sigman and Trevisan 2012), by the current expansion of analogue computing (Lazzarini and Timoney 2020), and it does not take into account the contributions of the makers’ movement to music-making (Timoney, Lazzarini and Keller 2020). ¹¹ Furthermore, ubimus research has shown that everyday musical creativity lies beyond the reach of this genre-specific view (Keller and Lima 2016; Keller 2020).

Interestingly, both professional and everyday music creativity might boost complementary sonic representations featuring verbal and semantic facets (Keller et al. 2020). On the one hand, a narrow understanding of musical knowledge does not encompass the richness of the multimodal experiences. Consequently, it may separate the musical methods from the actual multimodal musical experience (Traube 2015). On the other hand, some verbal descriptions used in sound-perception experiments aim to link the sonic-mediated experiences with the listeners’ everyday experiences (Mesz et al. 2012; Saitis et al. 2020). Pointing towards the semantic strategies introduced in the following section, perceptually oriented verbal tokens might yield alternative tools either to describe or to handle sound through diverse semantic analogies and allegories.

1.3. Beyond hierarchical chronotopical networks

Determinations of time and space are fundamental to the classification of creative practices. According to Mikhail Bakhtin, the ‘chronotope (literally time-space)’ is a notion that expresses the ‘intrinsic connectedness of temporal and spatial relationships’ in conjunction with ‘the inseparability of space and time’ in literary and artistic discourse (Bakhtin 1981: 84). Ubimus research has endeavoured to debunk the traditional chronotopical categories that are usually associated with musical instrumental practices. Keller et al. (2010), for instance, suggest that the concert – as a portion of time that is rigidly separated from the rest of life events – and the concert hall – as the dedicated space where music must happen – belong to a set of social paraphernalia that dissuade from musical engagement. Ubimus efforts in this area are concentrated towards overcoming the space–time categories inherited from acoustic-instrumental thinking.

Since the pioneering work by Rhodes (1961), place has been considered an important factor for creative endeavours. Nevertheless, this dimension is not yet fully incorporated into mainstream composition and musical-interaction design practices. Starting in the 1960s, site-specific art (Friedman, Smith and Sawchyn 2002) and soundscape composition (Truax 2002) fostered the use of local sonic resources. Recent ubimus practices emphasise the interrelated qualities of resources and agents, ¹² pointing to a set of relational properties that may gain increasing weight in ubimus design (Keller et al. 2015).

The ties of the Internet of Things (IoT) with ubimus are explicit. Nevertheless, it is not yet clear whether the ubimus activities enabled by the deployment of the IoT infrastructure should be strictly tied to the Internet of Musical Things (IoMusT). Turchet, Fischione, Essl, Keller and Barthet (2018) attempt an inclusive definition of this term. But the examples provided unveil a bias towards a subset of instrumental forms of musical thinking, rooted in the hegemonic acoustic-instrumental paradigm. Recent ubimus contributions to creative musical practices are not supported by the proposed IoMusT functionalities. For instance, both generative strategies (Kramann 2020) and free-improvisation practices (Clemente, Falleiros, Tavares and Fornari 2020; Stolfi 2020) present difficult scenarios for synchronous aesthetic decision-making. Without careful planning and design of the creative-action metaphors, some generative strategies could become unwieldy for novices. Free improvisation relies on a large pool of tacit knowledge usually shared through arduous, long-term investments in collective musical practice. The synchronous usage of the IoMusT resources does not necessarily address these caveats.

As alternatives to the notions inherited from acoustic-instrumental thinking, ecologically grounded creative practices apply metaphors of territorialisation and geopolitics for ubimus network-based collaborations (Messina, Svidzinski, Menezes, Bezerra and Costa 2019). For instance, collective live-patching experiences with the software Kiwi ¹³ give the participants freedom to delete the objects or to interrupt the connections created by other stakeholders. Depending on the number of simultaneously active users this tends to happen often (Messina et al. 2019), consequently, ubimus collaborative live patching embraces conflict, fostering a particular twist to the aural results. The simultaneous actions exerted by multiple subjects on the same canvas lead to conflicts over the computational objects, comparable to the fight for limited resources to secure territorial control. The connection between the visual algorithms and the interactions among multiple users that characterises Kiwi points to an affinity between live patching as a musical activity on a limited portion of space – in the case of Kiwi, the patch canvas – and the politics of territorialisation that inscribes human actions in space. While conflict is certainly embedded in instrumental music-making, we argue for an awareness not just of the aural results but also of the visual metaphors and the semantic contents that govern the creative processes and products (which are also different at each node of the network).

The limitations of physical distancing imposed by the Covid-19 pandemic has triggered a reflection and reassessment of the creative potential of asynchronous interaction (Keller, Costalonga and Messina 2020). Arguably, asynchronous strategies may provide an optimal creative environment for some forms of comprovisation (Messina and Aliel 2019; Aliel and Keller 2020) ¹⁴ and for the deployment of ubimus generative models (Kramann 2020). ¹⁵ Decentralised temporal decisions defy the established logics of power and normativity (cf. Ashcroft, Griffiths and Tiffin 2007) based on specific geographical locations, time zones and genres as the models to conform to. As we will discuss further with reference to Amazonia, this is particularly relevant when collaborative projects involve spaces typically imagined as ‘distant’, ‘peripheral’ or ‘underdeveloped’. It is no surprise that the aforementioned intercontinental live patching sessions were aligned, organised and regulated via the Paris time zone (Messina et al. 2019). It was precisely thanks to Kiwi’s partial support to asynchronous interaction that the participants managed to mitigate this chronotopical hegemony (cf. Bakhtin 1981; Müller 2010). The Bakhtinian category of ‘chronotope’ ¹⁶ provides a cue to introduce another concept formulated within the Bakhtin Circle (cf. Brandist 2002); namely, the idea that the linguistic sign, with its phonetic, morphologic and especially semantic dimension is always ideologically charged (Vološinov 1986). Semantic meaning, in other words, is always and inevitably political. This aspect adds weight to ASC’s relevance for politically inspired music.

The next sections present two ubimus artistic projects based on semantic strategies that address the three preceding issues; namely, they entail the use of semantic epimusical resources (Keller et al. 2020), ¹⁷ they employ local referents and resources to engage with aspects of everyday creativity and they adopt distributed decision-making procedures to avoid hierarchically oriented social exchanges. These methods share characteristics that justify their treatment as a block and constitute a family of approaches based on the use of semantics. They avoid the hierarchical division of labour of the acoustic-instrumental paradigm by proposing open-ended negotiations among the stakeholders, while playing with the listeners’ expectations by shifting the meanings of the referents.

2. STUDY 1: NTRALLAZZU

Ntrallazzu is the title of a cycle of works composed by Marcello Messina after an initial call for scores by the Lovemusic Collectif. ¹⁸ The cycle is entirely based on interactive live scores, and reflects on the multifaceted concepts of liveness and interaction both from a creative point of view and as a critical philosophical perspective. Typically, the pieces that compose the cycle are 5 to 6 minutes long and rely on a projected score that interacts in real time with the materials played by the performer(s). While one of the performers plays, the sound is fed to and processed by a patching software, and generates both electronic sounds and a score that may be performed by a second player. Both the live score and the electronics run on Max. ¹⁹

Two instances of the Ntrallazzu cycle are particularly relevant to the Radical ASC rubric: Ntrallazzu 4 (Messina and Aliel 2018) (Video Example 1), premiered in 2018 by Luzilei Aliel as a performer on the fife, electric guitar and effects at the Universidade Federal de São João del Rei, in Brazil, during the VIII UbiMus (Messina and Aliel 2019); and Ntrallazzu 6 (Video Example 2) for spoken voice, MIDI-operated piano and effects, premiered by Marcello Messina as composer and performer in 2019 in Marseille, France, during the Ubiquitous Music Workshop (Messina 2021). Both instances involve a single performer, although in Ntrallazzu 4 Luzilei Aliel played both the fife and the electric guitar and, thanks to an intelligent use of effects, managed to maintain the feeling of an interaction between two instruments.

Ntrallazzu 6 is a text-based piece of the cycle (Figure 1). ASC is relevant and operative in that articulated sentences in the piece are used as compositional material. At the same time, the use of spoken sentences permits a dialogue with the historical and philosophical premises of the cycle. In Ntrallazzu 6 (Video Example 2), the voice does not interact acoustically with any instrument. However, a MIDI-Out keyboard plugged to the patch provides a generative piano accompaniment that pushes the ecological balance of comprovisation away from the human being as the principal and exclusive centre of the creative process. In Ntrallazzu 6, the discursive gap between the sonic material, the programmatic explanation of the piece and its own programme note is eliminated, as all these elements conflate in the performance of the piece. Words spoken in English, their positional and combinational functions and meanings, become the core material of the piece, while the sound, in the form of oral speech, happens as a consequence of the semantic utterances. The live electronics and the MIDI piano accompaniment are, in turn, consequences of this consequence, as they react to the speech. Semantics, in this context, anchors the creative actions as it constitutes both the main trigger and the principal discernible result of the performance.

Figure 1. A snapshot of the live score of Ntrallazzu 6. Source: Marcello Messina’s personal archive.

Ntrallazzu 4 does not use text as its most immediate material. However, a series of prior verbal negotiations between the composer and the performer permitted the avoidance of any engagement with the notated material and the patch, making ASC also relevant in this case. Dialogism in ubimus has been abundantly explored by Lima et al. (2014), through the prism of Paulo Freire’s (1996) pedagogy of the oppressed. In this instance, however, Freirean thought is complemented by Bakhtinian dialogism (Bakhtin 1984), and with its aforementioned stress on semantics, and, generally, on the meaning of the linguistic sign, as conveyor of ideologically charged content (Vološinov 1986). This process of dialogical adjustments is markedly political, as it led to a thorough renegotiation of the original functioning of the piece, to the point that Luzilei Aliel layered effects generated via other patching programs (Pure Data) with the effects that were already taking place through the Ntrallazzu Max patch. This made the live interaction more complex and unpredictable, but yet more importantly, in Ntrallazzu 4, a patch compiled in an open source environment such as Pure Data disrupts the original piece that runs on a proprietary software (Max), literally redeeming the piece and opening up new possibilities for sustainable developments (cf. Puckette 2004; Messina 2017). As claimed by Miller Puckette:

I left IRCAM, and when it became clear that they would not permit my own further work on Max to be disseminated, I gave up and started a new project, Pd. (One meaning of Pd was ‘Public Domain’.) … Artifacts of art may be owned, but ‘digital art’ itself is not intrinsically ownable by anybody. This is bad news to composers, for instance, who obviously would like to own their scores. They do indeed own the paper and the ink on top of it, but the work exists only as a way of arranging things, not in the things themselves, and therefore can’t be owned. Composers and other digital artists must survive by the mechanism of attribution. (Puckette 2004: 2–3)

Subscribing to Puckette’s ideological position, we need to name the problematic status of Ntrallazzu as a cycle of works that is entirely dependent on Max: in this context, the disruption operated in Ntrallazzu 4 via Pure Data might be seen, as it were, as a form of redemption. Messina and Aliel (2019) have sardonically described this process as a factual ‘hijacking’ of the original piece, which nevertheless had the merit of ‘opening a wide range of unforeseen possibilities’ dialoguing ‘with ubiquitous music and with the transcendence of the rigid separation of roles (in particular, composer vs. performer) and social practices on which (Western) art music ²⁰ is predicated, through the use of technological devices that have the potential to grant universal access to the making and consumption of music’ (Messina and Aliel 2019: 693).

Overall, Ntrallazzu 4 and Ntrallazzu 6 benefit respectively from indirect and direct applications of ASC. If, in Ntrallazzu 6, the conceptual premises of the piece radically become the same textual/sonic material upon which the piece itself is predicated, in Ntrallazzu 4 these same premises inform verbal negotiations that end up shifting the very functioning of the piece. In both cases, we propose the rubric of ‘Radical ASC’ to describe the operative performativity of semantics in shaping the musical and epimusical characteristics of the creative processes, to an extent that transcends the pertinence of the ‘traditionally notated’ instructions.

Having addressed the key issues related to Radical ASC that emerge from the creative processes employed in Ntrallazzu, we now summarise the procedures adopted in the creation of the video-dance Atravessamentos. Given that a discussion of all aspects of this piece would fall outside the scope of this paper, we provide an example of one of the versions produced during the creative process and focus our analysis on the topics directly related to the proposed methodological framework.

3. STUDY 2: ATRAVESSAMENTOS

Two aspects of the application of eco-based ubimus practices are addressed in the soundtrack of the video-dance Atravessamentos (Video Example 3) (Aliel and Keller 2021). The compositional strategies are based on fixed imagetic material. But the adoption of video footage as a compositional resource does not necessarily constrain the collaborative dynamic of the aesthetic decisions. From the start, the composers avoid a simplistic Foley-oriented image-sound mapping procedure. Through a set of contextualised and flexible sound-processing techniques, the project engages with the conceptual issues that emerge from adopting eco-grounded methods in ubimus, highlighting the requirements of asynchronous collaboration in multimodal creative contexts.

3.1. Short description of the creative product and discussion on ASC

From the beginning of the video to 0:27, the screen remains completely dark (scene 1). The footage lasting from 0:27 to 1:04 features a dancer lying on dry leaves, wrapped in a veil, amidst a dark forest (scene 2). The first interaction of the dancer with the forest is featured on scene 3, with small gestures behind the forest bush (1:04–1:45). From 1:45 to 1:58 (scene 4), the dancer’s face is hidden by plants and shadows. Scene 5 (1:59–3:01) highlights the dancer’s interactions with a tree from different angles – a far video shot and a close shot. The dancer interacts incisively, as if wishing to get inside the tree. The footage from 3:02 to 3:49 (scene 6) explores extensively the use of body gestures. The camera is kept distant from the subject. From 3:50 to 4:53, body gestures become complex and accentuated (scene 7), suggesting that the dancer is no longer entangled with, but confronted by the pervasive presence of the forest. In scene 8 (4:54–5:18), the dancer seems to establish a symbiotic relationship with her surroundings. The section lasting from 5:19 to 5:52 features playful gestures (scene 9). Scene 10 (5:53–6:43) features a change of scenery, with the dancer standing in front of a river. The gestures seem to converge with the rhythm of the natural elements (e.g., sound of rain). The last scene hints at the dancer embracing nature: the body remains fully naked and static. Scene 11 (6:64 until the end of the footage) features a moving shot of the canopy that serves as a visual background for the artwork credits. ²¹

Based on research in ASC, Atravessamentos employs visual events that can be interpreted via semantic timbral descriptions. Keywords were used to determine the broad aspects of each scene. These characteristics guide the timbre choices. The keywords for the scenes were consensually chosen by both composers. They provided a summary of the visual contents aligned with the timbral targets of the creative process (Table 1). This strategy served to establish cross-modal aesthetic decision-making strategies within the context of radical ASC.

Table 1. The keywords-timbre association by individual scenes in Atravessamentos

3.2. Procedures

The footage was delivered semi-finalised. Final edits were pending, thus resulting in abrupt cuts of some of the video scenes. Consensually, the composers decided that a Foley soundtrack would not be feasible. For instance, the abrupt transitions of the original footage could hinder the ecological validity ²² of the sonic content (Keller 2000). Occasionally, the edits do not align with the gestural elements of the dance. Instead of treating these inconsistencies as negative features, the creators decided to highlight the abrupt transitions through sonic markers. Thus, some of these cues serve as pillars for the sonic structure and do not necessarily match the temporalities of the video (see further comments on issue 1 in the following).

Both composers engaged in a collaborative process throughout the elaboration of the piece. Each one had 48 hours to produce and hand over the material to the other participant. Work on the existing material was done through modifications, additions or removal of content. This procedure was iterated six times until the result was fit for evaluation by the video producers. After receiving feedback from the project producers, small temporal adjustments were carried out to align the visual and the sonic contents.

The compositional procedures feature three aesthetic goals: 1) consistency with the dance movements; 2) consistency with the video-cuts and with the camera positions of the video shots; and 3) usage of the constraints furnished by the sonic resources. This mix of creative strategies targets: 1) a musical piece with a simple but not necessarily simplistic structure; 2) the use of structural articulations based on the points-of-view adopted in the video shots, serving as referents for the sonic processes (e.g., 1:45, close-up of the dancer’s face and 4:53, scene of the samaúma tree); 3) the use of acoustic-instrumental synthesis techniques for the excerpts with complex choreographies (scenes starting at 3:50 and 5:52 of version 4).

3.3. Radical ASC in Atravessamentos

Atravessamentos exemplifies three strategies that may be employed within the context of Radical ASC methods: biophonic gridworks, acoustic-instrumental synthesis and cognitive dissonances. Cognitive dissonances use cross-modal conflicting references to challenge the listeners’ expectations. Acoustic-instrumental emulations serve as resources for an expanded sonic palette that bridges biophonic and instrumental sources. Biophonic recordings are also employed as structural pillars of sonic organisations and as a way to situate the work within the Amazonian landscape.

3.3.1. Biophonic gridworks

Recordings of natural sounds that hint at environmental processes provide the starting material for several structural decisions in Atravessamentos. These raw elements feature rain, insects and wind among a wide variety of local sonic sources. While the adoption of a Foley-based strategy involving a linear mapping of sound events to video-cuts and dance gestures would present serious limitations (see section 3.2), an abstract soundtrack would shy away from a challenge set by the image of a dancer at outdoor, ‘wild’ settings. ²³

Our use of the signifier ‘wild’ here deliberately refers to a multiplicity of entrenched stereotypes about Amazonia. Contrasting with a supposed ‘modernity’ (cf. Dussel 2000), imagined as a prerogative of a handful of global centres situated in the Northern hemisphere, the Amazon region is usually construed as a staunch stronghold of ‘backwardness’ (Messina 2016). This discourse is produced and reproduced both by the Brazilian media (spreading from large urban centres) and by mainstream international media. A series of dichotomies is operative in this discourse. ‘The wilderness’ is set against the allegedly ‘civilised’ hegemonic centres. Similarly, ‘the ethnic flavour’ of the Amazonian cultural, artistic, scientific and epistemic production is presented as antithetical to the ‘universality’ of European knowledge. Given this context and to avoid the reproduction of trite clichés, ²⁴ Aliel and Keller opt to lay out a gridwork of biophonic sources as a sonic organising principle. This gridwork is explored through sonic bridges between the local referents and the processed materials. We shall delve into three compositional strategies to exemplify this approach.

3.3.2. Acoustic-instrumental synthesis as an ecological modelling strategy

Since Keller and Truax’s (1998) initial implementations, ecologically grounded synthesis and processing techniques have been applied in various contexts – ranging from multimodal and multimedia installations (Basanta 2010; Capasso, Keller and Tinajero 2000–20) to audio haptics and textural synthesis (Barrass and Adcock 2002). Atravessamentos makes use of ecological modelling to expand the available sonic resources while keeping close ties to everyday events. A specific contribution of this piece is the incorporation of acoustic-instrumental synthesis within the context of ecologically grounded creative practices.

Atravessamentos features sonic-generative techniques based on acoustic-instrumental sources. These emulations are produced by decoding the dynamic spectral structure of the recorded sources to recreate or emulate acoustic-instrumental sounds, consequently yielding an expansion of the instrumental palette. ²⁵ Audio-processing techniques are employed to enhance the perceptual qualities of the recorded materials through timbral bridges linking the synthetic and the recorded resources. Reverberation provides distance cues that sometimes are consistent with the imagetic content and other times contradict the visual cues. ²⁶ In Atravessamentos the instrumental sounds furnish dense sonic textures, yielding dynamic clusters and glissando-based sound blocks. These generative techniques engage with cognitive dissonances through their harsh contrast with the sonic content of the biophonic scenes.

3.3.3. Cognitively dissonant strategies

For creative purposes, cognitive dissonances involve the use of conflicting matches among elements of various modalities, targeting the distortion of perceptual cues and possibly leading to a larger set of aesthetic relationships among images, sounds, behaviours and chronotopical cues. Complementarily, Yoganathan (2017) proposes simple mixing strategies – involving the usage of geographically disparate sonic cues – to explore the conflict among renditions of soundscape recordings, labelled artificial ecotones. Cognitive dissonances have been applied by eco-composers in various works (Keller 1999; Keller and Capasso 2006). But a careful assessment of their impact is still pending. The last section of this paper includes pointers to possible avenues of development of this proposal within the context of ubimus research.

Both the images and the sonic materials of Atravessamentos feature cognitive dissonances at various levels. A key element is the dancer’s immersion in the West Amazon forest scenery as an unusual phenomenon. This disruption of the scenery suggests the use of sonic resources foreign to the original context, hence ‘breaking’ or suspending the listener’s expectations. Images of rain are not present until the last video scene (featuring the naked dancer at a margin of a river). The sound of rain serves as a macro-level scaffold: it is at the foreground of the initial section and is also featured throughout the final section of the work. At a meso-level (Keller and Truax 1998; Keller 2000), it serves as a ‘percussive temporal grid’ to set the timings of some of the salient sonic events. The rain texture also furnishes a timbral bridge between the biophonic and the synthetic materials. Given that the string pizzicatos and the synthesised glass sounds are unlikely elements within the context of the Amazonian biophonic soundscape, they become good candidates to enable cognitive dissonances. However, the intended perceptual paradoxes demand a consistent set of geo-location references. These local referents (described earlier as anchors) are established by the use of recordings of the Amazonian forest soundscape.

4. RADICAL ASC: CONTRIBUTIONS AND FUTURE ENDEAVOURS

Radical ASC strategies are a family of approaches that expand the use of semantics for music-making. By means of open-ended negotiations among the creative partners, they help to avoid the hierarchical division of labour typically enforced by the twentieth-century genius-centric approaches. Through a shift of the sonic referents, they playfully engage with the listeners’ expectations and as a consequence they foster changes in the underlying frames of reference. We highlight two issues addressed in the two projects documented in this paper, which may be targeted in future ubimus research centred on the ‘politics of doing’.

4.1. Issue 1: Expanded temporalities

Both projects indicate a flexible handling of temporalities while dealing with the tensions emerging from the components of the ubimus ecologies. The process of ‘hijacking’ introduced in Ntrallazzu questions the centrality of the composer as the only source of meaningful aesthetic decisions. Within the context of Radical ASC, the semantics of the epimusical and musical resources constitute a key asset for creativity. As a result, the aesthetic decisions become closely tied to the local resources. Hence, this strategy can be classified as a form of adaptive ecology (see Keller et al. 2014).

4.2. Issue 2: Applications in domestic ubimus

A key contribution of ubimus is the expansion of the available venues of creative endeavours highlighting the opportunities provided by everyday settings. This new context for musical activities modifies the sets of demands for the creative practices. Radical ASC strategies may furnish answers to the stakeholders’ expectations in everyday contexts, especially for activities done at home. Territorialisation and spatialised politics are relevant to domestic creative endeavours, especially when the distinctions between the spaces dedicated to working activities and the spaces for leisure and family affairs are blurred. The politics of everyday territorial and temporal negotiations among household members highlights a potential impact of territorialities and temporalities on everyday musical creativity. How can ubimus contribute to the urgent demands for distant socialising at home? One approach involves tailoring the usage of epimusical resources for non-experts. For instance, food-related activities have played an important evolutionary role in socialization. Recent research indicates a potential expansion of distant socialising through musical collaborations that feature gastrosonic resources. ²⁷

4.3. Beyond the aural

As a final remark, extreme cases of Radical ASC strategies, where agents facilitate the production of creative outcomes through linguistic meanings, may become important and possibly the most pertinent musical parameters for some contexts. Spoken-word pieces conceived from an experimental-music perspective illustrate this approach. Consider, for instance, Lauren Redhead’s album solo speaking (2016), a compilation of pieces that use articulated verbal language as a primary element for the creative processes. Semantics supersedes phonetics in the aesthetic decisions, from both a compositional and a performatic ²⁸ point of view (cf. Zaldua 2014; Hunter 2016; Messina 2017). Slightly earlier examples are ‘My Name is Sarah Simpson’ (Parry 2010) and ‘Rock’s Music’ (Landy 2010) from the speech-art album Playing with Words: An Online Compilation (Lane 2010).

The three threads discussed in this paper trigger a further reflection: Radical ASC may be seen as promoting a shift within musical interaction from a fixation with the sonic outcome to the crafting of meaningful musical experiences that lie beyond the aural. In this sense, ‘the material’ in its most traditional score-, instrument- or soundtrack-based manifestations ceases to be a privileged conveyor of politically oriented meaning, as it has often been the case in the experimental or avant-garde practices (cf. Redhead 2015; Spahlinger 2015). Through Radical ASC, ubiquitous musical practices turn their attention to the semantics of verbal languages – the expanded accessibility, diversity and inclusiveness of this framework may perhaps justify the label ‘radical’.