From Site-Specific Sampling to Gamification: An exploration of performative engagement with the environment

Abstract

This article explores strategies that allow electronic music performers to engage their audiences and environments in live acts of co-creation. We outline our existing musical practice relying on site-specific sampling and digital mobile technologies that have been tested across a range of participatory music performances. Salient challenges within this performance context are identified and several tools and techniques are proposed as solutions. We then consider how setting-based and sample-based participatory performances can be expanded through gamification strategies. After exploring how notions of playful experience can offer new insights into the nature of audience engagement, we propose several approaches for introducing gamified elements into performative music practices that can expand the scope of audience participation while preserving key aspects of using concert location recordings and musical improvisation. We conclude by discussing the implications of these approaches for the performer–audience relationship and the prospect of musical engagement with the environment before suggesting directions for future research.

1. INTRODUCTION

This article explores contemporary performative electronic music practices that aim to engage concertgoers as active participants in the creative process. We seek to address the question of how mobile music creation tools such as iOS apps and portable recording devices can be used to reflect the performance environment and broaden audience participation in concert settings. Our observations are structured around the performance practice of one of the authors. However, in seeking ways to broaden the participatory techniques, we highlight selected areas where integration of musical play, games and gamification can be used to expand our existing performative experiences.

This article focuses on music technology-driven engagement with the sonic landscape that represents expressions of indoor or outdoor life. We analyse our past experiences of incorporating selected techniques and strategies focused on broadening audience participation in electronic live performance settings. We then suggest ideas inspired by musical gamification to further expand our discussion. Our analysis is based on experiences from a range of public live events such as iubar project performances in Melbourne, Australia at the Tilde New Music Festival 2017 and the Fringe Festival 2017 as well as in Warsaw, Poland at the Hashtag Lab 2023, ¹ and also KOshowKO performances at the Brunswick Library 2017 and the Darebin Music Festival 2019. ² We also teach similar performative techniques to undergraduate students at the University of Newcastle, Australia and these provided further opportunities to observe their practical implementation.

2. BACKGROUND

2.1. Setting-based mobile music creation

We examine the notion of how an audience can actively participate as a collaborative agent in live performances, with a focus on improvisation. This requires the artist to explore the unfamiliar and, in the context of participatory performances, to negotiate this process with the public. We agree with Eno’s assertion that when improvising, ‘people are deliberately putting themselves at risk in a way. Soaring out into the unknown and somehow dealing with it’ (Eno 2021). Eno contrasts the novelty of this process with a common occurrence of seeing a performer repeating something they have done many times before, the latter lacking the dramatism and uniqueness of improvisation.

Looking at historical predecessors in site-sampling and environmental mobile performance, Golan Levin’s 2001 piece Dialtones – a Telesymphony is a canonical example of audience participation before the iOS era (Lee and Essl 2016). Audience members registered their mobile phone numbers, received new melodic ringtones and took assigned seating. The performers selectively dialled the audience’s mobile devices in a sonic choreography distributed throughout the concert environment. Each phone activation was visually reinforced via overhead lighting and an on-stage display, creating a sense of personal recognition for audience members and allowing indirect participation in the performance.

An example of a mobile phone concert sharing common conceptual ground with our approach is Justin Cromwell’s piece Self-Spoken performed in 2009 at the University of Michigan using a custom-built looper app in the urMus environment (Essl and Lee 2017). The concert involved 8–10 performers engaging with the audience to capture field recordings for an improvised piece exploring the unpredictable nature of sampling and performer–audience interaction. Earlier iOS-based practices with commercially released software, such as the no longer available MadPad app, combined live recordings of audio and video clips and were designed with performance in mind (Kruge and Wang 2011).

Our observations and discussion concerning the use of mobile music equipment in the process of site-specific sampling refer to the creative practice that reflects the Model of Setting-Based Mobile Music Creation (Koszolko 2021) where performative music-making with handheld mobile equipment is exemplified by six elements that represent various aspects of musical storytelling with mobile tools. These elements include: pre-production, algorithmic composition, sequencing, instrumental performance, mixing and effect processing, and post-production (ibid.). The incorporation of mobile equipment, such as tablets and phones running iOS and various apps, capitalises on their affordances, including accessibility, flexibility, performance, acquisition of knowledge, sequencing and mobility (Koszolko 2019: 189).

Mobile music creation technologies used in our performative projects include battery-powered gear: iPads (of various generations), iPhones (of various generations), portable grooveboxes and workstations by Teenage Engineering (OP-1, PO-33, PO-35, OP-Z), and a standalone effect processing unit by Roland (Voice Tweaker E-4). Tablet and phone devices run various apps mentioned in Section 3. Our recording process involves built-in microphones and a Zoom iQ7 stereo microphone that connects to mobile devices via a lightning connector and enhances recording quality. In situations where equipment has to be limited due to portability or time constraints, tablets and phones running iOS apps are prioritised for their multifunctionality.

Site-specific sampling discussed in this article is a form of field recording performed with portable, battery-powered devices. These types of recordings concentrate on documenting the sound of a chosen location and can be used for sonic mapping and archives (Schafer 1994) as well as soundscape ecology studies (Krause 2015). Field recordings have been used to facilitate collaborative creativity structured around documenting and remixing the sounds of urban spaces (Freeman et al. 2011) of which an active example is the Cities and Memory project (Droumeva 2021). A related field to setting-based mobile music is locative music, where personal musical soundscapes can be created by interactions with elements of the environment. The sound is created by pre-determined mappings responding to input from wearable sensors measuring data such as noise levels, light intensity, object proximity and accelerometer. One example of this approach, pre-dating the iPhone era, is Sonic City – a wearable system designed for sonic interactions with the urban environment (Gaye, Mazé and Holmquist 2003).

Unlike with the documentary field recordings focused on purity of the recorded material and algorithmic processing of the locative music examples, our practice involves a range of performer-led manual manipulation of the recorded material achieved by performative sound processing during live music events. The question of how much intervention into the recorded material is required remains vital for any type of field recording. Norman’s observation about sound recordists and anthropologists guarding against their deciding presence applies here, as even the choice of what to record and when signals the ambiguity of the sampled material (Norman 2004: 62). And while the manipulation of field recordings inclusive of the content generated by concert audiences can make this ambiguity significantly more pronounced, it is usually done to serve the music created by the artist. An example is turning speech samples into percussive audio.

The act of performing music is intrinsically connected to the social and physical environment of performance and this connection leads to complex synchronies (Waters 2007; Green 2011; Borgo 2022). Engaging concert audiences in live music performance is connected to creating place-based sounds associated with the venue and its inhabitants. As noted by Bennett (2022), individuals often associate the significance of music with local venue landmarks that have personal and emotional meanings. Consequently, making music with sounds created in places that have personal meaning to concertgoers is connected to the notion of cultural memory.

2.2. Play, games and music

Play is crucial to human culture (Huizinga 1955), pervading our work, education and other spheres of contemporary life (Raessens 2006; de Lange, Raessens, Frissen, Lammes and de Mul 2015). Music is no exception, being characterised as intrinsically playful by scholars of play (Huizinga 1955), games (Bogost 2011) and music alike (Austin 2016; Kassabian and Jarman 2016). The ubiquitous music manifesto (Pimenta, Keller and Lazzarini 2014) proposes that ‘creating one’s own music’ satisfies a basic human need for play, and so ‘playfulness’ becomes a ‘critical quality for success’ (Pimenta et al. 2014: xvii) when engaging novices to participate in activities such as mobile music creation or live audience participation. Despite this alignment, limited research applies established theories of play and games to the context of site-specific or sample-based participatory performances. We suggest that a deeper exploration of play and games within this context can: 1) offer new understandings of how and why audiences engage with participatory musical performance; and 2) highlight new strategies for expanding setting-based and sample-based practices through ‘gamification’.

‘Play’ and ‘games’ are nebulous and closely entangled phenomena that warrant some disambiguation. The notion of ‘play’ encompasses not only free-form and formal activities (Caillois 1961) but a broader mindset applicable to near any situation, system or object (Fullerton, Swain and Hoffman 2004). Most scholars disavow the possibility of ever reaching a general definition of ‘play’ (Grieshaber and McArdle 2010; Kassabian and Jarman 2016; Moseley 2016) and focus on charting its varied forms, characteristics and associated activities. The Playful Experience (PLEX) framework (Arrasvuori, Boberg and Korhonen 2010) is one comprehensive taxonomy, assembling interdisciplinary views to systematise play into 22 categories of experiential phenomena such as expression, discovery, competition, fellowship and subversion. Each ‘playful experience’ represents a distinct emotional or motivational appeal for participants, promoting engagement within a given activity or context. We suggest the PLEX framework as a productive lens for apprehending the multifaceted nature of audience engagement with participatory music performances and later explore its application in an analysis of our own sample-based participatory practice (Section 4.1).

‘Games’ are a subset of ‘play’ and are similarly resistant to essentialist definitions, which tend only to encircle a subset of all phenomena described as ‘games’ (Wittgenstein 1953; Arjoranta 2014; Aarseth and Calleja 2015). Our interest lies in ‘gamification’ (Deterding, Dixon, Khaled and Nacke 2011), the use of ‘game design elements’ in otherwise ‘non-game contexts’ to increase enjoyment, motivate participation, or lower barriers to entry. ‘Game design elements’ are those ‘characteristic’ to ‘most games’ (ibid.), which can range from high-level system qualities such as ‘rules’, ‘conflict’ and ‘quantifiable outcomes’ (Salen and Zimmerman 2003: 80) to more granular features such as ‘points’, ‘teams’, ‘leaderboards’ and ‘quests’ (Buckley, DeWille, Exton, Exton and Murray 2018: 107). Introducing such elements into a musical activity, such as ranking audience members based on their creative contributions to a participatory performance, would constitute a gamification strategy. While our current works have yet to employ such approaches, we explore the potential for gamification strategies to extend our performance practice by expanding the scope of audience engagement with site-based and sample-based music-making (Section 4.2). Our choice of gamification also supports key conceptual concerns: first, its use of mere elements from games for a delimited purpose (e.g., encouraging participation) ensures that the performance activity remains primarily musical; second, its technology-agnostic focus on design elements (Deterding et al. 2011) preserves flexibility in the musical technologies available for engaging the audience and landscape.

2.3. Gamification in mobile music and participatory performance

Games have been historically used to facilitate musical performance, improvisation and composition, particularly to involve musical novices. This history includes eighteenth-century musical dice games (Hedges 1978), early music video games such as Otocky (Iwai 1987) and SimTunes (Iwai 1996), and participatory installations such as The Music Table (Berry, Hikawa, Makino, Tadenuma and Suzuki 2003) and reacTable (Kaltenbrunner, Jordà, Geiger and Alonso 2006). Today, the trend continues with mobile-based ‘sound toys’ (Dolphin 2014) and playful music-making apps (Kassabian and Jarman 2016; Wang 2016) such as Biophilia (Björk 2011) and Scape (Eno and Chilvers 2012). These apps not only lower technical barriers for novices but engage them through ‘disarming’ and ‘game-like’ qualities (Wang 2016: 186), which reframe the activity as intrinsically motivating and enjoyable. Increasing engagement, motivation and access in this manner is a common aim of gamification strategies founded upon the notion that gameful experiences appeal to our base psychological needs. Buckley et al. (2018), for instance, chart the relationships between individual game elements and our needs for competence, autonomy and relatedness as posited by self-determination theory (Ryan and Deci 2000), while Arrasvuori et al. (2010) link certain playful experiences to the desirable ‘Flow’ state (Csíkszentmihályi 1990). We suggest that gamification strategies have a comparable potential to further the aims of audience engagement and inclusion within participatory performance contexts.

Few musical performance works for large-scale, mobile-based audience participation have harnessed gamification strategies. While the few exceptions exemplify the participatory potentials of gamified approaches, they tend to adhere to paradigms of pre-defined sonic resources over live sampling practices. In Sound Games 1 and 2 (Gimenes 2018), audience members manipulate networked mobile devices to produce tones from constrained harmonic sets. Participants can earn ‘points’ by playing in time with others or using ‘leading’ instruments acquired by moving close to Bluetooth emitters distributed throughout the performance space. A similar experiment was conducted with the mobile app echobo (Lee and Freeman 2013), wherein networked audience members improvise with constrained eight-key instruments. They can record short phrases and broadcast them to fellow players, which if performed correctly will spread to others in a viral collaborative interaction. Also related is Crowd in C[loud] (Lee, Carvalho Jr and Essl 2016), a live-coded participatory performance using a gamified ‘dating app’ metaphor for intra-audience interactions. Participants compose a five-note melody as a personal ‘profile’, browse others’ profiles, audition their melodies and award ‘likes’ to collaborators. A real-time leaderboard displays total ‘likes’, including a graphical commendation for the ‘most liked’ player. What emerges from each of these gamified strategies is a unique meld of cooperative and competitive social interactions that not only motivates the audiences’ creative participation but also organises their individual musical engagements into a cohesive work. They do not, however, seek to engage their environments via any site-specific sampling practice.

Other performance works have used game elements to support participation without relying on the audiences’ own mobile devices. Urban Musical Game (Cera 2013), for instance, is a ‘music-enhanced sport’ using a physical ball with motion sensors for real-time audio synthesis. Bowls (Bown and Ferguson 2016) similarly utilises accelerometer data from distributed audio devices to generate music as participants roll and throw them in a lawn bowls game. Like their mobile counterparts, these works empower untrained audiences to manipulate sonic outcomes through familiar game interactions, but do not engage with any environment or audience sampling practice. This trend may be due to the technical challenges of live sampling in performance contexts, which we discuss in the following section. For the performer, eschewing audience recordings in favour of defined timbral and harmonic constraints allows for a more reliable musical outcome. Audiences may find live recording interaction more intimidating than working with the simplified interfaces and affordances of mobile music apps, which are often designed to preclude sonic ‘errors’. These questions and challenges warrant further examination through performance experiments and audience surveys. Until then, we can explore the theoretical potentials of applying playful and gameful strategies to the as-yet novel context of setting-based and sample-based participatory performance.

3. PERFORMATIVE ENGAGEMENT WITH THE ENVIRONMENT

3.1. Aspects of pre-production

Electronic music performances often require pre-production tasks such as recording audio and MIDI loops and structuring arrangement elements for use as a backing track. In our Setting-Based Mobile Music Creation, iOS apps such as AUM – Audio Mixer, DAW apps such as Gadget and BeatMaker, groove boxes such as Yellofier or Blocs Wave, samplers such as Koala Sampler and FieldScaper and granular synthesis apps such as iDensity and SpaceCraft are used for these tasks. The choice of app depends on workflow preferences, which in turn might suit various musical styles. Pre-production often focuses on sequencing and editing of audio parts featuring instruments and samples used by the performer. These audio tracks can then be used to create harmony layers and be processed live with various effects.

To extend audience participation, submissions of mobile phone recordings can be accepted prior to an event. These can be verbal or non-verbal responses to a set question related to the performance theme. For example, one week before the iubar project’s performance at Hashtag Lab in Warsaw, audience members were encouraged to upload their voice recordings responding to a set of questions exploring notions of collective and individual work. The same questions were asked at the venue during the event to enable live sampling. Collecting responses in advance allowed for more precise signal processing, addressing some of the challenges discussed in Section 3.4. This pre-production approach provided additional time to consider integration of the submitted responses in the software and accommodate various musical elements, such as specific rhythm track tempos, ambient sections, or audio pitched to match past compositions.

Our pre-production practice includes sourcing field recordings before the performance for incorporating this material alongside samples from the performance venue. This juxtaposes reflections of nature and/or urban life audible in the field samples with the settings and samples of the indoor concert venue. The process of landscape imitation in music performed live has been compared by Schafer (1994) to opening the windows and showing non-city images to the spectator. Field recordings allow for further expansion of this approach with performative mobile technologies enabling various forms of live modulation.

A technical aspect of pre-production is synchronisation and connectivity. When using multiple mobile devices, an important early consideration is the setup that enables clock synchronisation. There are multiple approaches to implementing MIDI connectivity for mobile equipment (McGuire 2020). The iOS apps increasingly feature the Ableton Link protocol, which we commonly use as it enables wireless synchronisation of apps on multiple iOS devices over WiFi. For more complex setups involving non-iOS equipment, a MIDI interface and cable-based connectivity are typically implemented.

Site-specific sampling is a crucial element of the event, which occurs before the improvised live performance. It is part of the pre-production stage in the Model of Setting-Based Mobile Music Creation (Koszolko 2021). Live sampling of audience members is an effective engagement strategy that enables a range of gamification strategies discussed in Section 4. Furthermore, it allows the performer to introduce narrative or conceptual intent and engage the audience in a form of creative discussion.

3.2. Improvisation, composition and performance

Improvisation is a form of composition (Borgo 2022) and, as stated by Fein (2017: 1), ‘Improvisation is composing in the moment with restrictions.’ These restrictions are typically the rhythm and harmony of the performed piece. Improvising performers also work with limitations that include their technical ability, ear development and music theory knowledge (ibid.). Live sampling presents additional unpredictability as to whether members of the audience will be willing to participate in the sampling process and produce sounds by singing, speaking or perhaps, tapping objects available in the venue.

When improvising with newly created, site-specific recordings, we have explored strategies that result in the creation of new compositions using only location recordings. Alternatively, we have also incorporated such recordings into a pre-existing musical skeleton. The complexity of this skeleton can vary, ranging from a fully formed arrangement with well-defined sections to a simple loop or a set of loops that serve as the foundation for live composition. From a compositional standpoint, the biggest challenge lies in the need to generate harmonic or melodic content using samples that were created within the constraints and challenges discussed in Section 3.4.

The performative approach taken by us to date was structured around live sound modulation and live performance with a range of electroacoustic instrumentation. Live sound modulation is a powerful aspect of improvised live performance with sampled material. In the context of an iOS-based approach, it is an element of affordances of workflow and performance (Koszolko 2019). Modulation can be achieved by implementing a range of audio processing effects that are built directly into performance apps such as Koala Sampler, SpaceCraft and Yellofier, or are used as inserts in a mixing app such as AUM. Another approach is to use external effect processors receiving a signal from devices storing the samples. The implementation of effects as performance tools enhances the compositional aspect by limiting repetition within loop-based material and creating more diverse soundscapes. Examples of commonly used types of effects include pitch shifter, granuliser, bit crusher, filters, gate, stutter, reverb, delay, reverse, distortion and flanger.

While many of the preceding compositional elements and sound-processing tools are well-known to musicians operating in production studio settings, the process of improvising and creating music live in front of an audience is significantly different from making music in the confines of a studio. Apart from creating and using new site-specific sonic material, there are many functions to control at once and in front of, as well as with, the participants. This kind of responsibility can be daunting and requires advanced planning, responsiveness to the challenges outlined in Section 3.4, and a high degree of fluency with the used technologies.

3.3. Example performance design

To provide more details on the technical and aesthetic framework of a performance focused on site-specific sampling, we will use KOshowKO’s performance at Darebin Music Festival 2019 as an example. The overarching aesthetic goal was to create a cohesive set by blending sonic improvisation based on samples collected on site with elements from the artist’s existing repertoire, as featured on the Mobile Positions EP (KOshowKO 2021). The resulting sound blended elements of electronica and musique concrète. In the pre-production phase, additional field recordings were made on Melbourne trains, and performance-ready versions of selected arrangements from KOshowKO’s repertoire were prepared using the apps Egoist and SpaceCraft Granular Synth.

The public was engaged by introducing the sample-gathering approach and inviting audience members to volunteer for recording. Site-specific sampling followed, and the audience was asked to present any sound as a form of personal expression. The artist approached willing individuals with an iPad Air 2 using its built-in microphone to capture singing, speech and found object sounds (see Figure 1 for images from other performances employing the same approach). The sampling process was completed in approximately 3 minutes. All samples were directly recorded into the Koala Sampler app. Once all samples were gathered, the artist performed quick edits on stage, such as trimming and adjusting audio levels.

Figure 1. One of the authors engaged in site-specific sampling with concert audiences.

The next step involved connecting the iPad to the venue’s sound system for the music performance. The sampling iPad, running Koala Sampler, Aparillo and SpaceCraft Granular Synth apps, was wirelessly synchronized with two other iOS devices through Ableton Link. These devices included an iPad Pro (model 7,4) running the Egoist app and an iPhone 6s running the AUM app, with additional samples and effects, and the ThumbJam app. Each device was connected to a separate audio input on the venue mixer. A multidevice approach was implemented to provide simultaneous access to more controls and minimise the need for app switching on a single mobile device.

Most of the mentioned apps offered sample-processing capabilities, influencing the aesthetics of the performance through live sample manipulation. This approach varied across apps, encompassing granular synthesis, sample sequencing and real-time effect modulation. The inclusion of location samples introduced dissonant harmonies alongside irregular rhythms of KOshowKO’s music, which were partially pre-programmed and partially processed live. This allowed for the creation of a sense of tension and unpredictability.

In our example, samples recorded by the artist signified performative engagement with the audience and sampling of two environments: the performance venue and city trains. Software synthesisers Aparillo and ThumbJam were played live, the latter also featuring sampling functionality. ThumbJam was played on the iPhone, taking advantage of the device’s small form factor for a dynamic approach that utilised finger gestures to change panning and the iPhone’s built-in gyroscope and accelerometer to add vibrato and tremolo through shaking, as well as apply pitch bend when adjusting the vertical tilt. This iPhone performance showcased the expressive qualities of the music and signalled the intention to surpass the limitations of pre-programmed electronic elements in the arrangement.

3.4. Salient challenges within site-specific sampling practice

The suitability of captured material for the intended or existing composition can be enhanced by providing clear guidelines to the audience before recording. For example, the audience can be asked to hum specific notes along with a guide offered in the form of a note sung or played live by one of the artists. This technique is more easily achievable for performing groups with at least two members, where one member provides the musical notes while another is among the audience with the recording device. Another approach could involve providing other guidelines to the audience, such as asking for verbal responses to questions posed by the performer(s).

Site-specific sampling is conducted live, presenting technical challenges that can impact the musical aspects of the performance. These challenges stem from imperfect recording environments and the diverse approaches of participating audience members. Some performance setups involve multiple devices and apps running simultaneously, increasing technological complexity and app interconnectivity. Table 1 provides a summary of the most common technical issues encountered and proposed solutions. The table primarily pertains to our use of mobile music creation technologies as outlined in Section 2.1 and does not cover computer DAW solutions.

Table 1. Salient challenges within site-specific sampling practice and proposed solutions

Note:

* indicates solutions utilised in KOshowKO’s 2019 performance, detailed in Section 3.3.

Implementing the proposed solutions is critical to ensure the creation of coherent musical pieces with minimal technical issues. Some challenges arise from the limited time available between recording and the start of the performance. Performers are required to have a strong command of the technology, as decisions must be made quickly and they may be difficult or impossible to reverse. The difference between the proposed simple and more complex solutions lies in their impact on immediacy and requirements for additional support personnel. Some of the simpler solutions are more viable for a single performer using recording and sequencing apps. In such cases, multiple samples collected live reside within a single app, which provides only stereo output, without the option for individual sample processing with external effects.

The distinctive functionality of iOS-based sampling tools is evident in their self-contained nature. Tablets or phones with built-in microphones are easily manoeuvred in various venue settings. Once samples are captured among the audience, the equipment can be quickly connected to the venue’s sound system via an audio interface or built-in audio output. This self-contained environment ensures that the samples are stored on the recording and performance device, eliminating the need for transfer to another digital location. When implementing simpler solutions from Table 1, recorded samples can be promptly edited in the apps used for recording, reducing setup and waiting time before the performance begins.

4. GAMIFICATION STRATEGIES

4.1. Exploring audience engagement as playful experience

Before outlining gamification strategies for extending our future practice, it is useful to consider how the ‘playful experiences’ described by the PLEX framework (Arrasvuori et al. 2010) can offer an accounting of audience engagement within our existing participatory performances. Some experiences can be assumed as inherent to most musical performances, such as sensation being the joy of ‘stimulating the senses’ (ibid.: 8), yet there are certain experiences warranting further exploration in our sampling context. Expression, the joy of ‘manifesting oneself creatively’ (ibid.: 7), emerges when a participant ‘leaves their mark’ on the performance through the personal creative decisions made during the conception and recording of their sampled content. This act then prepares for the possibility of discovery, the satisfaction of ‘realising relationships between actions and outcomes’ (ibid.: 7), as participants see the expanded musical potentials of their contribution via our executive transformation and arrangement efforts. Participants also experience discovery in their broader realising of the performance process, as our performance format exposes the activities of recording, auditioning and editing samples in a live opportunity for audience knowledge acquisition. This suggests that playful experiences can emerge across distinct stages of performative music-making, from creative expression during pre-production to the discovery of musical outcomes in later stages of mixing and effect processing (Koszolko 2021).

Another playful experience of interest is exploration, elicited by ‘investigating an environment, object, or situation’ (Arrasvuori et al. 2010: 7). In our context, audience participation in the sampling process is an active engagement in a sonic mapping of the performance site, whether exploring the acoustic affordances of the space itself (e.g., voice reverberation) or the timbral properties of one’s immediate surroundings (e.g., tables, glasses). Each act of sonic exploration influences the social ecology of the performance, informing other audience members of existing sonic potentials and inspiring new interpretations or experiments once their turn to contribute arrives. Exploration thus becomes a collective process between all audience members and the performer, eliciting another playful experience: fellowship, the joy of social interaction and of ‘sharing information between participants’ (ibid.: 8). The communality and intimacy of this shared context, wherein audiences explore their sonic environment, express their creativity within it and discover its expanded musical potentials, offers one account of how and why audiences engage with site-specific sampling. In other words, the emergence of so many playful experiences from such simple acts of environmentally embedded improvisation culminates in an activity that is accessible, creatively stimulating, socially enriching, and ultimately motivates participation.

The notion of playful experience offers a productive lens for the design and planning of future works involving audience interaction. One might consider how audience interaction could elicit desirable experiences such as expression, exploration, discovery and fellowship at different stages of the performance. Other playful experiences outlined by the PLEX framework (ibid.), such as fantasy, humour, simulation or subversion, can also be explored in the designed audience interaction. Two experiences, challenge and competition, can be used to propose potential gamification strategies for extending our live sampling-based participatory performances.

4.2. Gamifying performer–audience relationships

Competition, derived from ‘contest with oneself, an opponent, or a system’ (Arrasvuori et al. 2010: 6) and challenge, derived from exercising physical and mental skills to overcome demanding situations (ibid.: 6), can support the emergence of new performer–audience and intra-audience relationships in our participatory performances. We suggest four potential archetypes for such a relationship, as outlined in Table 2. It should be noted that these relationships are founded on a loose conception of a gamified ‘contest’, which does not necessitate the explicit awarding of ‘points’, ‘victors’, or other quantifiable metrics for evaluation; we require merely that a general spirit of contest arise between two or more participating forces.

Table 2. Proposed archetypes for performer–audience relationships in participatory music performances with gamified experiences of competition and challenge

The first strategy for gamifying participatory performances is to create a competition between the performer(s) and audience members. This is apparent in works such as Cello Fortress (van Dongen 2012), where a performer’s live cello improvisations control a virtual fortress that audience members must overcome using conventional video game controllers. ³ Diverse approaches are available for translating this dynamic into our live sampling context where audiences make more direct sonic contributions. For example, audience members might be challenged to improvise samples with surprising, subversive or otherwise difficult content. In response, the performer is challenged to demonstrate their live production skill by taming the samples into a cohesive, interesting or even humorous musical performance. This approach can elicit new playful experiences beyond the implicit notions of challenge and competition, such as humour and subversion (Arrasvuori et al. 2010). Another is derived from repositioning a core technical challenge of live sampling – the quality and musical suitability of recordings (Table 1) – as a driving creative stimulus for the performer and audience alike. This in turn presents a didactic opportunity for transferring a deeper understanding of live production practices to the audience by more explicitly exposing its inherent limitations, challenges and creative potentials.

Participatory performances can also be conceived as competitions between audience members themselves, with the performer(s) acting as neutral musical directors, judges or observers. This relationship is represented to varying degrees in works such as Sound Games 1 and 2 (Gimenes 2018), echobo (Lee and Freeman 2013), Crowd in C[loud] (Lee et al. 2016) and Urban Musical Game (Cera 2013). Within our site-specific sampling context, there are several means and metrics by which audience members might be prompted to contest one another on the basis of their recorded contributions. These could range from subjective ‘awards’, such as nominating audience samples with high quality, novelty or musicality, to more quantifiable metrics such as measuring which samples were used most by the performer. Software and hardware solutions could also be implemented to support these interactions; for instance, an automated monitoring of sample usage metrics (e.g., total duration played, total times triggered) could be used to assign ‘points’ to each sample’s contributor, which can then be displayed on a real-time ‘leaderboard’ (Buckley et al. 2018). The specific metrics or implementations chosen for a performance are of less significance than the intention for introducing dynamics of contest in the first place. Intra-audience competition and challenge can offer a novel means of implicitly influencing an audience to generate more desirable sonic materials and encourage a greater level of genuine effort or willingness to follow performer directions, thereby producing sonic resources more amenable to the live development of a cohesive musical performance.

Competitions between performers are less represented in audience participation contexts, possibly due to increased focus on inter-performer interactions over performer–audience interactions. However, such a relationship can be conceived within our site-specific sampling practice by drawing on ‘game pieces’ such as Iannis Xenakis’s Duel (1959) and Stratégie (1962), where a numerically scored combat is conducted between opposing orchestras with ‘points’ and a victor awarded using pre-determined payoff matrices (Sluchin and Malt 2011). A comparable contest could be conducted using sonic materials generated by the audience. One approach could be to divide our current performance design into two competing performer–audience ‘teams’, with each performer recording a select half of the audience members before pitting their group’s collaborative results against the other’s. Alternatively, the audience can remain unified by acting as judges for an exclusively inter-performer contest. For instance, two performers could gather a common pool of audience samples before competing to produce the live arrangement that best reflects a given set of aesthetic criteria (e.g., ‘abrasive’, ‘tranquil’, ‘viscous’). These prompts could be pre-emptively defined by performers to pursue particular musical outcomes, or devised extemporaneously by audience members as a further opportunity for their creative expression. Participants can then assign a ‘winner’ through either conventional social means (e.g., cheering, a show of hands) or via technology-driven voting platforms linked to a visual ‘leaderboard’. Such approaches would actively engage the audience’s critical faculties while drawing attention to the unique processes and challenges underpinning live mobile music performance.

Another gamified performer–audience relationship is one in which the two parties collaborate to overcome a challenge derived from the performance environment itself. This ‘challenge’ might be taken literally, such as recording in a setting with unfavourable acoustic properties or heavy noise pollution. We have greater interest, however, in considering how the audience’s existing experiences of sonic exploration and discovery of the environment can be reframed as their own conceptual or creative challenge. The performer can provide site-specific aesthetic directions to the audience, such as authentically representing the environment or extracting novel or unexpected sonic content from it. A related approach was explored in the locative work Net_Dérive (Tanaka and Gemeinboeck 2008), which used metaphors such as ‘archaeology’ and ‘air traffic control’ to create an ‘abstract narrative’ for participants and frame their sonified exploration of the environment as a ‘mission’ to be ‘carried out’. These narrative framings invoke the notion of a game ‘quest’ (Buckley et al. 2018) – a specific task or challenge to be overcome – for which the environment becomes the object. This relationship can then be extended by introducing additional game design elements, such as ‘collections’ (ibid.). To give one example, the performer could work with audiences to locate and record live samples from each of a given ‘collection’ of site-specific sound sources (e.g., biophonic, geophonic and anthropophonic sounds) as a shared creative challenge. Such approaches would seek to draw the audience’s attention towards their relationship with the performance environment, their role within its sonic ecology, and the significance of place within site-specific musical practice.

4.3. Further strategies and implications

Further gamification strategies can be applied within the performer–audience relationships we have outlined. One possibility is importing structures or mechanics from known games into the performance activity, such that an audience’s established familiarity can be used to motivate participation, lower inhibitions, improve comprehension and guide musical interactions. Past works outside of our live sampling practice have applied this strategy with varying scope. For instance, audience musical interactions have been designed around established sports such as ‘lawn bowls’ (Bown and Ferguson 2016), individual mechanics from games such as chess (Kramann 2022) and even abstractions of entire genres such as ‘board games’ (Kramann 2020). Harnessing the audience’s prior game ‘literacy’ (Zimmerman 2009; Zagal 2010) in this way can also facilitate the transfer of new musical understandings; Kramann (2022), for instance, illuminates complex polyrhythmic relationships for non-expert participants by sonifying the threats posed between individual chess pieces. Familiar game designs can thus act as new metaphors for creative action (Keller 2018), extending a participant’s capacity for creative planning and execution within performance contexts by reducing the time and cognitive load required to effectively model and engage with the musical activity.

Within our audience sampling context, it would be difficult to transfer the concrete rule structures of existing games (e.g., chess) without restricting or trivialising the intended participant interaction of live recording. We instead suggest adapting more abstract game concepts, such as dividing the performance into multiple structured ‘rounds’ to give audience members the opportunity to be sampled again after experiencing our improvisation with their initial recordings. This format allows audiences to apply the relationships discovered between their first contributions and our live transformations to new acts of sonic expression and exploration, thereby establishing a more active cycle of engagement and feedback between performer and audience. A cyclical round-based structure also supports the dynamics of challenge and competition within the performer–audience relationships outlined prior (Table 2), allowing audiences to more robustly challenge the performer by adjusting their approach to each subsequent recording. Such performances could be framed via metaphors of a musical ‘ping pong’ match, establishing a competitive ‘call and response’ where the creative onus passes repeatedly between performer and audience as they challenge one another to produce increasingly novel sonic material.

Evaluating the effectiveness of gamification strategies in engaging participants and supporting their creative activity requires dedicated experimentation and data collection in real performance settings (Keller 2018). A deeper accounting of how audiences perceive their musical participation, their relationship with the performer and environment, and the influence of gamified elements upon this experience can be approached through direct audience surveys of future performance experiments. Additionally, audiovisual documentation of the audience’s live interactions will be crucial for better understanding the aesthetic outcomes of these strategies, particularly during the recording and pre-production stages when their creative engagement is most explicit.

Our specific concern here is that gamification strategies – despite their creative potentials – might alternatively be viewed as inviting undesirable experiences or perceptions for audiences. Competitive experiences have been framed as being conceptually unconducive to accessible music creation (Dolphin 2014: 46; Kassabian and Jarman 2016: 129). Similar implications were highlighted by Clemente, Falleiros, Tavares and Fornari (2020) in a mobile-assisted free improvisation experiment, wherein saxophone students were asked to perform into their smartphones until an app measuring real-time melodic variation reached a ‘maximum’ level. The students ‘naturally treated’ this session as a game, foregoing any musical intentions to focus ‘only on winning’ by being the first to finish the process (ibid.: 131). Studley, Drummond, Scott and Nesbitt (2022) also observed a comparable effect in a user study evaluating single-player games for real-time music creation, where participants reported that competitive game elements such as ‘combat’ and ‘victory’ attracted a higher priority and distracted from music-making interactions (ibid.: 25–6).

As a precaution, this risk of ‘dampening musical interactions’ (Clemente et al. 2020: 131) via gamification should also be considered in our site-specific sampling context. Audience members might disproportionately focus on a perceived competition, disrupting the performance, inviting social tension, degrading the sample quality, or merely being deprived of the musical experiences intended by the performer. Performers should carefully balance these risks against the engagement potentials of any gamification strategy to ensure it enhances the performance themes rather than overshadowing them. Maintaining a clear focus on experiences of expression, exploration and discovery for audiences can cultivate a playful and productive performance based primarily in musical engagement with the environment. Nonetheless, further performance experiments, audience surveys and audiovisual documentation are needed to sufficiently evaluate these strategies and inform future participatory practices.

5. CONCLUSION

Live music performances present a unique, and yet rarely implemented, creative possibility for engaging concert audiences to become active participants in the act of creating and shaping the performed musical content. Integrating voice samples from the audience is a crucial element in creating identifiable material for improvisation within participatory electronic live performances. This integration provides the audience with an influential creative agency, enabling them to contribute creatively and play a significant role in the overall performance. The recognisable nature of the vocal samples enhances enjoyment and fosters stronger engagement between the performer and the audience. Furthermore, it generates anticipation for the moments when these samples will be played during the event. Audience sampling also helps to convey a narrative or conceptual idea connected to the performance/event. Consequently, the utilisation of audience-acquired voice samples proves to be an essential technique, elevating the impact and effectiveness of participatory performances.

This article has explored a range of strategies that allow artists using primarily mobile digital technologies to engage their audiences and environment in the act of musical co-creation. We identified salient challenges within our site-specific sampling practice and proposed several tools and techniques as potential solutions (Table 1). We also demonstrated how notions of playful experience can offer new insights regarding the nature of audience engagement with setting-based and sample-based participatory performances. We suggest playful experience as a productive theoretical lens for designing future performance interactions of this kind, and for apprehending musical participation by concert audiences more broadly.

While our existing musical practices have been successfully implemented in a range of participatory music performances, we recognise that more could be done to enhance such events. We propose that introducing gameful and playful elements in performative music practices can be a beneficial strategy for expanding the scope of audience engagement while preserving the key aspects of using concert location recordings and musical improvisation. We have outlined a variety of potential gamification approaches to support this, considering their impact on the performer–audience relationship and the prospect of musical engagement with the environment.

Future research can evaluate and extend the strategies explored herein through dedicated performance experiments and audience surveys. A closer account of how participants perceive their co-creative interactions, their relationship with the performer and environment, and the influence of any gamified elements upon this experience can lead to a more complete understanding of audience engagement and so better inform future participatory practices. We also suggest that audiovisual documentation of live audience interactions will be of critical benefit to these initiatives, particularly during any pre-production recording stages.