Off-the-shelf is a mixed reality duet musical improvisation that redefines co-creative artificial musical experiences through a minimal aesthetic dialogue.
Despite continuous advancements of technical complexity and capability in intelligent creative systems [1][2][3][4][5], the exploration of live performance practices involving human musicians and AI in-the-loop remains limited. Inspired by this year’s AIMC theme calling for an interdisciplinary perspective into musical AI research, this performance, leveraging minimalism in both technology and aesthetics, seeks to explore new dimensions of interaction and expression between human musicians and artificial musical agents. By embodying the AI contribution within a spatial computing interface, this work advances existing performing experiences with intelligent musical agents in a mixed reality context.
In this research, we discuss 10-minute performances created using a generative AI system controlling a compact synthesiser (we have performed with a MakeNoise 0-COAST, a Korg Volca FM, and experimented with other instruments), and a human electronic musician performing on a separate synthesiser and viewing the actions of the generative AI system in a mixed reality space. The musical agent, embodied as a virtual musical keyboard interface, visualises its musical actions next to human musician's instrument, this redefines co-creative artificial musical experiences through a minimal aesthetic dialogue.
Following our “off-the-shelf” methodology, our work focusses on leveraging previously developed and commercially available components. The two research-focussed components are our minimal intelligent music controller, the GenAI MIDI Plug, and our mixed-reality musical interface.
This low-cost, minimal battery-powered AI music controller runs on a Raspberry PI Zero and incorporates a mixture density recurrent neural network which is appropriate for generating expressive musical signals from small artist-centered datasets. The MIDI output can be directly connected to regular electronic music instruments and the system aims to support musical and artistic experimentation by a diverse population.
Our approach emphasises the dynamic and lived nature of co-creative artificial systems in live performance [6][7][8]. This dynamism is derived from creative collaboration among human musicians by establishing extra communication channels [9][10] and collaborative interfaces [11] that range from symbolic to embodied expressions. Such interfaces can include text-based instructions [12][13], shared control of sound parameters and instrument data [14], as well as musical gestures and performance cues [15]. Existing approaches to performative co-creative AI systems are often polarised: they are either overly complex with humanoid robots—which are complex to set up [16][17] and can lack richness of expression in music embodiment [18]—or the approaches take place entirely in software behind the computer screen [19], which does not facilitate sense-making between collaborators [7]. Instead, our performance embraces minimalism not only as a design choice by integrating the authors' existing creative works on a minimal intelligent musical controller [20][21] and a mixed reality collaborative system that supports live electronic music making, but also as an aesthetic principle that focuses on essential musical interactions that challenge the conventional boundaries of AI-assisted performance.
The mixed reality performer interface is part of the authors’s ongoing networked collaboration expression project, which aims to support awareness and engagement in digital musical collaboration.
The version of the interface system used in this paper visualises MIDI information played by the intelligent musical agent, including each know and keyboard press. It is presented as a 3D hologram of an electronic music keyboard inspired by Arturia Minilab 3 [22]. The interface is accessed through the Microsoft HoloLens 2 headset [23] and implemented with a WebSockets infrastructure to display interaction information from the intelligent musical agent system in real-time.
We have performed with the “Off-the-shelf” system in two configurations. The first was an improvised recording session where the AI system performed on a MakeNoise 0-Coast synthesiser. In the second was a public concert where the GenAI MIDI Plug was connected to a Korg Volca FM.
The improvised recording session took place in a university classroom. The purpose of this performance was as a proof-of-concept for the “off-the-shelf” idea where the connections between different systems could be tested reasonably thoroughly and video documentation created. The setup involved the human musician and GenAI MIDI Plug at either end of a table. The lighting in the room was muted to ensure that the image on the Mixed Reality Headset was clearly visible to the performer. For this performance, the human played a Teenage Engineering OP-1 and the GenAI MIDI Plug played a MakeNoise 0-Coast. This performance demonstrated the viability of this idea however, the limited MIDI input possibilities of the 0-Coast (two channels of MIDI note input are available) led us to feel that the improvisation was a bit one-dimensional.
The public concert involved a number of performances with intelligent music systems and other newly developed interactive computer music instruments. Off-the-shelf was programmed in the middle of the performance. For this performance, the 0-Coast was replaced by a Korg Volca FM. Significantly the Volca FM responded to MIDI CC messages which could be controlled by the GenAI MIDI Plug in addition to notes. This meant that the AI agent could control the timbre of the sound as well a pitch, leading to a more dynamic performance. Both note and CC value changes were visualised to the musician through the mixed reality interface. For this performance, the musician and AI system were located on the same table and the musician was able to adjust the Volta’s volume, this allowed the length of the performance to be controlled as the AI system is not able to start or stop performing independently.
In conclusion, our “off-the-shelf” methodology proved successful in advancing live performances with AI . As human musicians, we enjoyed performing with the musical agent. In particular, the experience of the AI playing in a “non-human way” — such as playing extremely quickly or making abrupt changes — prompted us to engage more in the performance. This perception of AI as a non-realism ingredient in prompting creativity in human-AI interaction has been discovered in studies of AI-generated dance [24] and computer creativity [25].
With the mixed reality interface displaying what the musical agent was playing musically, it provided clearer musical ideas that allowed human musicians to take the initiative in the co-creative process. However, there were moments when the human musician felt disconnected from the AI during the performance, as it seemed that AI played in a musically “clinical” manner based on the pre-trained behaviour. It followed its own way and failed to musically respond to human musician. In other words, it lacked the flexibility in musical responses , which could lead to a less satisfying collaboration at times.
These reflections have led us to future works to conduct user studies to understand the sense-making issues in human-AI co-creation [26] [27] and the role the intelligent agent plays in shaping the dynamic of collaboration in co-creative system.
The authors would like to thank Eric Byler and Lee-Francis Evatt for assistance recording the performances.