This project represents a decade-long exploration at the intersection of artificial intelligence and music, culminating in an audiovisual composition that seeks to bridge human perception with the synesthetic experiences reminiscent of Wassily Kandinsky's artistic endeavors. The project has evolved from its initial phase of purely audio compositions to a sophisticated integration of AI-generated visuals, employing image-to-image diffusion techniques, specifically through Disco Diffusion, to achieve a dynamic, synesthetic experience.
At its core, the project leverages artificial intelligence to interpret and transform procedurally generated visuals based on the music's perceived parameters. These parameters, such as "grittiness" or the "level of breakdown," are not extracted directly from the audio signal but are rather subjective interpretations of the music's emotional and textural qualities. This nuanced approach allows for a more profound connection between the music and the visuals, creating an immersive experience that transcends the mere combination of audio and video.
The visuals are inspired by the cognitive processes of Wassily Kandinsky, aiming to recreate the artist's synesthetic experiences through generative art. By studying Kandinsky's works and understanding his perception of color, form, and sound, the project endeavors to generate visuals that reflect the artist's unique brain processes. This ambition is realized by parameterizing visual systems and syncing them with manually drawn MIDI curves that represent the music's perceived foreground, thus establishing a visceral link between the auditory and visual components.
The incorporation of Disco Diffusion marks a significant advancement in the project's development. By inputting procedurally generated video into image-to-image diffusion systems with prompts inspired by Kandinsky's influence, the project achieves a new level of visual expression that closely aligns with the original vision of recreating synesthesia. Additionally, the application of StyleGAN introduces a painterly aesthetic to the visuals, adding depth and complexity to the overall composition.
This project aligns with the conference's theme by demonstrating the potential of AI to enhance and expand the musical experience, bridging the gap between technological innovation and artistic expression. It explores various AI systems, from procedural generation to diffusion techniques and neural style transfer, showcasing the versatility and potential of AI in creating new forms of art that resonate on a deeply human level. Through this endeavor, the project not only pays homage to the past, drawing inspiration from Kandinsky's groundbreaking work, but also looks to the future, pushing the boundaries of what is possible at the confluence of music, art, and technology.
Type of submission
This submission is a fixed video, suitable for:
Performance 2, with no live performers
Performance 3
Technical/Stage Requirements
This piece is a fixed media audiovisual work (simple stereo video file) that I typically play through a computer connected to the sound system and a projector (or screen).
Requirements are:
Computer to play the video file
Sound system connection for the audio output
Projector / screen for the visual output
Stereo audio that plays with default configurations
Program Notes
break me, ai is an enthralling audiovisual exploration, imagining Wassily Kandinsky's response to electro-acoustic music through the lens of his synesthetic vision. This piece creates a dynamic, evolving 'painting' that moves with the music, embodying the continuous interplay between auditory and visual stimuli. It leverages a sophisticated blend of biological insights and artificial neural networks to forge a deep connection between the sound's parameters and the ensuing visuals. Inspired by Kandinsky's iconic paintings, the visual elements undergo real-time transformations in response to musical cues, including MIDI inputs and artistically interpreted elements like "glitchiness."
As the composition transitions through different sections, the visuals adapt, ensuring a seamless, symbiotic relationship with the music. This adaptability is further enhanced by feeding the evolving visuals into a Generative AI model, guided by prompts reflective of Kandinsky's motifs. The integration of these motifs with the music's dynamics allows for a visual output that remains intricately linked to the auditory experience. break me, ai not only pays homage to Kandinsky's groundbreaking work but also showcases the potential of AI in bridging historical artistic visions with contemporary creative expression, offering a vivid, immersive journey through sound and sight.
The authors extend their profound gratitude to Stochastic Labs in Berkeley for their instrumental role in bridging our connection with the AI community. Their invaluable support and inspiration were pivotal in refining and elevating our project to its ultimate incarnation. We are also deeply thankful to the Center for Computer Research in Music and Acoustics (CCRMA) at Stanford for providing the essential groundwork and resources that served as the cornerstone of our project's development. These collaborations were vital in bringing our vision to fruition, and we are immensely appreciative of the guidance and opportunities afforded to us by both institutions.
Creative Commons Attribution 4.0 International License (CC-BY 4.0)
