AIYA
improvisation machine

year: 2019-present
type: software/research
SUBTYPE: PERFORMANCE, INSTALLATION
by: Melo

about

AIYA is an improvisation machine originally designed for the Chaosflöte and MIDI instruments and recently adapted for installation settings. Its identity is situated between appearing as a hybrid between (a) being an extension of the human performer and (b) functioning as a distinct improvising agent.

The name “AIYA” comes from the Chinese expression “Ai ya! 哎呀” which is used to express surprise, frustration, excitement, or empathy--all elements that capture the spirit of human-machine improvisation. That “AI” is present in the name is a welcome coincidence, due to the presence of AI processes that influences some (but not all) of AIYA’s behavior.

Built primarily in MaxMSP, AIYA connects to any other program that is capable of receiving OSC messages, for the intention of creating flexible performance setups (e.g. with Touchdesigner, Ableton, Supercollider, etc.). It spins textures and melodies that are comprised of both purely synthesized sounds as well as from live-recordings of the sound and motion behavior of the human performer during the performance. Through an analysis of performance attributes such as frequency, amplitude, density, spectral centroid, spectral flatness, body position, and join rotation, interaction and behavioral rules for the machine are derived. These explicit rules are combined with AI processes that generate behaviors that may be less predictable, but still intuitive. Important in AIYA's design is to have a balance between familiarity vs. surprise, experimentation, and the expanding of the improvisational pallete.

While the core of AIYA’s development began in spring of 2019, it is a fundamentally flexible and continuously evolving system. Over the years, different adaptations have been made, including versions for MIDI keyboards, PoseNet performer, sensor-augmented flute, human-machine quartet (two humans, two improvisation machines), vocalist, and percussionist. The spirit of the project is such that each different version of AIYA constitutes a completely new work, where the mappings between AIYA’s data and its audiovisual representation is considered itself a composition/indeterminate performance system.

The current version of AIYA is optimized for Chaosflöte. The following schema diagram summarizes the technical components of the AIYA-Chaosflöte improvisation system.

doctoral research

AIYA is a central component of my doctoral research project at the Künstuniversität Graz and Zürich University of the Arts.

Research Title:
Developing a Visual Framework for Musical Improvisation Machines: Investigating Impacts on Human-Machine Interaction, Posthuman Identity, and Performance Narrative‍

The practice of creating and performing with improvisation machines spans several decades, most notably seen through the works of George Lewis, Benjamin Carey, Guy Hoffman, Gil Weinberg, and IRCAM, among others. Arguably, the aforementioned developments have focused largely on the sonic sophistication of the improvisation machine and presents a research gap in the development of more sophisticated visual representations of the machine. My dissertation research aims to address this gap by developing a visual framework for improvisation machines that can be used as a defined, yet modular starting point for other creators of improvisation machines to use for their own visual representations.

The development of such a framework instigates reflections on the visual representations’ effects on the performance setting, specifically addressing the following questions:

- How might a visual representation for a musical improvisation machine be developed that aids in facilitating bidirectional human-machine communication and the blurring of perceived borders between human and machine?
- How does the visual representation contribute to the shifts in perception between the two identities of the improvisation machine, when it appears at times as an extension of the mindbody and at other times a distinct improvisation agent?
- What effects does this have on the perceived narrative of the improvisation?

The research will culminate into three distinct components: a suite of performances with the developed visual framework applied to the improvisation machine, a written dissertation reflecting upon the aforementioned issues raised by the research topic, and the software of the improvisation machine with the applied visual framework available as an open-source project for other creators to use and modify.

works with AIYA

‍