aiya meets self invites participants to explore this transformation of ‘other’ and ‘self’ by creating spontaneous, playful soundscapes together with the machine improvisor. The experience blurs the lines between physical and symbolic identities, plays upon fluctuating power dynamics in musical improvisation, and asks one to re-imagine the tangibility and affect that can emerge from one’s own virtual presence.

Melody Chua - (lead) concept/sound design/visuals/programming/interaction design/scenography
Lara Baltsch - scenography and organization
Kein Museum - organization and promotion
Jonathan Ospina - photo and video documentation

aiya meets self is an interactive installation utilizing pose detection and machine improvisation to create an audiovisual dialogue. The installation creates a space between machine, its creator, and the tensions that exist between the two as they navigate each other as ‘others’ and also as integral parts of each other’s identity and embodiment. In this interactive installation, participants use their physical bodies as an interface in which to communicate with a machine improvisor (aiya). Together, both entities build an improvisatory dialogue through sonic and visual mediums. In a time where virtual meetings and remote learning become an accepted normal, and where one develops a heightened sensitivity to what it means to have a human presence, this interactive installation plays with the notions of external embodiment, extended mind, and what constitutes an affective virtual presence in the context of improvisation.

George Lewis raises the question, „Why do we want our machines to improvise?“ (Lewis 2018). There is a certain mystery of the spontaneous process of improvisation, and a practice that reflects the interplays within societal structures and forms the core of human interaction. Some might argue that perhaps this mystery of the human spirit must remain a mystery; to encroach it would unravel the infinite, almighty identity that humans have crafted for their symbolic self. This threat is not only bounded to the mystery of music making, but takes inspiration from humans’ existential identity crisis and the ultimate subconscious denial of the mortal condition. As Ernest Becker describes, humans are confronted with an existential paradox: „the condition of individuality within finitude” (Becker 1973). On the one hand, the human has a „symbolic identity“ marked by an awareness of her history and an unbounded imagination that can speculate about concepts far beyond her immediate physical grasp. On the other, she is bounded by the physicality of her decaying body, destined to decompose along with every other organism in her universe. Humans, according to Becker, are both „out of nature and hopelessly in it; he is dual, up in the stars and yet housed in a heart-pumping, breath-gasping body that once belonged to a fish and still carries the gill-marks to prove it” (Becker 1973). The dual-nature of the human is difficult to reconcile, and at the core of this difficulty is the necessity to face one’s own mortal _vulnerability.

Augmenting the body with technologies entertains the inner desire to deny oneself this vulnerability and defy the eventual degradation of the flesh. Easy examples include the integration of pacemakers to regulate a misfiring heart, a notebook to augment the forgetful mind, or in the context of musical expression, microphones to extend the sound of the human voice to reach farther than is otherwise ‘humanly’ possible—among a myriad of others. Indeed, the extended mind and the cyborgian figure entertain the question, „Where does the mind stop and the rest of the body begin?" (Clark 1998). When we improvise with machines, they function not only as a separate entity with which to engage in musical conversations, but simultaneously as augmentations of our musical cognition. And yet, we return to the original threat that performing with machines—and more so when one creates those machines herself—poses on our identity: just as humans struggle with the paradoxical „condition of individuality within finitude,“ so do they resist the idea that machines could have the potential to explain the mystery of musical improvisation and make tangible, make physical, and thus, make vulnerable the symbolic, sublime nature of improvisation.

This brings back the original question: „Why do we want our machines to improvise?“ Lewis concludes, “…perhaps our improvising computers can teach us how to live in a world marked by agency, indeterminacy, analysis of conditions, and the apparent ineffability of choice. Through improvisation…we learn to celebrate our vulnerability, as part of a continuous transformation of both Other and Self...What we learn is not about machines, but about ourselves, and our environment” (Lewis 2018).

3.7 aiya meets self (oct-nov. 2020)

In an attempt to gain a different perspective of the improvisation machine, I decided to adapt it into an installation and observe other people interacting with it. Each participant stood in front of a webcam that would detect their skeleton/joints using Posenet (AI), and by moving, they could improvise with AIYA (improvisation machine), producing electronic sounds and visuals in real-time. This took place in a small, intimate environment that was experienced by one person at a time.

Playing with glitches
I created rather exact mappings between the skeleton/joint detection of the camera/Posenet and the resulting generated sounds and visuals from the improvisation machine. However, an interesting amount of indetermination was introduced due to the “natural” glitches that would occur from both the camera’s image instability when the ambient light would shift (e.g., from the shifting projection videos generated by the machine) and the resulting inaccuracy of the Posenet AI to detect the skeleton/joints. Sometimes additional notes or unexpected changes in the projection visuals would be triggered by the appearance of these glitches. Due to the fact that the other aspects of the interaction were mapped rather explicitly, the glitches provided an intriguing balance to the relationship between human participant and machine. The human participants perceived that their actions had an effect on the resulting audiovisual output of the machine, but the small indeterminacy introduced by the glitches created a slight uncertainty that is very attractive.

In this particular instance, I viewed the concept of the glitch as a means to create [perceived] agency for the machine. As the participants did not know how the machine was programmed, they accepted the glitches as part of the “standard” behavior of the installation, and interacted with them as if the machine itself was enacting the glitches as an agent, a distinct improvisation entity.

Control
For the first time since the initial prototype of the machine I made one and a half years prior to the conception of this installation, I did not include a physical on/off switch for the improvisation machine. The camera, the point of entry for interacting with the machine, remained on for the entire duration of the installation. The decision not to include a physical on/off switch helped to improve the user experience of the installation participants, but it was perhaps also my attempt at getting away from my desire to control the system.

Nevertheless, I also realize that despite not having a “physical” on/off switch for the machine, I still had an on/off mechanism for the machine: As long as no human skeleton is perceived by the improvisation machine, it would not play. This was a rather reliable means for me to make sure that the machine was “under control” and playing only when I expected it to. Thus, I suppose I did not escape my desire for control of the situation here, either. However, the difference between this on/off mechanism and the physical switch lies in the fact that it is not I who plays with the machine for the installation; it is the participants, and they are not made aware of this “on/off” mechanism. When the participants lose the control to turn on and off the machine, perhaps the machine appears to the participants as having more agency than a machine that the participants know they can turn off.


(above) signal flow of this version of AIYA, where instead of using the Chaosflöte as an input, the participants only use their bodily motions to communicate and play with AIYA. Ableton is also used for the first time as the sound "engine" for AIYA. The AIYA Max patch sends MIDI commands to Ableton to control the playback and sonic parameters of the sounds.

Interaction design
Participants interacted with the machine solely through their body presence in front of the small camera in the installation. The lack of any other physical inputs (e.g., buttons, touchscreens, etc.) was intentional, in an attempt to facilitate a feeling of extendedness between the physical human body and the machine. Entering a small, curtained room, the participant is in an instant surrounded with the immersive projection visuals and sounds of the machine, which are meant to represent the machine not only as an avatar, but also as part of the space itself. By moving their own body, they move the body of the improvisation machine (manifested via the reacting sounds and visuals), and become an extension of the space.

Through this experience, I realized that physical immersion [1] could be used as an effective means to achieve the feeling of extendedness between man and machine. Certainly, this is not the only way [2] but has been one that I continued to develop in my subsequent work. In the case of aiya meets self, the participant occupies both a physical space (the room itself) and a virtual space (the invisible detection mechanism of their body from the camera) with the machine. As noted by Ryan, the participant’s physical body can actually reinforce the effect of immersion when their physical actions make a perceivable effect on the virtual world (Ryan, 1999). In fact, this is an aspect used by VR developers to reconcile immersion and interactivity. When the physical body of the participant in aiya meets self coordinates with the virtual elements in the digital space, the feeling of extendedness can be reinforced between man and machine. The main factor that determines the effectiveness of this is the transparency in the mappings between physical action and virtual (re)action.

The one-on-one setting and interaction design of the installation also contributed to the machine's appearance as something in between a distinct entity with which to improvise and an instrument that would react to the movements of the participants. The main factors influencing the perception of the machine as being more like one than the other was the immediateness of the machine's reactions to the participants' movements and the clarity of the mapping between movement and machine behavior. The machine was effective in creating a sense of immediateness from the participants' actions, however, the machine did not provide much transparency, resulting in the perception of the machine as more of a hybrid between instrument and improvisation partner.

_control

_agency

Representation
The motif of reflection appears repeatedly throughout aiya meets self, which paints a rough picture of how I had come to regard my relationship with the improvisation machine. Pipilotti Rist’s words resonate with how I felt about the machine, which she revealed in a 2019 interview with the Lousiana Museum of Modern Art: “Because the machines were made by humans too -- their character is in there, their senses are there, and when the machines produce these pictures, they become close to our inner pictures.” The improvisation machine, as much as I strived to give it its own identity, would inevitably be a form of reflection of my own inner pictures, of my own identity and signature as an improviser.

Several elements within the installation exemplify the motif of reflection:
- The bottom portion of the installation space was lined in mirror foil, where the participants could see a warbled reflection of themselves as they moved for the improvisation machine.
- The visual projections were distributed across four projectors that spanned two corner walls. Two projectors were assigned to each wall, with each projecting the same image, only in a different size (one smaller, brighter image and one larger, dimmer image, positioned crookedly in relation to each other). Each wall was a mirror image of the other.
- As the installation took place within a very small, enclosed space, the participants were essentially confronted with a giant hybrid mirror of themselves and their actions, with no space to escape the gaze of the mirror (camera). In this case...as aiya meets self, so do the participants meet their selves.
- A separate TV displayed in the outside-facing window of the space (outside of the curtained space) also served as a form of reflection, as it displayed an itinerant view of the main installation to the participants waiting outside, showing at times the camera feed of the active participant that was moving within the space.

_representation

_reflection

Video documentation of this work can be seen below:

next ►◄ back

[1] When referring to “physical immersion,” I take Bouko’s description of immersion as one that is framed from the perspective of the participant (as opposed to an objective/external judgement) (Bouko, 2014, p. 260).

[2] For example, the keyboard I am typing this thesis on can feel like an extension of my body without necessarily being “immersive.”