[RECONFIGURTING AIYA's SONIC IDENTITY]
With the new abundance of data I now had access to from the position and rotation values of the rigid bodies, I had to think about how this would affect the audio programming of AIYA. I still retained the basic architecture of AIYA as a buffer-based improvisation machine with five voices of differing qualities, but now I was able to more viscerally connect my movements with AIYA's sound character.

First, I reprogrammed AIYA to record the entirety of the improvisation, from the moment I would begin playing. This means, instead of previously only having access to the last 10-20 seconds of my playing, it had access to the entire history of the improvisation. From here, it had the option to choose material from any point in the past as the starting point for its sound generation.

Then, the question became, how should the machine choose which sound material to work with? I reflected back on my movements in A->B, this meandering walk from one side of the stage to the other that somehow created a potent spatial connection between myself and the machine (due to the feedback this walk would produce). This spatial connection, already palpable in the shifting grid projections, became another important concept to develop in the audio of the improvisation machine.

Since my position within the IAS was tracked, I created a virtual map of my position in the IAS and connected it to the buffer selection process of the machine improviser. The x-axis of my position was mapped to the point in time in the past from which the machine improviser could choose material from (e.g., the far left on the x-axis would make the machine improviser choose buffer material from the very beginning of the improvisation, and the far right on the x-axis would make it choose material from the present). The y-axis was mapped to a deviation value, where the higher the y value, the higher the machine could randomly choose buffer material from the surrounding time periods (e.g., a high y value with a low x value could increase the chance of the machine to choose material from the middle part of the live-recorded improvisation, instead of exclusively material from the very beginning of the improvisation). The beauty of this paradigm is that in the beginning, it can be easy to "steer" the improvisation machine's choice of source audio material, but as the improvisation goes longer and longer, it becomes more difficult to be precise about this selection, because more and more sound material is available to the x-axis parameter.

Aside from the improvisation machine's sound generation, I also mapped my body movements directly to live effects processes in Ableton, which would transform my flute sound in real-time, according to the rotation of my joints. I used Reaper to spatialize the sound with the ZHdK ICST (Institute for Computer Music and Sound Technology) VST Ambisonic plugins.

Technically, the signal flow from the human performer to Chaosflöte and improvisation machine can be summarized in the diagram below:

(2) black box finding (interactive installation)
I reimagined the black box I constructed for the 360° video recordings into an interactive installation for up to six participants. Each of the rigid bodies originally used on my body in the 360° recording were distributed to each audience member, as somewhat of a metaphor for the disembodiment or distributed embodiment of my body (as perceived by the machine) to the six participants. Three of the rigid bodies controlled the live visual projections and three rigid bodies controlled the sound generation. A view of the installation space can be seen on the right.

I staggered the entrances of the participants into the installation space, two at a time, for a gradual increase in the denseness and activity of the installation. What resulted was a general decrease from having clearer transparency in the human-machine interaction to less transparency, and accordingly a general decrease in feelings of extendedness with the machine as the number of participants in the space increased.

Aesthetically, I would have preferred to keep the transparency of the installation more clear and consistent than it was at the actual event. However, it was interesting to observe the machine fading between these extremes and to try to observe the point in which the machine shifts from appearing as an extension of the participants to being a separate entity and its own space. This point differed from group to group, but generally this shift occurred the moment the third and fourth participants entered the space and/or if the participants were more prone to move/rotate the rigid bodies more drastically than normal.

(above) the installation setting of black box finding. Three rigid bodies allowed participants to steer the virtual camera within the virtual [2] 3D space of the scene (altogether the rigid bodies controlled XYZ position, rotation, and zoom). As the virtual camera was moved and rotated, one could "walk" within the virtual scene towards text snippets from the prologue that were distributed across the virtual 3D environment. Sometimes it would be necessary to rotate the virtual camera or zoom into the scene in order to clearly read the text.

Between concert and interactive installation
black box fading as an event spanned both fixed media “concert” elements as well as an interactive installation, making it already difficult to describe beyond simply calling it “an event.” The spatial audio work functioned as a typical “concert” fixture where one listened in a linear format, the installation functioned as an indeterminate composition of sorts with flexible duration, and the 360° video itself exhibited hybrid elements between concert and interactive installation:
‍
- The physical scene of the recordings used in the 360° video already behaved like an installation to me, especially in regards to my relationship to that space. As the OptiTrack system tracked the position and rotation of my body and the Intel RealSense camera captured an infrared representation of my body, every movement or sound illicited a degree of response from the projections and electronically produced sounds. During breaks, it was interesting to see people working at the IAS walking around within the tracked performance setting and behaving as if it was a type of interactive installation (even though their bodies were not tracked via OptiTrack, the RealSense camera still picked up their movements, serving as their interaction point).
- My representation within the 360° video also played between the attitude of "playing a concert" and the atmosphere of observing an abstraction of an installation space. At times I am seen playing the flute (allusion to the concert setting), but at other times I could be seen sitting in the IAS, peacefully observing the dynamic projection visuals surrounding me (allusion to the installation setting).
- Furthermore, the nature of the 360° video format is that one cannot fully guarantee the field of vision the audience will choose to experience;. One can only influence it by predicting what elements the audience will find the most interesting to pay attention to at a given point in time and use that information to organize the dramaturgy and narrative of the video, while at the same time acknowledging that there will be variations on that narrative depending on how the audience chooses to interact with it. This is, to me, much like an interactive installation, where you have a general idea of what the audience might choose to perceive, but there is still an element of indeterminacy.

Juxtaposition of orientation/disorientation
black box fading uses the balance of orientation and disorientation as a metaphor for my personal experience in improvising with the improvisation machine. At times, being the programmer of the machine gives me a certain "orientation" when performing with the machine in that I can discern its general behavior and character. On the other hand, I am not actively thinking about the interaction rules I have programmed for the machine, and there are also surprises produced by the glitches in the machine. These result in some moments of "disorientation," or when referring to Lösel's described model of improvisation, the point of "observation...at which an improviser realizes that his mental model diverges from others'" (Lösel, 2018, p. 190).

Navigating the balance between orientation and disorientation also speaks to the negotiation of control and vulnerability, issues that I have continued to grapple with throughout AIYA's development. Some disorientation can be very stimulating and propels the dramaturgy of the narrative work. With the machine, I am testing the limits of my vulnerability and asking, "how much control do I really need?" I try to represent this disorientation/orientation balance, literally speaking, through the immersive aspects of the work in black box fading, as outlined below:

One of the first subtle moments of disorientation occurs in the spatial audio work, where there are no visual cues to orient oneself within the work in the traditional sense. The participants adapt to this by instead directing their orientation sense on the audio level, where the sounds within the work have been carefully spatialized across the 42 speakers of the IAS Ambisonics system.

Once the spatial audio work is finished, the projections of the immersive interactive installation space gradually appear. The image is mostly static until the first participants are led into the installation space and pick up their rigid bodies. From here, the mapping on the visual level is fairly straightforward: Move in the physical space to move the camera in the virtual space. Stop moving in the physical space to stop moving in the virtual space. The same is true for the audio control, where moving the rigid bodies creates sound and stillness stops it. The disorientation creeps in when more people are added to the installation. Who is controlling what? And who plays when? If all six participants are moving at the same time, it is no longer easy to discern the mappings made between motion and audiovisual activity, and one becomes disoriented in their relationship with the machine.

When the participants arrive at the VR portion of the event, a further disorientation is introduced through the complete darkening of the IAS and the "takeover" of the visual field with the 360° video shown on the VR headsets worn by the participants. Within the VR video, participants are confronted with the following juxtapositions of orientation/disorientation:

0:08-2:51 - A clear left-right orientation is outlined through the repeated appearance and disappearance of two rectangles (their appearance and disappearance is always occurring in the same locations). However, within the rectangles themselves is a scene of complete disorientation: the equirectangular 360° recordings shown in these rectangles have been decoded for playback in the cube map format. This mismatch of source and destination video format has a slightly destabilizing effect. On the audio level, the initial appearances of the rectangle correspond to the same location in the spatial audio map, but as the section progresses, the locations are reversed and eventually overlap each other to create auditory disorientation.
2:51-3:54 - A grid forms into view (white lines on black), to create a uniform canvas for the viewers to orient themselves.
3:55-5:29 - This sense of orientation is once again decomposed as the grid disintegrates and morphs into a new, shifting texture. This texture fades between sharpness (orientation) and blurriness (disorientation).
5:30-5:43 - The scene finally composes into the view of the IAS stage, where one can clearly see me playing the flute and have a general sense of "which way is up." My flute sound can be localized to the visual position of myself on the stage. There is, however, the amplified flute signal that sounds on the opposite side of my physical location, which in the spatial audio creates the perception that I am in two places at once, aurally speaking.
7:07-10:58 - This sense of orientation is, again, undermined by the morphing and overlapping of multiple 360° videos. The 360° camera alternates between stationary positions and moving recordings to create further cycles of orientation/disorientation until the end of the section.
11:05-16:26 - The opening of this section outlines a faded box around myself playing the flute to serve as starting visual reference point for the work. Throughout the piece, this box expands, contracts, changes shape, and even rotates (13:32) in an alternation of orientation and disorientation.
17:05-18:19 - 360° footage captured with the camera in a stationary position (orientation) is juxtaposed with volatile visual noise that at times obfuscates the 360° footage (disorientation).
18:48-19:38 - The whole IAS stage is revealed in bright light (orientation)...
19:41-20:16 - ...only to be faded into a series of continually shifting lines and boxes reminiscent of the video segment from 3:55-5:29.
20:32-20:37 - The texture morphs one last time, back to the original, uniform grid seen in the beginning of the work (but this time with black lines on white: orientation).

At the end of the video, the participants, upon removing their headsets, find themselves in yet another enclosed black box, whose curtains are slowly opened to (again) reveal the full view of the Immersive Arts Space, which is now illuminated to establish the last cycle from disorientation to orientation [3].

(above) signal flow of this version of AIYA, using the OptiTrack motion capture system in the ZHdK Immersive Arts Space. The addition of the RealSense camera allowed me to capture a live infrared camera feed of the scene, which I used for some of the source material for the visuals in the TouchDesigner patch; the virtual outlines/reflections of my physical body is one example, as seen in the later subsection "The reflection as a metaphor for failed human-machine emancipation."

The Chaosflöte and the motion tracking afforded by the OptiTrack system contributed to the main sensor inputs used by the improvisation machine to read human behavior and act according to its pre-programmed interaction rules. 5 rigid bodies were affixed to the human performer (myself) to track the global position and local rotation of each major limb of my body. AIYA—programmed in MaxMSP—would take this data and control the output of sounds and visuals generated live in the programs Ableton (sound) and TouchDesigner (visuals). On the audio level, the signal goes through another program, Reaper, to be decoded for the Immersive Arts Space Ambisonics speaker array. On the visual level, the TouchDesigner visuals get transferred to the projection mapping program S.P.A.R.C.K. [1] and finally distributed to four projectors that cover three walls and the floor of a giant black box lined with black curtains.

(above) circled are the motion tracking cameras distributed throughout the IAS.

(above) 5 rigid bodies are attached to my body, and a sixth is attached to the 360° camera filming me.

(left and above) my own grids I constructed out of inspiration from the calibration grids. Grids are also a prominent motif in cyberpunk culture, of which I am also a fan, from early childhood until now.

(3) black box fading (360° video in VR):

I began the development of my final master's project with the improvisation machine knowing only that I had wanted to work with immersion as a loose concept for the improvisation machine's representation and that the project would take place at the Immersive Arts Space. Much of the work came together in constant dialogue between my original ideas and the ideas the space and technology themselves gave to me. The result is a work of hybrid ideas, where I use the limitations of the space and technology in an intentional way that informs the overall aesthetic of the work.

[SHIFTING GRIDS]
With the idea of immersion, I knew I wanted to create some sort of box with interactive visual projections on each side of the box, however, the original idea for the box was for it to be constructed out of white screen material for better color contrast. In the end, it was only possible to hang black curtains as a projection surface, so I adapted my visual concept to a "black box" with white projections.

Even then, I imagined a dense visual identity for the improvisation machine, with thick geometric forms to encompass my black box. However, the moment I arrived in the space and saw the constructed box for the first time, it was clear that the nature of the space called for a sparser scenography in the visual projections. Too much white in the projections would create too much light in the room and diminish the seemingly infinite void of the black box (as the materiality of the black curtains would be blindingly exposed). The technical manager of my project, Sébastien Schiesser, showed me the calibration of the four projectors used for the visuals on the black curtains, where a set of grids were projected onto each surface as a way to check the alignment and stitching of the four images that would comprise the unified box texture. Although the grids themselves were fairly simple, I was fascinated by how they were able to create such powerful illusions of space and scale. Shifting the grids in the calibration itself already induced a slight sense of disorientation and unstable architecture. I somehow felt a connection between the tension of orientation/disorientation of the shifting grids and the tension between the control/loss-of-control that I sometimes had with my improvisation machine. The grids became an important motif throughout the entirety of black box fading.

I created my own grids in TouchDesigner and experimented with different ways to shift them in real-time. Using the OptiTrack motion capture system of the IAS, I was able to map "rigid-bodies" (special objects whose position and rotation were made trackable by the motion capture system) to the position and rotation of the virtual camera in my sea of grids. I wore five rigid bodies (1 on the head, 2 for each hand, and 2 for each foot) that allowed me to shift my virtual view of the grid projections with my body.

Between improvisation and composition
Among the shifting grids of the black box, I wanted to create a live concert with the improvisation machine. After all, one of the key attributes of improvisation is its “liveness,” which communicates a certain authenticity and energy difficult to replicate in a documentation format. However, due to the second wave of COVID-19 in the fall of 2020, I had to make the decision early in the creation process of the project to adapt the final project into a series of documentations (representations) instead.

This led me to the idea of creating an immersive documentation of my improvisations with the machine/black box, using a 360° camera and an Ambisonics microphone (seen on the right).

(above) the 360° camera + Zylia Ambisonics microphone in action.

(above) the revised AIYA Max patch, which includes a new management system for the "voices" of the machine, and mapping rigid body parameters to the machine's buffer selection, sound effects processes in Ableton, and shifting grids in TouchDesigner.

(above) the recording rig used to hold and move the 360° camera and Ambisonics microphone.
Somehow I feel conflicted over creating a work about improvisation in a composed setting…pandemic aside, perhaps this format illustrates precisely my blurred perception of improvisation vs. composition…..if you watch recordings of an improvisation, does it still feel improvised for you? If you edit and combine recordings of improvisations together, does it already make the work a composition?

Listening to the fingerprint of the machine
Recording the material used in black box fading proved to be a challenge due to the low-light setting of the scene. The footage of the 360° video would appear dark, with low contrast and an excessive amount of noise. Imperfections such as these began to inform my aesthetic direction of the work. At first, I wanted to reduce the imperfections as much as possible, and it is true that much of the footage had already been “denoised” by the cinematographer (Valentin Huber). However, I soon realized that in order to more authentically embrace the aesthetic of the machine, I should rather work with the low-res/imperfect nature of the recorded footage and use that as an inevitable aesthetic motif of the work, a fingerprint from the machine. Throughout the 360° video, I worked with the poor signal-to-noise ratio of the original recordings more emphatically by at times exaggerating the graininess (the extreme of which culminated into “pixel dissolve” video effects) and at other times reducing the graininess to the point where the footage morphed into its own blurriness and no longer resembled its original form. This resulted in an interesting aesthetic for me: On the one hand, the subject of the documentation was rather futuristic, with state-of-the-art motion tracking, projection systems, and Ambisonics sound system as the foundation. It carried a certain dystopian vibe as one saw myself in the video wearing and interacting with this technology. On the other hand, the fidelity of the documentation seemed nostalgic in character, almost as if it was a reimagination of film noir; it carried a wear and age to it. It was as if one was capturing a memory of the future.