➡️ This talk is part of IRCAM Forum Workshops Hors-les-Murs Tokyo 2026

1. Artistic Motivation

I have always been deeply drawn to environmental sound. To me, rain, wind, traffic, machinery, and human breathing can all become part of music.

These sounds usually have no fixed beat, pitch, or duration, which makes them difficult to perform directly like traditional instruments. Da Yin Xi Sheng uses Grain FX, resonance, spatial movement, and real-time control to transform environmental sounds into material that can be organized and performed, while retaining their original texture and recognizable source. Processed rain, for example, should still preserve the continuity of rain, and mechanical sound should not be reduced entirely to an ordinary synthesizer timbre.

One of the central challenges of this project is to make environmental sound more playable without erasing its original character.

(1) Project Title

The project title comes from the phrase "Da Yin Xi Sheng" in the Tao Te Ching. It is commonly understood to mean that a truly expansive sound need not be loud or appear in an explicit form. This meaning closely reflects my artistic approach: environmental sounds that are often overlooked can also become musical material.

For this reason, I chose not to invent a new concept for the work, but to use the classical phrase "Da Yin Xi Sheng" directly. It expresses both my understanding of sound and the project's practical method: finding material that can be performed and organized within recordings of rain, wind, urban noise, machinery, and other environments.

(2) Originality and Related Work

This project does not present its sound-processing plug-ins as new audio technologies in themselves. The Grain FX module uses fundamental methods similar to those found in conventional granular effects processors; likewise, related approaches to controlling real-time graphics with audio-analysis data can be found in other audiovisual works.

The independently conceived aspect of this project is the integration of graphic notation with live sound processing, allowing the score to be generated in real time during performance.

Before the performance begins, the score does not exist. The processed sound from each effect, together with data from the A, B, and C effect chains, is continuously written into a frequency–time display as the performance unfolds. The generated result can not only be viewed, but also saved, replayed, edited, and exported. It therefore records neither a pre-composed piece nor a purely visual representation of sound, but the process through which sound is processed and organized in a particular performance.

iAnalyse is an important precedent in graphical music analysis. It can import audio, video, and scores; calculate spectra and audio-description data; and create graphics, animations, and annotations synchronized with a media timeline. Its official documentation, however, focuses primarily on pre-existing media files, musical analysis, and guided listening.

2. Project Overview

Da Yin Xi Sheng is a real-time audiovisual system that uses environmental sound as performance material. Sound is processed in Max/MSP and controlled with an MPE controller and an AtomS3R IMU sensor. The project can record live environmental sound or use prepared environmental recordings, which are then processed through Grain FX and Resonator. The graphic-score application "draws" these sonic changes on frequency–time coordinates and records the performance for later replay, editing, and export.

An environmental recording can preserve the sound of a place during a particular period, but once the recording has been made, its playback order and duration are usually fixed. The performer can play, stop, or adjust its volume, but cannot easily continue reshaping the sound's internal structure during performance.

Da Yin Xi Sheng is designed in response to this limitation. Here, an environmental recording is neither accompaniment nor a finished object that becomes immutable once playback begins. The performer can select sounds from the recording, divide them into grains, alter their number, duration, pitch, and read position, and apply delay, reverb, distortion, diffusion, and resonance. The processed sound can also move through auditory space. Environmental sound itself can therefore function as an electronic instrument that is played live.

3. How Sound Enters the System

The system can receive live microphone input as well as prepared environmental recordings and fixed-media files. All audio is first written to or read from buffers in Max/MSP, and then passed into the Grain FX module.

Grain FX divides continuous sound into many short fragments. The performer can control the number of grains, their duration, envelope shape, pitch, read position, reverse-playback probability, freeze state, and the balance between dry and processed sound.

All samples pass through Grain FX before entering the subsequent effects modules. Instead of working with a recording that can only play from beginning to end, the performer works with a collection of sound fragments that can be rearranged and transformed at any moment.

(1) Effects Processing Chains

After granular processing, the sound enters three parallel effects-processing branches.

Chain A:

Chain A consists of Delay and Reverb. It primarily changes the duration of the sound and its perceived spatial distance.

Chain B:

Chain B consists of Distortion and Diffusion. Distortion changes the sound's edge and energy, while diffusion makes it wider, more dispersed, and less spatially defined. This chain is mainly responsible for producing rough, fragmented, and spatially ambiguous sound.

In addition to Chain A, which combines delay and reverb, and Chain B, which combines distortion and diffusion, the system includes an independently operating Chain C:

This module receives environmental recordings or live input and reshapes their spectral structure through ten band-pass resonators. The performer can separately adjust resonance regions resembling the mouth, throat, and teeth, giving rain, urban noise, breathing, machinery, or animal recordings characteristics associated with vocal organs and bodily resonance. It does not imitate a specific human voice; instead, it uses the input sound to generate an unstable, rough, breath-like timbre.

The "wetness" parameter introduces low-frequency noise whose level follows the input amplitude; "instability" causes the ten resonant frequencies to drift slowly; "size" changes the overall resonance range; and "pitch_shadow" blends pitch relationships centered on a fundamental frequency into the formants. Drive, output gain, and stereo width control saturation, intensity, and left–right variation.

This branch is used primarily in passages that require a strong impact and clear sectional contrast. Chain A extends the sound and creates spatial tails, Chain B produces fragmentation and diffusion, and Chain C pushes the input toward a more intense, unstable, and bodily timbre.

(2) Resonator

Resonator is another principal sound-processing module in the system. It uses environmental sound as its excitation signal.

The MPE controller supplies note, velocity, pitch-bend, and aftertouch data. When environmental sounds without a definite pitch enter the resonator, they can produce sustained tones with clear pitch relationships while retaining part of the original recording's texture.

Rather than replacing the environmental sound with a conventional synthesizer timbre, this method uses the environmental sound itself to excite the resonator.

4. Performance Controls

AtomS3R IMU Sensor

MPE Controller

The system assigns a clearly defined role to each control device.

The MPE controller provides note input, adjusts the levels of the main sound modules, and changes parameters in the delay, reverb, distortion, and diffusion chains.

The AtomS3R IMU sensor produces subtle pitch variation in the resonator and can continuously modify selected sound effects. In a dedicated mode, it instead controls the position of the musical sound source, allowing the sound to follow the performer's body movement.

This division of functions reduces control conflicts. The performer can understand what each gesture is changing without having to operate a large number of knobs simultaneously.

5. Standalone Real-Time Graphic Score

Real-Time Graphic Score Interface

The current demonstration uses version 0.1.4 of the Da Yin Xi Sheng Graphic Score. The real-time graphic score is not an ordinary web page that must be opened manually, but a macOS application packaged with Electron. Version 0.1.4 bundles the graphical interface, local data receiver, web runtime, file-management functions, and video-conversion tools inside the application. After launching it, the user does not need to start Safari, Terminal, or an external Node.js service.

Max Audio Bus:

→ Audio Analysis Module → Score Message Bus → Text and Byte Conversion → Local UDP Port 8798 → Local Bridge Process → Electron Inter-Process Communication → Graphics Renderer → Real-Time Score Display

The graphic score contains four main track types:

• Sound Result
• Grain FX
• Resonator
• Fixed Media

All of these signals are smoothed. When the input level changes abruptly, the image does not jump immediately; when the sound gradually fades, the existing visual forms also fade out progressively.

The Sound Result track is jointly controlled by the three parallel effect chains A, B, and C. Reverb in Chain A controls the rain, diffusion in Chain B controls the sea surface, and the creature-formant morphing in Chain C controls the storm. These three groups of data, together with a 32-band analysis of the final sound, determine the image's frequency position, intensity, and movement.

The Grain FX track retains particles generated over a period of time. Particle count, particle duration, win_shape, and spectral distribution all affect the granular forms in the image.

The Resonator track can receive voice number, amplitude, note, frequency, pitch-bend, and sustain data. In version 0.1.4, however, the system cannot yet identify the precise pitches of all sixteen voices reliably from a mixed audio signal. A later version will resolve this through performance data from the individual voices, with audio analysis used only as a fallback when performance data is unavailable.

The Fixed Media track displays the state of the Fixed Media material.

Together, these tracks form a real-time frequency–time score. The horizontal axis represents performance time, and the vertical axis represents sound frequency. The image records where sound enters the system, which processes it passes through, how the parameters change, and when these changes occur during the performance.

(1) Recording, Replay, and Export

The graphic-score application provides prepare, start, pause, end, and reset functions, and can switch between the full score and individual tracks.

The performer can save the current colors, thresholds, track intensities, display ranges, and other mapping settings. A complete configuration can also be saved as a startup preset and restored the next time the application is opened.

Control events generated during a performance can be saved as JS/JSON event files. A saved performance can be replayed, and the timing of events, complete event passages, section markers, and other details can be edited before exporting again.

The application can record the screen as video and export an entire performance as a long-format score image 2,160 pixels in height. This preserves both the live visual output and the complete temporal structure for review after the performance.

(2) Independence of Audio Output and Score Analysis

The analysis gain used by the graphic score is separated from the main audio output. Increasing the analysis gain only makes quieter sounds more visible in the score; it does not change the level heard by the performer or affect the final output.

The system's audio path has been tested. The final sound and resonator signals can enter the analysis module through a stereo input and then be transmitted over the local network to the graphic-score application. When the sound stops, the corresponding data returns to zero, and the image does not continue generating new score content without input.

6. AI-Assisted Development

During development, this project primarily used AI Agents in OpenAI Codex as programming and technical support tools. They contributed to initial code drafts, debugging, module refactoring, technical documentation, and the review of test results.

I determined the system architecture, sound-processing methods, interaction logic, parameter choices, visual rules, and performance design. AI-generated code and text were included in the project only after human review, revision, and practical runtime testing. The AI Agents do not participate directly in the live performance and do not replace the author's musical or sonic decisions; their role is to assist development across Max/MSP, gen~, JavaScript, Electron, and the graphics-rendering application.

7. Objectives and Key Questions

The first objective of Da Yin Xi Sheng is to transform environmental recordings from fixed playback material into sound that can be performed live. Rather than simply playing recordings, the performer continually changes their time, pitch, density, texture, and spatial position through different forms of sound processing.

The second objective is to establish clear relationships between gesture and control. The MPE controller handles MIDI control signals, while the IMU sensor is responsible for subtle pitch variation, effects modulation, and sound-source movement in a dedicated mode. Each device has a distinct role, allowing the performer to understand precisely what each action changes.

The third objective is to record the processes that are normally invisible in an electronic music performance. The graphic score displays not only the final sound, but also receives sound-processing parameters that modulate the image. After the performance, the data can be replayed, edited, and exported, allowing a live performance to be revisited and analyzed.

The project addresses three specific questions:

1. Can environmental recordings be released from a fixed timeline and become a live instrument that can be continuously manipulated?
2. Without traditional instrumental performers, how can a performer use controllers, sensors, and a spatial-audio system to produce a performance with a clearly defined structure?
3. Can sound processing, parameter changes, and performance gestures in real-time electronic music be preserved as a score that can be read and replayed?

The central concern of the project is not to add decorative images to the music, but to make sound processing and graphic documentation use the same data. When the audience hears a change in the sound, they can also see its frequency position, duration, and processing source in the score. For the creator, the stored score data can also be used to review the performance, inspect control mappings, and revise the structure of a subsequent performance.

Da Yin Xi Sheng ultimately tests whether a system comprising environmental recordings, real-time sound processing, gestural control, spatial audio, and graphic notation can function as a stable, playable, recordable, and repeatably testable live electronic instrument.