From Gesture to Phrase: Real-Time Dance-to-Music Recomposition by Zhitao Lin

  • Published on 09/02/2026 by maxlzt
Computer Assisted Composition Dance–Music Interaction Gesture–Sound Interaction Machine learning Phrase Modeling Real-Time Interaction

A real-time system that analyzes dance movement as phrase-level musical information, transforming motion phases into structured musical responses, timbral changes, ornaments, sample recomposition, and silence.

➡️ This talk is part of IRCAM Forum Workshops Hors-les-Murs Tokyo 2026


From Gesture to Phrase presents the continued development of Phrase-Based Motion Analysis for Real-Time Dance-to-Music Recomposition, a system that interprets live dance movement as musical structure. Developed from the interactive framework of Liminal, the project shifts from open-ended audience interaction toward the temporal phrasing of a dancer.

A camera-based pose-tracking system extracts body-relative hand motion, limb movement, energy, suddenness, motion attack, and changes across a dance phrase. A phrase-and-accent model organizes these features into attack, sustain, release, and stillness. Rather than triggering an individual note for every detected gesture, each movement phase opens a musical response window.

Within this window, the system controls note density, register, articulation, duration, ornament, timbre, and silence. A temporal model generates melodic material, while a sample-recomposition layer selects and reshapes textures, patterns, melodic fragments, and effects in Ableton Live. Strong movement may produce an accent or timbral shift; sustained movement may develop a longer musical phrase; release may lead to a harmonic tail or silence.

During the demonstration, I will trace a dance phrase through camera input, MediaPipe pose tracking, feature extraction, temporal interpretation, musical decision-making, and real-time output. The presentation will compare direct movement triggering with phrase-based response and show how thresholds, smoothing, phase detection, and live control affect the musical result.

The dance video shown in this technical demonstration was sourced from publicly available online material. We do not claim copyright; all rights remain with the original creator or rights holder.

 

The project proposes a performer-centered approach to dance–music interaction. Its central contribution is a phrase-level mapping layer that connects motion analysis, machine-learning models, musical generation, sample recomposition, and performer feedback without reducing the dancer’s body to a collection of isolated control signals.