Posté par: paik402 Il y a 2 mois

(0 comments)

1. Abstract

The growing demand for spatial audio in immersive media, such as virtual reality (VR) and augmented reality (AR), highlights the need for advanced tools that allow for flexible manipulation of complex sound fields. However, existing techniques for remixing and editing spatial audio—especially in high-resolution formats like 5th-order Higher-Order Ambisonics (HOA)—remain technically challenging.

This ongoing research proposes the development of a comprehensive system leveraging deep learning (DL) and machine learning (ML) for source separation, trajectory tracking, and reverberation estimation within 5th-order ambisonic audio environments. The system aims to provide seamless manipulation of individual sound sources (from mono up to 5th-order ambisonics), spatial trajectories, and environmental reverberations, enabling the flexible exchange, removal, or addition of specific audio elements across different spatial mixes.

By offering detailed control over sound sources and their movements within a 3D soundfield, this system opens up new possibilities for spatial audio remixing. The ultimate goal is to develop a system that is both automated and adaptable, capable of addressing the complex audio needs of VR, AR, and other immersive media applications.

Current progress includes the creation of a multi-format dataset containing mono, ambisonic, and XYZ trajectory data, alongside the ongoing development of multichannel source separation models. These advancements pave the way for an efficient and intuitive system for spatial audio editing.

 

2. Background and Motivation

Spatial audio, particularly ambisonic audio, plays a vital role in immersive media such as VR, AR, gaming, and film production. Higher-order ambisonics, like 5th-order ambisonics, provide high-resolution, 3D soundscapes that greatly enhance immersion. Although ambisonics can be adapted into different formats using spherical harmonics and open-source tools, their manipulation remains challenging, often requiring specialized equipment and advanced digital audio workstations (DAWs).

While the video industry has made great strides in object replacement, background compositing, and seamless scene manipulation through ML/DL-powered tools, audio editing has not reached the same level of flexibility. Spatial audio remixing and sound source manipulation, such as adding, removing, or exchanging sources, trajectories, or reverberations, remain far more complex, especially when dealing with ambisonic audio in VR/AR applications. The need for specialized equipment and software complicates the process, limiting creative possibilities for audio engineers and content creators.

3. Research Objectives

The primary goal of this research is to develop an automated system for 5th-order ambisonic audio remixing, using a data-driven approach to enable sound source separation, trajectory tracking, and reverberation adjustments. These controls will allow tasks such as removing, pasting, swapping, or modifying sound sources, changing their position and movement, and adjusting or removing reverberation. The specific objectives are:

• Source Separation: Develop a deep learning model that accurately identifies and separates multiple sound sources from a 5th-order ambisonic mix, outputting them as dry mono sources for easy manipulation. This provides precise control over individual elements within the mix.

• Trajectory Tracking: Utilize machine learning techniques to extract and modify the 3D spatial trajectories (x, y, z coordinates) of the separated sound sources, enabling precise control over their movement within the soundfield.

• Reverberation Modeling (RIR Extraction): Analyze and model room impulse responses (RIR) to accurately capture environmental acoustics, allowing for realistic reverberation control during the remixing process.

• Spatial Remixing: Develop a framework that supports remixing tasks such as swapping sound sources between ambisonic mixes, modifying their trajectories, or adjusting reverberations, all while maintaining the spatial integrity of the original soundfield.

4. System Components

The proposed system comprises the following components:

• Source Separation Module: A deep learning model that separates sound sources from the input ambisonic mix, outputting them as individual mono channels.
• Trajectory Tracking Module: Uses machine learning to track and modify the 3D spatial trajectories (x, y, z coordinates) of sound sources for remixing.
• RIR Extraction Module: Estimates room impulse responses to model the acoustic characteristics of the environment, allowing realistic reverberation control.
• Spatial Remixing Framework: Provides tools to swap, move, or modify sound sources and backgrounds based on extracted data, enabling flexible spatial remixing.
• Ambisonic Re-encoding Component: Re-encodes the remixed audio into a 5th-order ambisonic format to ensure spatial accuracy and VR/AR compatibility.

 

5. Expected Benefits

The outcomes of this research will offer key benefits such as:

• Seamless control of various audio formats, from mono to 5th-order ambisonics, within a single system—eliminating the need for ambisonic microphones and enabling comprehensive spatial audio control.
• Automated and efficient source separation for complex ambisonic audio mixes, simplifying the process of isolating sound sources.
• Accurate trajectory tracking in 3D space, allowing for precise and dynamic manipulation of sound sources within the soundfield.
• Enhanced spatial remixing tools that enable easy swapping, adjusting, and editing of sound sources and backgrounds.
• Improved accessibility for audio engineers, VR/AR developers, and content creators by reducing reliance on separate modules, specialized hardware, or complex software.
• Broader applications in immersive media, including VR, AR, gaming, and cinematic experiences, where precise spatial audio placement and manipulation are essential.

 

6. Current Progress

The project is currently ongoing, with the following tasks completed and in progress:

• Dataset Development: A comprehensive dataset containing mono sources, 5th-order ambisonic recordings, and corresponding 3D spatial trajectory pair data has been completed. This dataset is derived from the author's exhibition and music projects, comprising 110 tracks featuring a variety of sounds, including soundscapes and vocals. The dataset, titled 'AMBISONIC-DML: A Higher-Order Ambisonics Audio Dataset for Spatial Audio Research and Creative Applications', has been submitted to ICASSP 2025 for review.
• Multichannel Source Separation: We are currently working on implementing a multichannel source separation model capable of handling complex ambisonic mixes. This involves training deep learning models to effectively isolate individual sound sources within the ambisonic soundfield, enhancing the separation capabilities for spatial audio remixing.

Partager sur Twitter Partager sur Facebook

• ← Modular and Interfaced Spat5 SPAT in Max for Live device “TSofL” by Jing-Shiuan Tsang
• Ircam tools overview for creative sound design and generative music production by Guillaume Piccarreta and Paola Palumbo →