David Franzson - Autocoder: a Spectrum Based Variational Autoencoder Environment

Abstract :

The Autocoder package is a tool based around a variational autoencoder––a neural network capable of learning a spectral representation of a soundfile and synthesizing a novel output based on the trained model.

A spectral representation is extracted from an input sound by running it through an STFT analysis, producing a number of spectral frames representing the sound moment by moment. The software through an encoder that compresses the data into a latent layer. The latent  layer has eight values, each encoding some aspect of the input data. By feeding the training data through the encoder after training. By feeding arbitrary values as input into the latent vector layer, the encoder returns a new spectral frame that represents an unseen point within the spectral space of the training data and can then be used in any number of ways, e.g. for synthesis or convolution, as an impulse response in a hybrid reverb, or for cross–synthesis.

The package provides a simple and easily extendable ecosystem to assist with the experimentation and development of sound software and hardware based on the underlying neural network architecture. It is available both in code and in hardware form and comes with an osc interface allowing for easy integration with Max/MSP.

The code is available HERE 

Sound examples are available HERE

Bio :

Projects include an Urban Archive as an English Garden, developed in collaboration with Halla Steinunn Stefansdottir (Nordic Affect), Russell Greenberg (Yarn/Wire), Julia Mogensen, and Matt Barbier (wasteLAnd), developed during a 2019 Artistic Resarch Residency at Ircam/ZKM; the Negotiation of Context, developed in collaboration with Yarn/Wire--described as "engagingly tactile" by the NY Times, "compelling" by the Wire which selected the release as one of their top 10 modern composition releases of 2014, and as "sonic art that is clearly going places" by Gramophone; the cello concerto on Matter and Materiality, commissioned by the BBC Scottish Symphony Orchestra and the Icelandic National Radio's commissioning fund--described as "strikingly static" by the Guardian; and the Cartography of Time, an ongoing exploration of the experience of time, developed in collaboration with Gnarwhallaby, Vicky Chow (Bang on a Can), Mariel Roberts, Matt Barbier and Weston Olencki (RAGE thrombones), Rusell Greenberg (Yarn|Wire), Matthias Engler, and Ingolfur Vilhjalmsson (Ensemble Adapter).

@David Brinjar

Back to the abstracts collection