Moving Towards Synchrony: A Brainwave to Music Translation System

Moving Towards Synchrony: A Brainwave to Music Translation System is an immersive work whose purpose is to explore the reciprocal relationship between electrical activity in the brain and external stimuli that has been generated -and defined by- those same physiological events.

Biofeedback Dsp Max Max 8 Neurofeedback Neuroscience

Moving Towards Synchrony: A Brainwave to Music Translation System is an immersive work whose purpose is to explore the reciprocal relationship between electrical activity in the brain and external stimuli that has been generated -and defined by- those same physiological events. It investigates the neurological effects of modulating brain waves and their corresponding physiological effects by use of a Brain-Computer Music Interface, which allows for the sonification of the data captured by an electroencephalogram which is translated into musical stimuli in real time.

The work presents an interactive computer-assisted compositional performance system that can teach participants how to influence a positive change in their own physiology by learning to influence the functions of the autonomic nervous system through neuro- and bidirectional feedback. 

The research methodology explores how to collect and quantify physiological data through non-invasive neuroimaging, effectively using the subject’s brainwaves to produce real-time interactive soundscapes that, being simultaneously experienced by the subject, have the ability to alter her/his physiological responses.

The melodic and rhythmic content are derived from, and constantly influenced by, the subject’s EEG readings. A subject, focusing on the generative stimuli, will attempt to elicit a change in their physiological systems through their experience of the bi-directional feedback. 

The resulting physiological responses will be recorded and measured to determine the efficacy of using external stimuli to affect the human body both physiologically and psychologically.

EEG brainwave data has shown high levels of success in classifying mental states [1], which affect “autonomic modulation of the cardiovascular system” [2], Furthermore, there are existent studies investigating how music can influence a response in the autonomic nervous system. [3] It is with these phenomena in mind that this work was first conceived and developed. 

Increased activity in the alpha wave frequency range is “usually associated with alert relaxation”. [4] Methods intended to increase activity in the alpha wave frequency range through feedback, autogenic meditation, breathing exercises, and other techniques, that can be classified as alpha training.

Positive changes in alpha is what I will investigate here, since research has shown that stimulating activity within alpha causes muscle relaxation, pain reduction, breathing rate regulation, and decreased heart rate. [4] [5] [6] It has also been used for reducing stress, anxiety and depression, and can encourage memory improvements, mental performance, and aid in the treatment of brain injuries.

In addition to investigating these neuroscience concerns, this project is designed to explore the validity of using the scientific method as an artistic process. The methodology will be to create an evidence-based system for the purpose of developing research-based projects. This will limit, initially, subjective interpretation of the work and will encourage a mindful and intentional interaction with the experience itself. 

As Gita Sarabhai expressed to John Cage “Music conditions one’s mind, leading to ‘moments in [one’s] life that are complete and fulfilled’.” [5]. Music, in this case, can also be used by the mind to condition one’s body.

 

Information on EEG:

An electroencephalogram (also know as an EEG) is an electrophysiological monitoring method used to record the electrical activity of the brain. A typical adult human EEG signal is between 10 and 100 µV (microvolts) in amplitude when measured from the scalp. It was invented by German psychiatrist Hans Berger in 1929 and research into how brainwaves can be interpreted and modulated started as shortly thereafter.  Using an EEG, you are able to directly measure neural activity and capture cognitive processes in real time. Berger proved that alpha waves (also know as Berger waves) were generated by cerebral cortical neurons.

In 1934, English physiologists Edgar Adrian and Brain Matthews first described the sonification of alpha waves derived from EEG data. [8] They found that “non-visual activities which demand the entire attention (e.g. mental arithmetic) abolish the waves; sensory stimulation which demand attention also do so” [9], showing how concentration and thought processes affected activity in the alpha wave frequency range.

The brain wave activity recorded in an EEG is a summation of the inhibitory and excitatory post synaptic potentials that occur across a neuronal membrane. [10]

The measurements are taken by way of electrodes placed on the scalp.  The readings are divided into five frequency bands, delineating slow, moderate, and fast waves.  The bands, from slowest to fastest are:

Delta, with a range from approximately 0.5Hz–4Hz, 

which signifies deepest meditation or dreamless sleep

Theta, from approximately 4Hz–8Hz, 

signifying meditation or deep sleep. 

Alpha, from approximately 8Hz–13Hz, 

representing quietly flowing thoughts.

Beta, from approximately 13Hz–30Hz, 

which is a normal waking state.

And 

Gamma, from approximately 30Hz–42Hz 

which is most active during simultaneous processing of information that engages multiple different areas of the brain.

 

History of EEG use in music:

Physicist Edmond Dewan began the study of brainwaves in the early 1960s and developed a ‘brainwave control system’.  The system detected changes in alpha rhythms which were used to turn lighting on or off. “The light could also be replaced by ‘an audible device that made a beep when switched on’, allowing Dewan to spell out the phrase ‘ I can talk’ in Morse code”. [8] Dewan subsequently met experimental composer Alvin Lucier which inspired the first actual brainwave composition.

Alvin Lucier first performed Music for Solo Performer in 1965. It involved the composer sitting in a chair on stage, with his eyes closed while his brainwaves were recorded.  The data from the recording was amplified and distributed to speakers set up around the room.  The speakers were placed against different types of percussion instruments, so the vibration of the speakers would cause the instrument to sound.

Lucier was able to control the percussion events through control of his cognitive functions, and found that a break in concentration would disrupt that control.  Although mastery over the alpha rhythm was (and is) difficult, Music for Solo Performer greatly contributed to the field of experimental music and illustrated the depth of possibility in using EEG control over musical performance.

Computer scientist Jacques J. Vidal published the paper Toward Direct Brain-Computer Communication in 1973, which first proposed the Brain-Computer Interface (BCI), which is a means of using the brain to control external devices. 

This was the very beginning of Brain-Computer Music Interfacing (BCMI) research, which has evolved into an interdisciplinary field of study “at the crossroads of music, science and biomedical engineering” [11]. BCMIs (also referred to Brain Machine Interfaces, or BMIs) are still in use today, and the field of research around them is still in its early stages.

Paul Lehrer Ph.D., who I studied under at UMDNJ, contributed significant research to the field of psychophysics from the 1990s to today, with studies on biofeedback and stress management technics.  Dr. Lehrer also set standards for music therapies and their uses as relaxation technics and their beneficial physiological affects by testing benefits amongst subjects with asthma. One of his recent research papers from 2014, Heart Rate Variability Biofeedback: How and Why Does it Work? [13] investigated the effectiveness of heart rate variability biofeedback (HRVB) as a treatment for a variety of disorders, as well as its uses for performance enhancement.

In a December 2019 article published in Cybernetics and Physics [12], Sergei A. Plotnikov, Danila Semenov, and Mikhail Lipkovich used artificial intelligence tools to create deep neural networks for the purpose of increasing the amount of information that make up an EEG data set while maintaining a minimum number of sensors capturing the data.

In the study Asynchronous BCI Based on Motor Imagery With Automated Calibration and Neurofeedback Training [15], Rafal Kus, et al, developed a new multi-class brain-computer interface (BCI) through the application of advanced signal processing tools available with the use of machine learning.

 

Project Overview:

This project records EEG signals from the subject using four non-invasive dry extra-cranial electrodes from a commercially available MUSE EEG headband. Measurements are recorded from the TP9, AF7, AF8, and TP10 electrodes, as specified by the International Standard EEG placement system, and the data is converted to absolute band powers, based on the logarithm of the Power Spectral Density (PSD) of the EEG data for each channel. Heart rate data is obtained through PPG measurements, although that data is not used in the current version of this project. EEG measurements are recorded in Bels/Db to determine the PSD within each of the frequency ranges.

The EEG readings are translated into music in real time, and the subjects are instructed to employ deep breathing exercises while they focus on the musical feedback.

Great care was taken in defining the compositional strategies of the interactive content in order to deliver a truly generative composition that was also capable of producing musically recognizable results. 

Permutations of the scales, modes and chords being used, as well as rhythms, and performance characteristics, needed to be considered beforehand so the extraction of a finite set of parameters from the EEG data set could be parsed and used to produce a well-formed and dynamic piece of music.

 

There are 3 main sections of this Max patch:

1: The EEG data capture section.

2: The EEG data conversion section.

3: the Sound generation and DSP section.

 

The EEG data capture section receives EEG data from the Muse headband, which is converted to OSC data and transmitted over WiFi via the iOS app Mind Monitor.  That data is then split into the five separate brainwave frequency bandwidths: delta, theta, alpha, beta and gamma.  Additional data is also captured, including accelerometer, gyroscope, blink and jaw clench, in order to control for any artifacts in the data capture.  Sensor connection data is used to visualize the integrity of the sensor’s attachment to the subject. PPG data is also captured for use in a future iteration of the project.

The EEG data conversion section accepts the EEG bandwidth data representing specific event-related potential, and translates it to musical events. 


First, significant thresholds for each brainwave frequency bandwidth are defined.  These are chosen based on average EEG measurements taken prior to the use of the musical feedback. When those thresholds are reached or exceeded, an event is triggered.  Depending on the mappings, those events can be one or more of several types of operations: the sounding of a note, a change in pitch or scale or mode, note values and timings, and/or other generative performance characteristics.

 

This section is comprised of three subsections that format their data output differently, depending on the use case:
1. Internal Sound Generation and DSP for use completely within the Max environment.

2. External MIDI for use with MIDI equipped hardware or software.

and 

3. External Frequency and gate, for use with modular synthesizer hardware.

Each of these can be used separately or simultaneously, depending on the needs of the piece. 

 

For the data conversion, the event-related potentials are mapped in the following way:
Changes in alpha, relative to the predefined threshold, govern the triggering of notes, as well as the scale and mode.

Changes in theta, relative to the threshold, influence note value. 

Changes in beta, relative to the threshold, influence spatial qualities like reverberation and delay.

Changes in delta, relative to the threshold, influence the degree of spatial effects.

Changes in gamma, relative to the threshold, influence timbre.

Any of these mappings or threshold decisions can be easily changed to accommodate a different thesis or set of standards.

 

The third section is Sound generation and DSP. It is responsible for the sonification of the data translated from the EEG data conversion section. This section includes synthesis models, timbre characteristics, and spatial effects.

This projects uses three synthesized voices created in Max 8 for the generative musical feedback.  There are two subtractive voices that each use a mix of sine, sawtooth and triangle waves, and one fm voice.  

The timbral effects employed are waveform mixing, frequency modulation, and high pass, band pass and low pass filters. The spatial effects used include reverberation, and delay.  In addition to the initial settings of the voices, each of the timbral and spatial effects are modulated by separate event-related potential data captured by the EEG.

 

Conclusions:

This project is a contemporary interpretation of an idea I've been interested in for many years, starting with investigation into bidirectional EKG biofeedback. 

My initial experience with the subject was during a university degree at Rutgers University, in psychophysics (underwritten by The University of Medicine and Dentistry of New Jersey). While at UMDNJ, I worked directly with the doctors who were at the forefront of psychophysiological research, whose work was rooted in reducing stress in asthmatic subjects for the purposes of lessening the frequency of attacks. [12]

At the time, the technology required to explore this idea was of considerable size, and prohibitively expensive, for all but medical or formally funded academic purposes. With the current availability of low-cost electroencephalography (EEG) devices and heart rate monitors, the possibility of autonomous exploration of these concepts has become a reality.

The procedure, when using this work for the exploration of the physiological effects of neuro- and bi-directional feedback, starts with obtaining and comparing two data sets: a control and a therapeutic one.  The control set records EEG data without utilizing musical feedback or breathing exercises.  The therapeutic set records EEG data with the feedback and breathing exercises.

Although this project is primarily concerned with changes in the alpha EEG brainwave frequency range, changes in other frequency ranges are used to trigger events in the feedback. This approach was adopted to ensure that a subject’s loss of focus (and/or a drop in the Power Spectral Density of alpha) would not negatively affect the generation of novel musical feedback. With the help of consistent feedback, the subject would be able to regain their focus and continue. Depending on the subject’s state of relaxation (and the PSD of the other four EEG frequency ranges measured), the performance and phrasing of the musical feedback would change in such a way as to encourage greater focus.

For the initial proof of concept trials, I tested myself and a small sampling of other subjects. Preliminary data shows that alpha readings were higher, on average, during the therapeutic phase. Also, a higher overall peak value was achieved during the therapeutic phase This suggests that this feedback model is an effective way of increasing activity in the alpha brainwave frequency range, which is the beneficial physiological and psychological effect I was hoping to find, although much more data needs to be collected before any definitive conclusions can be drawn. At this point, the system has been tested and is functional, and further research can begin. The modular design of the work allows for most any variable to be included or excluded, which will be necessary moving forward with the research, in order to more thoroughly test the foundational elements of the thesis, as well as any musicological exploration and analysis that defining the feedback raises. 

In the meantime, I am already using the software as a compositional system to create recorded works and live soundtracks. I am also planning to mount the project as an interactive installation in a live setting.

 

Contact Details:

Johnny Tomasiello
johnnytomasiello@gmail.com