Moving Towards Synchrony

This is a link to the video presentaton: 
https://vimeo.com/514333273 

Introduction:

My name is Johnny Tomasiello and I am a multidisciplinary artist and composer, living and working in New York. 

My piece, titled Moving Towards Synchrony, version 3, is an immersive work whose purpose is to explore the reciprocal relationship between electrical activity in the brain and external stimuli that has been generated -and defined by- those same physiological events.

It investigates the neurological effects of modulating brain waves and their corresponding physiological effects by use of a Brain-Computer Music Interface, which allows for the sonification of the data captured by an electroencephalogram.

The work presents an interactive computer-assisted compositional performance system that can teach participants how to influence a positive change in their own physiology by learning to influence the functions of the autonomic nervous system through neuro- and bidirectional feedback. 

The methodology involves collecting physiological data through non invasive neuroimaging. A subject’s brainwaves are used to generate realtime interactive music compositions which are simultaneously experienced by that subject. The melodic and rhythmic content, are derived from, and constantly influenced by, the subject’s EEG readings. A subject, focusing on the generative stimuli, will attempt to elicit a change in their physiological systems through their experience of the bidirectional feedback. The resulting physiological responses will be recorded and measured to determine the efficacy of using external stimuli to affect the human body both physiologically and psychologically.

EEG brainwave data has shown high levels of success in classifying mental states [1], which affect “autonomic modulation of the cardiovascular system” [2], and there are existent studies investigating how music can influence a response in the autonomic nervous system. [3] It is with these phenomena in mind that this work was created. 

Increased activity in the alpha wave frequency range is “usually associated with alert relaxation”. [4] Methods intended to increase activity in the alpha wave frequency range through feedback, autogenic meditation, breathing exercises, and other techniques, is called alpha training.

Positive changes in alpha is what I am primarily concerned with here, since research has shown that stimulating activity within alpha causes muscle relaxation, pain reduction, breathing rate regulation, and decreased heart rate. [4] [5] [6] It has also been used for reducing stress, anxiety and depression, and can encourage memory improvements, mental performance, and aid in the treatment of brain injuries.

In addition to investigating these neuroscience concerns, this work is designed to explore the validity of using the scientific method as an artistic process. The methodology will be to create an evidence-based system for the purpose of developing research based projects. This will limit, initially, subjective interpretation of the work and will encourage a mindful and intentional interaction with the experience itself. What is learned will determine the value of the work.

As Gita Sarabhai expressed to John Cage "...music conditions one's mind, leading to ‘moments in [one's] life that are complete and fulfilled’” [5]. Music, in this case, can also be used by the mind to condition one's body.

Information on EEG:

An electroencephalogram (also know as an EEG) is an electrophysiological monitoring method used to record the electrical activity of the brain. A typical adult human EEG signal is between 10 and 100 µV (microvolts) in amplitude when measured from the scalp. It was invented by German psychiatrist Hans Berger in 1929 and research into how brainwaves can be interpreted and modulated started as shortly thereafter.  Using an EEG, you are able to directly measure neural activity and capture cognitive processes in real time. Berger proved that alpha waves (also know as Berger waves) were generated by cerebral cortical neurons.

In 1934, English physiologists Edgar Adrian and Brain Matthews first described the sonification of alpha waves derived from EEG data. [8] They found that “non-visual activities which demand the entire attention (e.g. mental arithmetic) abolish the waves; sensory stimulation which demand attention also do so” [9], showing how concentration and thought processes affected activity in the alpha wave frequency range.

The brain wave activity recorded in an EEG is a summation of the inhibitory and excitatory post synaptic potentials that occur across a neuronal membrane. [10]

The measurements are taken by way of electrodes placed on the scalp.  The readings are divided into five frequency bands, delineating slow, moderate, and fast waves.  The bands, from slowest to fastest are:

Delta, with a range from approximately 0.5Hz–4Hz, 

which signifies deepest meditation or dreamless sleep

Theta, from approximately 4Hz–8Hz, 

signifying meditation or deep sleep. 

Alpha, from approximately 8Hz–13Hz, 

representing quietly flowing thoughts.

Beta, from approximately 13Hz–30Hz, 

which is a normal waking state.

And 

Gamma, from approximately 30Hz–42Hz 

which is most active during simultaneous processing of information that engages multiple different areas of the brain.

History of EEG use in music:

Physicist Edmond Dewan began the study of brainwaves in the early 1960s and developed a ‘brainwave control system’.  The system detected changes in alpha rhythms which were used to turn lighting on or off. “The light could also be replaced by ‘an audible device that made a beep when switched on’, allowing Dewan to spell out the phrase ‘ I can talk ’ in Morse code”. [8] Dewan met experimental composer Alvin Lucier which inspired the first actual brainwave composition.

Alvin Lucier first performed Music For Solo Performer in 1965. It involved the composer sitting in a chair on stage, with his eyes closed while his brainwaves were recorded.  The data from the recording was amplified and distributed to speakers set up around the room.  The speakers were placed against different types of percussion instruments, so the vibration of the speakers would cause the instrument to sound.

Lucier was able to control the percussion events through control of his cognitive functions, and found that a break in concentration would disrupt that control.  Although mastery over the alpha rhythm was (and is) difficult, Music for the solo performer greatly contributed to the field of experimental music and illustrated the depth of possibility in using EEG control over musical performance.

Computer scientist Jaques Vidal published the paper Toward Direct Brain-Computer Communication in 1973, which first proposed the Brain-Computer Interface (BCI), which is a means of using the brain to control external devices. 

This was the very beginning of BCMI research, which has evolved into an interdisciplinary field of study “at the crossroads of music, science and biomedical engineering” [11]. BCMIs (also referred to Brain Machine Interfaces, or BMIs) are still in use today, and the field of research around them is in its infancy.

Project Overview:

This project records EEG signals from the subject using four non-invasive dry extra-cranial electrodes from a commercially available MUSE EEG headband. Measurements are recorded from the TP9, AF7, AF8, and TP10 electrodes, as specified by the International Standard EEG placement system, and the data is converted to absolute band powers, based on the logarithm of the Power Spectral Density (PSD) of the EEG data for each channel. Heart rate data is obtained through PPG measurements, although that data is not used in the current version of this project. EEG measurements are recorded in Bels/Db to determine the PSD within each of the frequency ranges.

The EEG readings are translated into music in real time, and the subjects are instructed to employ deep breathing exercises while they focus on the musical feedback.

Great care was taken in defining the compositional strategies of the interactive content in order to deliver a truly generative composition that was also capable of producing musically recognizable results. 

All permutations of the scales, modes and chords being used, as well as rhythms, and performance characteristics, needed to be considered beforehand so the extraction of a finite set of parameters from the EEG data set could be parsed and used to produce a well-formed and dynamic piece of music.

There are 3 main sections of this Max patch:

1: The EEG data capture section.

2: The EEG data conversion section.

3: the Sound generation and DSP section.

The EEG data capture section receives EEG data from the Muse headband, which is converted to OSC data and transmitted over WiFi via the iOS app Mind Monitor.  That data is then split into the five separate brainwave frequency bandwidths: delta, theta, alpha, beta and gamma.  Additional data is also captured, including accelerometer, gyroscope, blink and jaw clench, in order to control for any artifacts in the data capture.  Sensor connection data is used to visualize the integrity of the sensor’s attachment to the subject. PPG data is also captured for use in a future iteration of the project.

The EEG data conversion section accepts the EEG bandwidth data representing specific event-related potential, and translates it to musical events. 

First, significant thresholds for each brainwave frequency bandwidth are defined.  These are chosen based on average EEG measurements taken prior to the use of the musical feedback. When those thresholds are reached or exceeded, an event is triggered.  Depending on the mappings, those events can be one or more of several types of operations: the sounding of a note, a change in pitch or scale or mode, note values and timings, and/or other generative performance characteristics.

This section is comprised of three subsections that format their data output differently, depending on the use case:
1. Internal Sound Generation and DSP for use completely within the Max environment.

2. External MIDI for use with MIDI equipped hardware or software.

and 

3. External Frequency and gate, for use with modular synthesizer hardware.

Each of these can be used separately or simultaneously, depending on the needs of the piece. 

For the data conversion, the event-related potentials are mapped in the following way:
Changes in alpha, relative to the predefined threshold, govern the triggering of notes, as well as the scale and mode.

Changes in theta, relative to the threshold, influence note value. 

Changes in beta, relative to the threshold, influence spatial qualities like reverberation and delay.

Changes in delta, relative to the threshold, influence the degree of spatial effects.

Changes in gamma, relative to the threshold, influence timbre.

Any of these mappings or threshold decisions can be easily changed to accommodate a different thesis or set of standards.

The third section is Sound generation and DSP. It is responsible for the sonification of the data translated from the EEG data conversion section. This section includes synthesis models, timbre characteristics, and spatial effects.

This projects uses three synthesized voices created in Max 8 for the generative musical feedback.  There are two subtractive voices that each use a mix of sine, sawtooth and triangle waves, and one fm voice.  

The timbral effects employed are waveform mixing, frequency modulation, and high pass, band pass and low pass filters. The spatial effects used include reverberation, and delay.  In addition to the initial settings of the voices, each of the timbral and spatial effects are modulated by separate event-related potential data captured by the EEG.

Conclusions:

This project is a contemporary interpretation of an idea I've been interested in for many years, starting with investigation into bidirectional EKG biofeedback. 

My initial experience with the subject was during a university degree in psychophysics (a branch of psychology). Some promising research at the university focused on reducing stress in asthmatic subjects for the purposes of lessening the frequency of attacks. [12]

At the time, the technology required to explore this idea was of considerable size, and prohibitively expensive, for all but medical or formally funded academic purposes. With the current availability of low-cost electroencephalography (EEG) devices and heart rate monitors, the possibility of autonomous exploration of these concepts has become a reality.

The procedure, when using this work for the exploration of the physiological effects of neuro- and bi-directional feedback, starts with obtaining and comparing 2 data sets: a control and a therapeutic data set.  The control set records EEG data without utilizing musical feedback or breathing exercises.  The therapeutic set records EEG data with the feedback and breathing exercises.

Although this project is primarily concerned with changes in the alpha EEG brainwave frequency range, changes in other frequency ranges were used to trigger events in the feedback. This approach was adopted to ensure that a subject’s loss of focus (and/or a drop in the PSD of alpha) would not negatively affect the generation of novel musical feedback, and with the help of consistent feedback, the subject would be able to return their focus and continue. Depending on the subject’s state of relaxation (and the PSD of the other four EEG frequency ranges measured), the performance and phrasing of the musical feedback would change in such a way as to encourage greater focus.

For the initial proof of concept trials, I tested myself and a small sampling of other subjects. Preliminary data shows that alpha readings were higher, on average, during the therapeutic phase.  Also, a higher overall peak value was achieved during the therapeutic phase This suggests that this feedback model is an effective way of increasing activity in the alpha brainwave frequency range, which is the beneficial physiological and psychological effect I was hoping to find, although much more data needs to be collected before any definitive conclusions can be drawn. At this point, the system has been tested and is functional, and further research can begin. The modular design of the work allows for most any variable to be included or excluded, which will be necessary moving forward with the research, in order to more thoroughly test the foundational elements of the thesis, as well as any musicological exploration and analysis that defining the feedback raises. 

In the meantime, I am already using the software as a compositional system to create recorded works and live soundtracks. I am also planning to mount the project as an interactive installation in a gallery setting.

Contact Details:

Johnny Tomasiello

johnnytomasiello@gmail.com

Credits & Acknowledgments:

IRCAM

Cycling ’74

Carol Parkinson, Executive Director of Harvestworks

Melody Loveless, NYU & Max certified trainer

Dr. Paul M. Lehrer and Dr. Richard Carr

InteraXon Muse electroencephalography headband 

James Clutterbuck (Mind Monitor developer)

References:

[1] “Mental Emotional Sentiment Classification with an EEG-based Brain-Machine Interface.” 

Bird, Jordan J.; Ekart, Aniko; Buckingham, Christopher D.; Faria, Diego R., 2019

[2] “Effects of mental state on heart rate and blood pressure variability in men and women.” 

K Madden , G K Savard, 1995

[3] “How Can Music Influence the Autonomic Nervous System Response in Patients with Severe Disorder of Consciousness?” 

Francesco Riganello,* Maria D. Cortese, Francesco Arcuri, Maria Quintieri, and Giuliano Dolce, 2015

[4] Neurofeedback: A Comprehensive Review on System Design, Methodology and Clinical Applications

Hengameh Marzbani, Hamid Reza Marateb, and Marjan Mansourian, 2016

[5] Stress Management Techniques: Are They All Equivalent, or Do They Have Specific Effects?

Paul M. Lehrer and Richard Carr, 1994

[6] Alpha activity and cardiac correlates: three types of relationships during nocturnal sleep

J Ehrhart, M Toussaint, C Simon, C Gronfier, R Luthringer, G Brandenberger, 2000

[7] “A Composer's Confessions" 

John Cage, 1948 

[8] Brainwaves in concert: the 20th century sonification of the electroencephalogram
Bart Lutters, Peter J. Koehler, 2016 

[9] The Berger Rhythm: Potential Changes From The Occipital Lobes in Man, 

Adrian, Matthews.1934

[10] How To Interpret an EEG and its Report

Marie Atkinson, MD, 2010

[8] Brainwaves in concert: the 20th century sonification of the electroencephalogram
Bart Lutters, Peter J. Koehler, 2016

[11] Brain-Computer Music Interfacing: Interdisciplinary Research at the Crossroads of Music, Science and Biomedical Engineering
Miranda, ER 2014

[12] Relaxation and Music Therapies for Asthma Among Patients Prestabilized on Asthma Medication

Paul Lehrer, Et al. 1994