Moving Towards Synchrony: A Brainwave to Music Translation System

My name is Johnny Tomasiello and I am a multidisciplinary artist and composer- researcher, living and working in New York.

Moving Towards Synchrony: a Brainwave to Music Translation System, is an immersive work whose primary purpose is to explore the reciprocal relationship between electrical activity in the brain and external stimuli that has been generated -and defined by- those same physiological events.

It investigates the neurological effects of modulating brainwaves and their corresponding physiological processes through neuro- and bidirectional feedback through use of a Brain-Computer Music Interface (or BCMI). The BCMI allows for the sonification of the data captured by an electroencephalogram, effectively using the subject’s brainwaves to produce real-time interactive soundscapes that, being simultaneously experienced by the subject, have the ability to alter her or his physiological responses.

This work can be presented as an interactive computer-assisted compositional performance system, and I have staged performances with it to that end.
But its original intent is to directly engage the audience, inviting others to use the system, which could teach them how to affect a positive change in their own physiology by learning to influence the functions of the autonomic nervous system.

While developing the project, I was concerned with maintaining a balance between the mindfulness the experience was meant to inspire, and the meaningfulness of the result. The work demands active engagement from listener, if they are participating directly, and is concerned with staying in the process. This represents, for me, “...a move away from making objects to making processes” [1], as well as a move away from the subjective, where the process, the experience, and the quantitative and qualitative analysis of those things, are more significant than anything that’s produced as a result.

In addition to investigating these neuroscience concerns, this project is designed to explore the validity of using the scientific method as an artistic process. The methodology will be to create an evidence-based system for the purpose of developing research-based projects.

As Gita Sarabhai expressed to John Cage “Music conditions one’s mind, leading to ‘moments in [one’s] life that are complete and fulfilled’.” [2]. Music, in this case, can also be used by the mind to condition one’s body.

Basis:

The research methodology explores how to collect and quantify physiological data through non-invasive neuroimaging. The melodic and rhythmic content are derived from, and constantly influenced by, the subject’s brainwave readings. A subject, focusing on the musical stimuli, will attempt to elicit a change in their physiological systems through the experience of the bi-directional feedback system.

The resulting physiological responses will be recorded and, along with the results of other subject’s data sets, quantitative analysis performed, to determine the efficacy of using external stimuli to affect the human body, both physiologically and psychologically.

Brainwave data captured by an EEG has shown high levels of success in classifying mental states [3], which affect “autonomic modulation of the cardiovascular system” [4], Furthermore, there are existent studies investigating how music can influence a response in the autonomic nervous system. [5]

This work is particularly interested in the amount of activity in the alpha brainwave frequency range. Increased activity in the alpha wave frequency range is “usually associated with alert relaxation”. [6] Methods intended to increase activity in the alpha wave frequency range through feedback, autogenic meditation, breathing exercises, and other techniques, are classified as alpha training.

Brainwaves are generally faster and have higher frequencies during wakefulness, and occur at a lower frequency during deep sleep. Although alpha waves can occur between alertness and the beginnings of sleep, there is a difference between the physiological benefits of sleep, and those reported when there is greater activity in alpha. Perhaps the most basic distinction between alpha training and sleep is the conscious awareness and regulated breathing patterns - with the ability to control and adapt the alpha training for maximum benefit.

Positive change (compared with the control group) in the amount of activity in alpha is what I will investigate here, since research has shown that stimulating activity within alpha causes muscle relaxation, pain reduction, breathing rate regulation, and decreased heart rate [6] [7] [8], This has also been used for reducing stress, anxiety and depression, and can encourage memory improvements, mental performance, and aid in the treatment of brain injuries. It is with these phenomena in mind that this work was first conceived and developed.

Information on EEG:

An electroencephalogram (also know as an EEG) is an electrophysiological monitoring method used to record the electrical activity of the brain. A typical adult human EEG signal is between 10 and 100 μV (microvolts) in amplitude when measured from the scalp. It was invented by German psychiatrist Hans Berger in 1929 and research into how brainwaves can be interpreted and modulated started as shortly thereafter. Using an EEG, you are able to directly measure neural activity and capture cognitive processes in real-time. Berger proved that alpha waves (also initially know as Berger waves) were generated by cerebral cortical neurons.

In 1934, English physiologists Edgar Adrian and Brain Matthews first described the sonification of alpha waves derived from EEG data. [9] In doing so, they found that “non-visual activities, which demand the entire attention of the subject (e.g. mental arithmetic) abolish the waves; other sensory stimulation which demand attention also do so” [10], showing how concentration and thought processes affected activity in the alpha wave frequency range.

The brainwave activity recorded in an EEG is a summation of the inhibitory and excitatory post synaptic potentials that occur across a neuronal membrane. [11]

The measurements are taken by way of electrodes placed on the scalp. The readings are divided into five frequency bands, delineating slow, moderate, and fast waves. The bands, from slowest to fastest are:

Delta, with a range from approximately 0.5Hz–4Hz, which signifies deepest meditation or dreamless sleepTheta, from approximately 4Hz–8Hz,
signifying meditation or deep sleep.

Alpha, from approximately 8Hz–13Hz, representing quietly flowing thoughts. Beta, from approximately 13Hz–30Hz, which is a normal waking state.

And
Gamma, from approximately 30Hz–42Hz
which is most active during simultaneous processing of information that engages multiple different areas of the brain.

History of EEG use in music:

Physicist Edmond Dewan began the study of brainwaves in the early 1960s and developed a ‘brainwave control system’. The system detected changes in alpha rhythms which were used to turn lighting on or off. “The light could also be replaced by ‘an audible device that made a beep when switched on’, allowing Dewan to spell out the phrase ‘ I can talk’ in Morse code”. [9] Dewan subsequently met experimental composer Alvin Lucier which inspired the first actual brainwave composition.

Alvin Lucier first performed Music for Solo Performer in 1965. It involved the composer sitting in a chair on stage, with his eyes closed while his brainwaves were recorded. The data from the recording was amplified and distributed to speakers set up around the room. The speakers were placed against different types of percussion instruments, so the vibration of the speakers would cause the instrument to sound.

Lucier was able to control the percussion events through control of his cognitive functions, and found that a break in concentration would disrupt that control. Although mastery over the alpha rhythm was (and is) difficult, Music for Solo Performer greatly contributed to the field of experimental music and illustrated the depth of possibility in using EEG control over musical performance.

Computer scientist Jacques J. Vidal published the paper Toward Direct Brain-Computer Communication in 1973, which first proposed the Brain-Computer Interface (BCI), which is a means of using the brain to control external devices.

This was the very beginning of Brain-Computer Music Interfacing (BCMI) research, which has evolved into an interdisciplinary field of study “at the crossroads of music, science and biomedical engineering” [12]. BCMIs (also referred to Brain Machine Interfaces, or BMIs) are still in use today, and the field of research around them is still in its early stages.

Paul Lehrer Ph.D., who I studied under at UMDNJ, contributed significant research to the field of psychophysics from the 1990s to today, with studies on biofeedback and stress management technics. Dr. Lehrer also set standards for music therapies and their uses as relaxation technics and their beneficial physiological affects by testing benefits amongst subjects with asthma. One of his recent research papers from 2014, Heart Rate Variability Biofeedback: How and Why Does it Work? [14] investigated the effectiveness of heart rate variability biofeedback (HRVB) as a treatment for a variety of disorders, as well as its uses for performance enhancement.

Project Overview:

This project records EEG signals from the subject using four non-invasive dry extra- cranial electrodes from a commercially available MUSE EEG headband. Measurements are recorded from the TP9, AF7, AF8, and TP10 electrodes, as specified by the International Standard EEG placement system, and the data is converted to absolute band powers, based on the logarithm of the Power Spectral Density (PSD) of the EEG data for each channel. Heart rate data is obtained through PPG measurements (although that data is not used in the current version of this project). EEG measurements are recorded in Bels/Db to determine the PSD within each of the frequency ranges.

The EEG readings are translated into music in real time, and the subjects are instructed to employ deep breathing exercises while they focus on the musical feedback.
The time-base for the musical events can be variable and based on the brainwave data, or set to a fixed clock, or some combination of the two .

The use of scales, modes and chords, as well as rhythms, and performance characteristics, needed to be considered beforehand so the extraction of a finite set of parameters from the EEG data set could be parsed and used to produce a well-formed, dynamic, and recognizable piece of music.

There are 3 main sections of this Max patch:

1: The EEG data capture section.
2: The EEG data conversion section.
3: the Sound generation and DSP section.

EEG data capture

The EEG data capture section receives EEG data from the Muse headband, which is converted to OSC data and transmitted over WiFi via the iOS app Mind Monitor. That data is then split into the five separate brainwave frequency bandwidths: delta, theta, alpha, beta and gamma. Additional data is also captured, including accelerometer, gyroscope, blink and jaw clench, in order to control for any artifacts in the data capture. Sensor connection data is used to visualize the integrity of the sensor’s connection to the subject. PPG data is also captured for use in a future iteration of the project.

EEG data conversion
The second section, EEG data conversion, accepts the EEG bandwidth data

representing specific event-related potential, and translates it to musical events.

First, significant thresholds for each brainwave frequency bandwidth are defined. These are chosen based on average EEG measurements taken prior to the use of the musical feedback. When those thresholds are reached or exceeded, an event is triggered. Depending on the mappings, those events can be one or more of several types of operations: the sounding of a note, a change in pitch or scale or mode, note values and timings, and/or other generative performance characteristics.

This section is comprised of three subsections that format their data output differently, depending on the use case:

1. Internal Sound Generation and DSP for use completely within the Max environment.
2. External MIDI for use with MIDI equipped hardware or software.
and

3. External Frequency and gate, for use with modular synthesizer hardware.
Each of these can be used separately or simultaneously, depending on the needs of the work.

For the data conversion in this iteration of the project, the event-related potentials are mapped in the following way:
Changes in alpha, relative to the predefined threshold, govern the triggering of notes, as well as the scale and mode.

Changes in theta, relative to the threshold, influence note value.
Changes in beta, relative to the threshold, influence spatial qualities like reverberation and delay.
Changes in delta, relative to the threshold, influence the degree of spatial effects. Changes in gamma, relative to the threshold, influence timbre.

Any of these mappings or threshold decisions can be easily changed to accommodate a different thesis or set of standards.

Sound generation and DSP

The third section is Sound generation and DSP. It is responsible for the final sonification of the data translated from the EEG data conversion section. This section includes synthesis models, timbral characteristics, and spatial effects.
This projects uses three synthesized voices created in Max 8 for the generative musical feedback. There are two subtractive voices that each use a mix of sine, sawtooth and triangle waves, and one fm voice.

The timbral effects employed are waveform mixing, frequency modulation, and high pass, band pass and low pass filtering. The spatial effects used include reverberation, and delay. In addition to the initial settings of the voices, each of the timbral and spatial effects are modulated by separate event-related potential data captured by the EEG.

Conclusions:

This project is a contemporary interpretation of an idea I've been interested in for many years, starting with investigation into bidirectional EKG biofeedback.

My initial experience with the topic was during a university degree at Rutgers University, in psychophysics (underwritten by The University of Medicine and Dentistry of New Jersey). While at UMDNJ, I worked directly with the doctors who were at the forefront of psychophysiological research, whose work was focused on reducing stress in asthmatic subjects for the purposes of lessening the frequency of attacks. [13]

At the time, the technology required to explore this idea was of considerable size, and prohibitively expensive, for all but medical or formally funded academic purposes. With the current availability of low-cost electroencephalography (EEG) devices and heart rate monitors, the possibility of autonomous exploration of these concepts has become a reality.

The procedure, when using this work for the exploration of the physiological effects of neuro- and bi-directional feedback, starts with obtaining and comparing two data sets: a control and a therapeutic set. The control set records brainwave data without utilizing musical feedback or breathing exercises, while the therapeutic set records the brainwave data with them.

Although this project is primarily concerned with changes in the alpha brainwave frequency range, changes in other brainwave frequency ranges are used to trigger events in the feedback in such a way as to provide cues that a course correction is required by the subject. This approach was adopted to ensure that a subject’s loss of focus (and/or a drop in the Power Spectral Density of alpha) would not negatively affect the generation of novel musical feedback. Depending on the subject’s state of relaxation (and the PSD of the other four EEG frequency ranges measured), the performance and phrasing of the musical feedback is designed to change in such a way that it is expected to encourage greater attention. With the help of consistent feedback, the hope is that the subject would be able to regain their focus.

Preliminary data has shown that alpha readings were higher, on average, during the therapeutic trials. Also, a higher overall peak value was achieved during the therapeutic phase. This suggests that this feedback model is an effective way of increasing activity in the alpha brainwave frequency range, which is the beneficial physiological and psychological effect I was hoping to find, although more data needs to be collected before any definitive conclusions can be drawn.

At this point, the system has been tested and is functional, and further research can begin. The modular design of the system allows for any variables to be included or excluded, which will be necessary moving forward with the research, in order to more thoroughly test the foundational elements of the thesis, as well as any musicological

exploration and analysis that defining the musical feedback raises.

In the meantime, I am already using the software as a compositional and performance system to create recorded works and live performances. I am also planning to mount the project as an interactive installation in a live setting and to create a tangible two- dimensional representation, in some visual language, of each session’s narrative, compressing the entirety of the experience into a single frame.

Credits & Acknowledgments:

IRCAM
Cycling ’74
Carol Parkinson, Executive Director of Harvestworks Melody Loveless, NYU & Max certified trainer
Dr. Paul M. Lehrer and Dr. Richard Carr
InteraXon Muse electroencephalography headband James Clutterbuck (Mind Monitor developer)

Contact Details:

Johnny Tomasiello

johnnytomasiello@gmail.com

References:

[1] Brian Eno. ”Empty Formalism”
Brian Eno in conversation with Thomas Oberender on “Hexadome.”

[2] J. Cage, R. Kostelanetz. John Cage Writer: Previously Uncollected Pieces. New York: Limelight (1993)

[3] J. J. Bird, A. Ekart, C. D. Buckingham, D. R. Faria. “Mental Emotional Sentiment Classification with an EEG-based Brain-Machine Interface”, International Conference on Digital Image & Signal Processing (DISP’19), Oxford, UK (2019)

[4] K. Madden and G.K. Savard. “Effects of Mental State on Heart Rate and Blood Pressure Variability in Men and Women” in Clinical Physiology 15, 557–569 (1995)

[5] F. Riganello et al. “How Can Music Influence the Autonomic Nervous System Response in Patients with Severe Disorder of Consciousness?” in Frontiers in Neuroscience vol. 9, 461 (2015)

[6] H. Marzbani et al. “Methodological Note: Neurofeedback: A Comprehensive Review on System Design, Methodology and Clinical Applications” in Basic and Clinical Neuroscience Journal vol. 7, 143–158 (2016)

[7] P.M. Lehrer and R. Carr “Stress Management Techniques: Are They All Equivalent, or Do They Have Specific Effects?” in Biofeedback and Self-Regulation” (1994)

[8] J. Ehrhart, M. Toussaint, C. Simon, C. Gronfier, R. Luthringer, G. Brandenberger. “Alpha Activity and Cardiac Correlates: Three Types of Relationships During Nocturnal Sleep” in Clinical Neurophysiology vol. 111, 940–946 (2000)

[9] B. Lutters, P. J. Koehler. “Brainwaves in Concert: the 20th Century Sonification of the Electroencephalogram” in Brain 139 (Pt 10), 2809–2814 (2016)

[10] A Matthews, “The Berger Rhythm: Potential Changes From The Occipital Lobes in Man” in Brain 57 Issue 4, (December 1934)

[11] M Atkinson, MD, “How To Interpret an EEG and its Report” (2010)

[12] E.R. Miranda. “Brain–Computer Music Interfacing: Interdisciplinary Research at the Crossroads of Music, Science and Biomedical Engineering” in E.R. Miranda, J. Castet, ed. Guide to Brain-Computer Music Interfacing. London: Springer-Verlag, 1–27 (2014)

[13] P.M. Lehrer et al. “Relaxation and Music Therapies for Asthma among Patients Prestabilized on Asthma Medication” in Journal of Behavioral Medicine 17, 1–24 (1994)

[14] [10] P. M. Lehrer, R. Gevirtz. “Heart Rate Variability Biofeedback: How and Why Does It Work?” in Frontiers in Psychology vol. 5, 756 (2014)

[15] S. A. Plotnikov et al. “Artificial Intelligence-Based Neurofeedback” in Cybernetics and Physics vol. 8, 287–291 (2019)

[16] J. Cage, R. Kostelanetz. John Cage Writer: Previously Uncollected Pieces. New York: Limelight (1993)