Breath and Brass

2013-11-10 14.14.12

“Breath and Brass” visualizes the act of musical performance – both the physicality of the musician’s breathing and the technical nature of the generated music. It newly illuminates the interplay between the instruments and incorporates a more conscious awareness of the musician’s exertion into the audience’s experience.

Working with Poulenc’s Sonata for Horn, Trumpet and Trombone we visualize inhales and exhales together with the notes, volume and speed of the music to reveal the connection between the physicality of music making and the relationships of the sounds and instruments to each other.

2013-11-10 14.14.00

The project relies upon audio analysis and breathing measurement.
The audio analysis uses Fast Fourier Transform (FFT) thanks to an existing code library written in C++. The library breaks the audio signal into 17 channels of different frequencies from low to high and also provides additional data such as amplitude.


2013-11-10 14.17.32

To measure the breathing we built custom hardware, consisting of a velcro and elastic band that wraps around the musician’s torso. The front of the band is made of conductive rubber, which measures the expansion and contraction as he or she breathes. A WiFly unit attached to the band sends the data wirelessly to the computer running the software.


Breathing Sensor Fritz

We integrated all the data into two narratives, dubbed “Orbits” and “Lines.” “Orbits” consists of a series of concentric circles of increasing radius. The difference between radii maps to the breathing data, so the circles expand and contract. There is one circle for each of the 17 bands of frequency data, and activity on each frequency populates the respective orbit with orbiting “planetoids.”
“Lines” consists of three creatures made up of lines, one creature for each musician. The breathing data controls the length of the lines themselves, causing the creatures to grow and shrink, while the amplitude of the audio maps to the opacity, causing the creatures to brighten and fade.

Screen captures of the two movements can be found in the Google Drive folder.

photo 3 copy

Final Software Movement 1
Final Software Movement 2
Final Firmware
Full code repository

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Creative Commons License

Wieniawski Violin Etudes


The Wieniawski Violin Etudes are pieces for students meant to represent the core virtuosic abilities of a accomplished violinist, with each one focusing on one of these fundamentals. Our task was to create a visual system for these Etudes to tell a story to the audience outside of the music alone.

We were interested in a strong narrative component, one that works more on a human emotional level than just purely a machine translation of sound to light. The first thing we did to prepare was, of course, listen to the tracks quite a bit, then ask ourselves, “What is the the story the musicians are trying to tell? Not just the sequence of notes, but the violence of the strokes, the volume, the timbre. All of it combines to make us feel something, but what do we feel?” What we heard and worked with was a story of tension and release, corruption and redemption.

The Etudes we were scripting for were Op. 18 movements II, III, and IV. The second movement relied heavily on a regular geometric rotational system that gets corrupted and re-skinned over and over again. The third movement plays with the idea of corruption by introducing different particle systems to play over the notes of the piece. And finally, the fourth movement explores the motion and space between players as the similarities and differences of the two violinists music and motions would determine the space and speed of the objects on the screen.

The way we chose to interact with the performers was through physical motion by using accelerometers, and Fast Fourier Transform (FFT) code to translate pitch, attack, and other aural qualities of the music into visual output, or programmatically manipulatable variables. As for the physical motion, we felt it would be advantageous to connect accelerometers directly to the musicians wrists. This would allow a noise free, one-to-one connection to the movement of the violin bow, yielding orientation and movement data in 3 axes.

In order to wirelessly receive this data from the musicians, we decided to use an Arduino Fio v3 connected directly to a Roving Networks WiFly module in order to stream the data to our applications via wifi. The actual accelerometer readings were collected with an LSM303 accelerometer+magnetometer module. The data from the accelerometer was read by the arduino and sent via wifi using UDP communication to an OpenFrameworks receiver application.

All the code we used for both the arduino and the openframeworks sketches are available on the project repository.

Github Repo


Many more images

Group Null – Adiel Fernandez, Owen Herterich, Yi Ning Huang, K Anthony Marefat, Joseph Moore, Jorge Proano, Tiam Taheri

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

Creative Commons License