a movement-driven multimedia performance

This week we ran the first test of the full-body gesture-driven audio and visual performance system, downtown at our warehouse space.
The concept is a performance involving live music, video and lighting, controlled in real-time by an improvising dancer.
We're basically turning Forest's body into an instrument, expanding the range of her dance performance from movement into music and video art.

In this demonstration piece, lower body movements are associated with the bass parts. Initially, by moving from a crouched position to standing, Forest brings in the bass arpeggios; after this point movement in the hips and knees control the mix and interplay between four synth bass lines. These motions are also controlling the compositing of the background elements in the video mix (in this cases a series of animated alchemical symbols).
The arms are controlling spacious pads in the audio mix, and animated typography in the video mix, as well as bringing up and down lighting effects.
Rapid motion of the head (or a spin of the entire body) and quick gestures of the hands invoke musical pads, swells and hits with accompanying animations in the video mix synced with flashes from the lighting at either side of the performance space. Imagine a wizard making things happen with a flick of her hand and you get the idea.

Some technical details: the sensor rig tracks rotation at the hip and knee joints and elbow (potentiometers), rotational acceleration of the head/body (head-mounted gyroscope), hand movement (accelerometers) and finger flex (flex sensor). These signals run though a wireless A-D system and are fed to a Max patch running on the Mac for initial processing (filtering, scaling, etc.).



MIDI continuous controllers are then fed to several softsynths (mostly Rob Papen Albinos, in this case) hosted in Ableton Live (which is also controlling the overall tempo of the performance and driving arpeggiators and a backing rhythm part) and also to a hybrid Director/Flash piece running in the ThinkPad.


The Director/Flash application generates real time animations based on the movement gestures, which are scan-converted and fed to the projector. Gestures also control LED lighting fixtures via DMX.