Vision Systems Design, an engineering tech journal, has an interesting discussion on the technology behind the Reactable synthesizer interface.

The Reactable, developed by researchers at the Pompeu Fabra University, is a tangible interface for manipulating synth sounds built from an off-the-shelf camera, computer projector, PC, and a custom-built light-table.

The Reactable hardware is based on a translucent, round multitouch surface. The camera beneath the table continuously analyzes the surface, tracking the player’s fingertips and the nature, position, and orientation of physical objects that are distributed on its surface. These objects represent the components of a classic modular synthesizer. The musician can move these objects, changing their distance, orientation, and the relation to each other. These actions directly control the topological structure and parameters of the sound synthesizer. A Sanyo projector, also underneath the table, draws dynamic animations on its surface, providing visual feedback of the state, activity, and main characteristics of the sounds produced by the synthesizer.

The Reactable projects markings onto the surface of the table to make the instrument easy to operate. These confirm to the musician that the object has been recognized and provide additional information regarding the status of the generated tone and its interaction with neighboring objects, allowing the artist to see the connections and a dynamic graphic presentation of the generated sound waves on the table. The musician can change individual sound parameters by touching the projected information with his finger.

After images are captured by the Guppy camera, the images from each Plexiglas object are analyzed and their position and orientation measured. Corresponding sound information is then generated and transferred to the PC’s loud speakers as an audio signal, as well as being graphically projected onto the tabletop.

More at Vision Systems Design.