Projected Multitouch User Interface

As part of my MA, I created a collaborative, educational, multi touch musical instrument that allows users to create various different kinds of sound waves (sine, sawtooth, square and triangle) modify the music note pitch, and view the visual representation of the resulting change on the sound wave. When a user tapped the “Sawtooth” or “Sine” buttons, a musical note would begin to play. The user can then drag their fingers to increase or decrease the frequency of the note, view the change on the visual representation of the sound wave, and hear the resulting change in pitch. The musical notes, represented as circles on the projected interface, were also draggable. It was also possible to create combinations of the different types of sound waves by switching multiples on at the same time.


There were several parts to the construction of this project that did not involve programming or design. For this project, I created a wooden box to house the projector, an infrared camera and IR lamps. A piece of perspex was used to create a transparent surface and placed on the lid of the wooden box, to which I attached a piece of semi-opaque parchment paper which reduced the possibility of the IR camera reading the projected interface, causing visual feedback.


An Infrared camera was created by purchasing a used Sony Playstation Eye Camera on ebay, taking off the lens and covering the lens with standard camera film to enable it to block out light that was not Infra Red.

How it Works

When a user placed a finger on the semi-transparent surface, the Infrared light was diffused, creating a mark that can be read by the Infrared camera. The camera is connected to a laptop which is running the ReacTIVision software, which converts the human fingerprint into a value, including the coordinates of the 'blob'.


One of the main challenges of this particular project was the fact that we were going to change the user behaviour from asking for the user's postcode at the beginning of the flow, to asking for their full address (postcode, house number and street name). For this reason, the 'address flow' became one of the biggest design challenges, and required several iterations, along with being the primary focus of the user testing sessions.

Final Video