Sunday, November 28, 2010

On Gestures, Kinekt and Beyoind

There has been a great deal of excitement lately about the Kinect video game console. This console uses the motions of the human body rather than a Wii controller or something like it to control a virtual video game world. Many have naturally thought about its adaptation as a musical controller. I will hold off my applause for a while but I wanted to express a few concerns.

One application would be a kind of 3D theremin. The idea would basically be to map parameters representing 3D space to synthesizer parameters. That's fine and a marginal advancement over the theremin but I would not really call it groundbreaking. An example of what I do find at least a little ground breaking is the Eigenharp. It's still simply parameter mapping but there is a very fine degree of control over the controllers on each pad not to mention the 2D array. Many I have discussed this new technology with have likened it to being an instrument rather than a controller. I agree. Any thing I have seen for Kinect places it more as a controller. That's ok, but its not groundbreaking IMHO.

So what would be? I believe that controllers will truly break ground when they move from controller to gestural controller. What do I mean by that? Simple. We all use gestures. We first use them when we learn to speak. Our own body had a very complex synthesizer built right in. A voice box that acts as an oscillator and our throat, mouth, tongue and lips that all act as filters and our muscles which control these as modulators. But rather than thinking about position in space (the current paradigm be it kinect, Roland D-beam, theremn, ect), all of these are defined by morphology. Confused, ok. Morphology is just a way of describing how something changes over time. This is why I am interested, fascinated even memorized by developments studying the brain. The brain does not think in terms of coordinates in space. When someone for example extends their hand to use we don't start to think, ok, what is the coordinates of their hand. No! We see gesture. The position of the hand, the open hand, the extension of the hand to the other person, a smile, the direction of our eyes, all of these gestures get processed by our brains and our brains interpret them as a handshake.

Now here is the trick, moving beyond coordinates to gesture. Apple has done this a bit with there computers and I even have a Sony Vaio that interprets two quick finger pats on the mouse pad as a mouse click. Now consider a conductor and how, without using any physical device other than a conductors wand, is able to communicate to the orchestra musical information.

This is my criticism of Kinect as a musical controller. I don't see it moving beyond an XYZ controller (yawn) because to interpret gesture takes a very quick computer and some very sophisticated programmers. Do I think we will get there? Sure and Buchla already has done this with with the Buchla Lighening which is a rudimentary gestural controller albiet at a high price. My problem with Buchla is that they want someone to invest a lot of money in a product without even having a manual or sufficient demos to look at first. The demos that are out there really don't explain the gestural interpretation engine and frankly some of them look more like the motions of an escaped mental patient than a musician.

Anyway, I have bright hopes for the future but is Kinect the answer? I don't think so but as I said, I will at least partially suspend my judgement.