It turns out that a couple of people have demonstrated what we are attempting, to combine web cam head tracking and a 3D video display:
djine85's video is in French, and seems to work quite well.

From Danny's blog, http://danielbaggio.blogspot.com/, he seems to be getting pretty good performance.

Nonetheless, we will forge on!

A few weeks ago I stumbled upon this amazing video on youtube about Head Tracking for Desktop VR Displays using the WiiRemote:

How can you not say wow? This Johnny Lee is truly a genius! The guy is the definition of "out of the box" thinking. Who knew that a wii remote could serve such a purpose...and holy batman he's publishing his source code so that people can try it at home?!

As any other tech geek, I had to give this a try right away! I jumped off my seat, quickly blew the dust off my laptop, downloaded the source code and realized I had little problem: I don't own a Nintendo Wii...

Dishing out $250+tx for a console that I don't think I'll be using after this project didn't really appeal to me. There's gotta be another way to do this, yes there must be. So I thought, hey there's no reason why we wouldn't be able to achieve the same kind of effect using a cheap $20 webcam...or is there?

I phoned up my buddy Derek to do a little brainstorming and decided to start this blog. Here we'll keep you updated on our progress in creating a webcam based virtual viewpoint (or wii free desktop VR). Stay tuned...

In the meantime you can check out Johnny Lee's other amazing Wii projets on his project site.