Thesis Blog

http://www.bradhayes.info/thesis.pdf

As May draws nearer, it is increasingly important to solidify (and implement) the final goals of the project. With this in mind, the following is the final list of demos that will be implemented for the touchscreen:

• Conventional Usage Software
  • Mouse emulation, Gesture support to facilitate standard computer usage
• Physical Object Interaction Software :: Entertainment
  • Introducing real objects into virtual gameplay, as implemented through games utilizing nerf guns for interaction with virtual worlds.
• Physical Object Interaction Software :: Practical
  • Moving the “quick launch” or “dock” bar into the physical world, representing programs as blocks of wood. When touched onto the screen, they will launch their associated program.
• Infinite Desktop Model
  • Creating an infinitely high resolution desktop with persistent applications. This will be facilitated by the use of touch navigation, making it a practical and extremely useful extension of our current window manager standard.

More detailed write-ups for each of these programs will follow.

After spending spring break on campus working on the thesis, some notable results have been achieved:

Coating the screen with polyurethane spray enhanced the touch sensitivity ten-fold. It now takes much less pressure to use the device, and maintains adequate brightness during motion.

While interviewing at the MIT Media Lab, Dr. Lippman brought up an interesting point regarding a weakness of touch technology. When using the mouse, or other abstracted pointing device, the user is afforded a preview of where an action (eg: click) is going to take place before actually committing. I had inadvertently enabled this functionality when coating the acrylic with the polyurethane, as diffused light will reflect back from any object within an inch of the screen. This allows for a ‘preview’ effect, such as giving the user a magnified view of the area about to be interacted with.

I am nearing completion of implementing my Gesture framework, which is XML driven. Ideally, someone knowing very little about programming with a little XML knowledge should be able to put together gestures to accomplish simple tasks.

Automated Screen Calibration via Homography:

This has been implemented, though it is not 100% automated. As of this post the system presents the user with 5 contact points to touch, computing the homography from that information. One idea to consider regarding complete automation is placing IR markers on the frame just outside the screen and using the readings from that to obtain the “true” points.

Implement Touch Tracking:

This has been accomplished, and proved to be more difficult than initially anticipated. To do this quickly, I had originally created a greedy algorithm to find optimal matchings between frames at (t) and (t-1). This algorithm had trouble handling multiple points with similar paths within a certain distance from each other, as it would frequently confuse the two (velocity information was not robust enough to distinguish them). Upon realizing this, my thesis adviser and I put together a better algorithm that has no trouble with these cases (utilizing dynamic programming to perform a search on a filtered list of potential ‘best’ matches).

Due to the amount of Youtube feedback that I have received, I will (eventually) publish all of the algorithms used, along with explanations for how/why they work on the Projects page.

(New Window) Obligatory “Fingerpaint via Multitouch Surface” Demo

My thesis’ role in my (CS335) Principles of Multimedia Systems final project:

Fast Object Tracking

“Digital Lab Hockey”

So, it’s been about a month since the last post and a lot has happened over the semester break. The touch detection algorithm is fast enough to reliably detect contact points and report on shape information. This shape data is not being utilized at this point, but can be used in the future to perform object detection - enabling physical interaction with on-screen objects.

Design considerations regarding the use of the Nintendo Wii Remote’s infrared sensor (the viability of which has gained significant internet attention) have been all but scrapped, as the limitation of 4 single-point readings would render exploration into shape-recognition based systems impossible.

Next Steps:

• Automated Screen Calibration via Homography
• Implement Touch Tracking (Kalman Filter)
• Establish Grouping/Gesture Rules
• Implement Grammar for Defining Gestures and Associated Actions
• Smart Corners for Application Windows