Click, Drag With Body Language
One of these days, when you walk into a room your computer will probably know you’re there because it can smell you.
But for now, if you’re on the cutting edge of research into the human-computer interface, the best you can hope for is a computer that will recognize your voice, know what you are looking at on the monitor, and allow you to reach out and grab it.
Scientists at Rutgers University have created a sophisticated system that does all of that as part of an effort to build a better mouse, so to speak.
“The motivation was to try to interface with a computer the way people interface with each other,” said Edward J. Devinney, senior associate director of Rutgers’ Center for Advanced Information Processing.
Project STIMULATE (Speech, Text, Image, and Multimedia Advanced Technology Effort) is the brainchild of James Flanagan, director of the center. It is funded by a three-year, $780,000 grant from the National Science Foundation, and builds on earlier work funded by the Pentagon’s Defense Advanced Research Projects Agency (DARPA).
It doesn’t exactly read your body language, but it comes pretty close. We don’t move a cursor when we want to communicate with other humans. We talk, we make eye contact, and sometimes we even reach out and touch them, and that’s the way we should be able to work with our computers, Devinney said.
The immediate goal is to improve the computer interface for people with high-stakes jobs such as fighter pilots and brain surgeons. But somewhere down the pike, there may be something in this for the rest of us.
A fighter pilot has to react so quickly it would be far better if the pilot could select a target by simply looking at it instead of physically aiming a device.
And future surgeons participating in computer-aided procedures from distant locations might find it easier and more natural to manipulate a glove than to move a mouse.
So the wizards at Rutgers decided to blend three streams of information--sight, sound and touch--into a single computer system that is, at this stage of development, as close to another human as they can get. And they have done it without resorting to the awkward helmets and other paraphernalia that make so many virtual reality devices uncomfortable and unwieldy.
The user sits at a workstation and talks to the computer, which monitors his or her eyeballs to see where the user is looking. A thin glove monitors the movement of the user’s fingers, allowing objects on the monitor to be moved by a simple flex of the digits.
“For sound, we have a speech recognizer and a speech synthesizer, and for sight, we are using a gaze tracker,” Devinney said.
The gaze tracker, made by Iscan Inc. of Burlington, Mass., is not much bigger than a Web cam, and it sits on the desk near the monitor. It sends out two beams of infrared light similar to a remote control unit for a television set. One finds the center of the pupil in the user’s eye, and the other bounces back from the curvature of the eye closest to the tracker. The angle between those two beams tells the computer where the user is looking, taking into account the precise distance from the screen to the user, which is measured by an ultrasound ranging device.
“Another cursor, a little circle on the screen, represents your gaze, so wherever you are looking on the screen, there is this circle,” Devinney said. “So you could look at an icon on the screen, and using the speech recognizer, you could say ‘identify.’ ”
The machine answers with a verbal identification of the icon, if everything works right. Of course, there are still a few bugs to work out. Eye glasses can confuse the system, and sunglasses drive it wild, but, Devinney said, it works most of the time.
For touch, the system uses a nifty “force-feedback” glove developed by Grigore C. Burdea, an associate professor at the center. The glove, patented by Rutgers, weighs less than 3 ounces and reads gestures by detecting fingertip positions relative to the palm.
The glove allows the user to select objects on the screen by pointing at them. You can reach out in the space in front of the monitor and “grab” an on-screen object and move it someplace else, taking drag-and-drop to a new level. Yet another cursor, one that looks like a tiny glove, follows this action.
In human-to-human interactions, not all of our activities are equal in importance. It means one thing if you look at someone, but another if you reach out and touch them.
So the researchers have programmed their computer to give different weights to different activities. The glove, for example, overrides the gaze in the computer’s value judgment system.
It may be a few years before the entire system is available to the public, if it ever is, but some of its parts are already in demand. The glove, for example, is expected to be a valuable tool in hand rehabilitation after a stroke or surgery. It can detect finger movements 20 times a second, Devinney said, measuring recovery and weaknesses over a long period.
And as a tactile instrument that can convey the sensation of resistance, the glove could have many applications. “In a virtual reality world, you can actually reach out and grab an object and squeeze it to find out if it’s firm or soft,” Devinney said.
Are you listening, Hollywood?
* Lee Dye can be reached via e-mail at leedye@gci.net
