Trying to make the pen as mighty as the keyboard

Published: November 11, 2004, 7:13 AM PST

Portable computers that let you give commands or enter text using an electronic pen on the screen could usher in a new era of information sharing.

But pen computers like those using Microsoft's Tablet PC software have had only lukewarm success, and writing notes on a palmtop requires learning a kind of shorthand.

Pointing at and clicking on on-screen icons using a digital pen can be cumbersome and counterintuitive, and writing on a screen in longhand is much slower than typing. Factor in the huge amount of variation in handwriting and shorthand and it's little wonder computers have had a hard time recognizing them.

"Lots of people have trouble recognizing their own handwriting, so it's not surprising a computer can't do it," said Brad Myers, a professor in the human-computer interaction institute at Carnegie Mellon's school of computer science.

Now researchers are devising new ways to interact with tablet PCs and other pen-based computers. The machines use a special version of Microsoft Windows and a digitizer under the screen that samples pen strokes to let users write with digital "ink," peck out messages with an on-screen keyboard and navigate their desktop and folders with a stylus instead of a mouse.

The research could make it easier to enter text quickly and accurately, and it could encourage collaboration by introducing gesture commands that let friends and co-workers share photos and files with a flick of a pen.



	For the latest breaking news, visit NYTimes.com Sign up to receive top headlines Get Dealbook, a daily corporate finance email briefing Search the jobs listings at NYTimes.com Search NYTimes.com:

"We all realize that the desktop computing platform has matured, and we need to go beyond that, creating an environment for when you're on the move," said Shumin Zhai, a research staff member at IBM's Almaden Research Center in California. "But point-and-click interfaces have meant that tablets and PDAs suffered, because their screens are smaller. They do have an advantage--a stylus--but it's not being used to its full capability."

Last month Zhai and his team released software called Shark, or shorthand-aided rapid keyboarding, on IBM's AlphaWorks Web site, where the company previews emerging technologies from its labs. The software replaces the standard on-screen keyboard that a tablet PC usually displays with a stylus-friendly keyboard that places common letter combinations near one another in a hexagonal grid. Users write words by tracing a stylus across the virtual keys, forming shapes that Zhai calls sokgraphs.

Each time a user traces a word, the computer shows the shape on the screen. After enough practice, "you learn so much of the shape that it comes to mind and you can gesture it based on memory," he said. Meanwhile, the keyboard remains on the screen as a mnemonic device.

A tolerance for sloppiness is built in. Algorithms recognize that some letter combinations are more likely to form words than others, so users can deviate from the exact shape of each symbol. Shark includes a lexicon of about 8,000 words, and it can analyze e-mail messages and text files to learn new ones. According to user tests, people can learn about 15 sokgraph words in each 45-minute training session. Learning 100 patterns covers about 40 percent of the words most people write, Zhai said.

Ben Bederson, director of the human-computer interaction lab at the University of Maryland, said the Shark project "is an example of where the interface brings you from novice to expert with the same interaction." He added, "You think the word 'the' and execute the gesture without a huge amount of visual feedback and concentration."

Keeping people's attention on their work and not on their computers is also a goal of Ken Hinckley, a research scientist at Microsoft. Hinckley has developed software technology called Stitching that lets tablet PC and palmtop users share files by flicking their pen toward the screen of the person they want to share with.

"If I'm in a meeting and I want to send you a file, the conversation ends there, and I have to ask you for your e-mail address," he said. "Some devices have infrared ports, but then you have the awkward dance of lining them up and putting them in the right mode. With this approach, you can keep your attention in the real world and with the person you're working with."

Prototype software called StitchMaster lets users select photos on their tablet PC by tapping or circling groups of them, then wirelessly pass those files to another tablet user sitting within arm's reach by making a gesture that spans both screens. Stitching works by measuring the timing, the angle and the exit location of pen gestures that leave the edge of a machine's screen. Since tablet PCs can broadcast their Internet address using the 802.11 wireless networking standard, if a second machine within two and a half feet picks up the continuation of a pen stroke across its edge, the software can tell which machine sent it and infer a handoff.

Microsoft's usability tests have shown that a dozen participants were able to connect their tablets this way on their first or second try. But some were uncomfortable reaching across another person's computer, or even placing their machines too close together. Hinckley's latest work, which he calls collaborative stitching, aims to address that by letting a user pass a document to several people sitting beyond arm's length by dragging the file to the top of the user's screen. Then those people in the room who want the file can pull it down from the top of their screen.

According to Andy van Dam, a computer science professor and vice president for research at Brown University, who also serves on Microsoft's technical research advisory board, tablet PCs and other pen-driven computers won't take off until pen gestures provide new ways of interacting with the machines instead of simply substituting for a mouse. Pen computers could find markets in education, architecture, graphic design and user-interface design, he said. "For these people, a pencil and a piece of paper are more natural almost than a computer keyboard on a desktop."

But pen software needs more testing to find out what users really want, he said.

"For a relatively pure gesture-driver user interface, it's all research," he said. "None of these have had a field trial with a thousand users, let alone ten thousand. Gestures are never going to be for everyone, so I don't want to impose them. But we can make them an option."

Have an opinion on this story? Share it with other News.com readers.