[AI] Say It, and Program Signs It

vishnu ramchandani vishnuhappy at yahoo.com
Sun Sep 30 23:48:53 EDT 2007

Say It, and Program Signs It

Tracy Staedter, 

Converting speech into sign language is normally a job
for a human translator. But now an animated character
 is up to the task.

The "Say It Sign It" system translates spoken words
into sign language and then engages an avatar to
communicate using gestures. The onscreen translator
could work as a pop-up on a television, personal
computer, mobile phone or auditorium screen, giving
the hearing impaired wider access to television, radio
and education.

"It was inspired by a vision that a deaf colleague on
my team had of seeing, not words being brought up on
his phone, but an animated character signing
in British Sign Language," said Andy Stanford-Clark,
master inventor at IBM Hursley in the U.K.

The colleague was Ben Fletcher who, at the time, was
an intern for 
Extreme Blue
 program. Fletcher is deaf and 
British Sign Language
 is his first language.

"He was able to give feedback and keep us honest,"
said Stanford-Clark. After returning to school and
completing his degree, Fletcher became a permanent
member of the team, which also includes researchers
from the Universities of Oxford, Durham, Glasgow and
East Anglia.

The program has about three main components. First a 
voice recognition system
 takes words spoken into a microphone and converts
those into a stream of text. The text is then sent
through a translation program that looks for patterns
in the words and applies different rules based on
those patterns.     

For example, if someone said "My name is Andy," a rule
would transform that to the phrase in British Sign
Language, which is "Name me Andy."

Once the word order is worked out, the program refers
to a dictionary of gestures supplied by the 
Royal National Institute for Deaf People
. These are used to direct the computer-animated

Tying it all together is a piece of software, or
middleware, that allows different applications and
computers to link together without being in the same
building or even the same country.

Although the system is still in the prototype phase,
the team envisions a couple of different scenarios for
its use. The program could be hosted by a service
Web site. A speaker would sign in on one end, while
the recipient would sign in on the other end. As the
person speaks, the words would be converted on
a central server and then animated on the end-user's

The system could also work from a converter box atop a
television, or it could be sold individually as a
program that speakers and receivers would install
on their own computers.

"The model is quite different from the things that I
have seen in the past," said Guido Gybels, director of
new technologies at the Royal National Institute
for Deaf People in London. "It's built in a way that,
in principle, can be applied to other foreign

But, said Gybels, "There are significant technical and
scientific challenges, in addition to cultural ones,
that need to be addressed before we can see
this as an off-the-shelf product."

For starters, the speech recognition technology is
still a long way off from capturing free-flowing
unconstrained human dialogue.

Furthermore, while the English language (as well as
other languages) has been studied in great depth, sign
language has not, said Gybels.

If scientists are going to accurately convert speech
into sign language they will need that understanding.

And lastly, he points out the notion of using virtual
humans and animated characters doesn't always fly with
people you're asking to use these kind of services.
Not all humans are going to accept a replacement,
especially one that may not have the natural, subtle
motions — the finger, lip and movements — of the
real thing.

Currently Say It Sign It translates spoken English
into British Sign Language but Stanford-Clark said
that it would not be a big leap to translate between
other spoken languages and forms of sign language.

      Now you can chat without downloading messenger. Go to http://in.messenger.yahoo.com/webmessengerpromo.php

More information about the AccessIndia mailing list