[AI] For the deaf new software brings sign language to cell phone video

pradeep banakar pradeepsocialwork at gmail.com
Wed Sep 1 22:29:26 EDT 2010

For the deaf new software brings sign language to cell phone video

By Christina Hernandez | Aug 31, 2010 |

In an effort to improve communication for the deaf, engineers at the
University of Washington are developing software that transmits
American Sign Language over cellular networks.

I spoke last week with project leader Eve Riskin about how the
software works — and about its implications for the deaf community.

How did you go about creating this software?

We started working on the project about five years ago. Our plan was
to work on software to compress American Sign Language video. Our goal
was to do it on PCs or work stations. We figured that would be
challenging enough. But in the course of working on a project new
ideas pop up. Four years ago, I got a new cell phone that happened to
have a video camera on it. We had an ‘ah-ha’ moment that we could do
this on a cell phone. We picked the HTC TyTN II. It was the first
phone we found that was Windows Mobile and had the camera on the

How similar is your software to the iPhone4 Facetime feature?

That’s exactly the idea. It’s got snazzier software because they’re
Apple and we’re just a research group, but the idea is there. We’ve
had our program running for two and a half years now, so we’ve been
able to go around and give talks about our project. [We'd] talk to
kids in high school to get them interested in engineering and then
show them [our] phones. They were flabbergasted because no one had
ever seen it before.

Why is it important to be able to transmit sign language via video,
instead of just using text?
[For] many people who speak sign language, [it] is their first
language. You’re texting in a foreign language. Some people who are
deaf have rather limited English. Also, a lot of the emotion,
expression, nuances are conveyed through the face.

Talk about the field testing you’ve done of the software.

It was a limited field test. It was nine students and two [teaching
assistants] who are here in Seattle this summer. We’ll have a larger
field study in the winter where we’ll investigate more of the
technologies we’re working on. The field study we had here was more
about giving people the phones and seeing how they use them. In our
next field study, we’ll be able to test out our battery, extension
algorithms. We’ll learn what people think when they’re actually using
the technology.
Once the field testing is complete, how will this technology be
available to the public? Would it be through one phone only or through
a phone application?

We are interested in porting it to other devices. We’re very
interested in trying it on the Android phone. Some of what we’ve done
could mesh with Facetime if we had the person-power to do the extra
software packing. It could be an app for different phones. We just
have to find the funding to do that.

The software looks like Facetime, but does it also include specific
features geared toward the deaf audience?

Our user interface was designed with input from a young woman who is a
native [sign language] speaker. It’s based on video We do some signal
processing to make the quality better in the hands and face. Our goal
wasn’t to have beautiful video like Facetime. Our goal was to have
intelligible sign language, so you could have a conversation.
People who are deaf take turns just like people who are hearing. If
you are signing and I’m not signing, then it’s going to lower my
quality. It saves battery lifetime because the cell phone is doing
less work.

We don’t send audio, so that saves bandwidth and time. You don’t need
voice for our target audience.

We have what we call a contact list. It’s just like [the green dots on
Skype and Google Chat that indicate when a person is available to
talk]. It makes it easier than just calling somebody and not knowing
they’re there.

Click on the links below to watch videos of the software in action:
•    UW working on app to help deaf, hard of hearing
•    Hands-On: MobileASL- Video conferencing for the deaf community
•    http://www.youtube.com/watch?v=J7640Pe6shc&feature=search

Deaf, hard-of-hearing UW students test sign language cell phone
•    Posted on August 18, 2010 at 6:08 PM

The advent of texting finally allowed the deaf community to join the
cell phone world, but only in a limited way.
"(With) texting you can not really convey emotion as much as seeing
the person," said Peter Michor, who has been hearing-impaired since

That's why UW researchers are working on a special video app, capable
of transmitting American Sign Language over US cellular networks.
Unlike the more expensive 4G video phones, they wanted this app to be
affordable and accessible to anyone with any kind of video-enabled
cell phone. That also means some trade-offs.

Jessica Tran is a UW doctoral student in electrical engineering who
has been working on the prototype.
"Because we're sending out 30 kilobits per second, the video quality
is lower, but we also have different algorithms that Face and hands
are what matter most to these users. The prototype phone also frees up
both hands for signing.

To maximize video elements, researchers decided to eliminate one
feature that won't be missed - sound.

Participants in a UW summer school computer program for deaf and
hard-of-hearing students are the first to field test the new
technology. They liked what they saw.
"From where it is now, it's a very good prototype. In the end, I do
believe that it will be a very good thing to have in the deaf and hard
of hearing community so can't wait for it," said Michor.
Fellow student Adam Zaelit summed up his reaction in one word:

Researchers hope to have the sign language app on the market in a year or two.

More information about the AccessIndia mailing list