TIPS for Parents: Learning ASL Through Interactive Play and AI Technology

Can artificial intelligence-powered tools help enrich child development and learning?

That question is the crux of a series of research projects led by Zhen Bai, an assistant professor of computer science and the Biggar Family Fellow in Data Science at the Goergen Institute for Data Science. From tools to help parents of deaf and hard-of-hearing (DHH) children learn American Sign Language (ASL) to interactive games that demystify machine learning, Bai aims to help children benefit from AI and understand how it is impacting them.

Bai, an expert in human-computer interaction, believes that, despite all the concern and angst about AI, the technology has tremendous potential for good. She believes children are especially primed to benefit.

“Over the years, I’ve seen how kids get interested whenever we present technology like a conversational agent,” says Bai. “I feel like it would be a missed opportunity if we don’t prepare the next generation to know more about AI so they can feel empowered in using the technology and are informed about the ethical issues surrounding it.”

Minimizing language deprivation in DHH children

During one of Bai’s earliest experiences at the University of Rochester, she met a key collaborator who led her to a new avenue of research. At a new faculty orientation breakfast, she happened to sit next to Dr. Wyatte Hall, a Deaf researcher and assistant professor at the University of Rochester Medical Center’s Department of Public Health Services. The two bonded over a shared interest in childhood development and learning.

Hall explained some of the unique challenges DHH children face in cognitive and social development. More than 90 percent of DHH children are born to hearing parents, and often the very first deaf person that parents meet is their own baby. In early human development, there’s a neurocritical period of language acquisition—approximately the first five years of a child’s life—in which children need to acquire a first language foundation. Having parents who do not know a signed language, and the limits of technology such as the cochlear implant and hearing aids, increases the risk of DHH children experiencing negative developmental outcomes associated with language deprivation.

“I learned a lot from Dr. Hall about this concept of language deprivation and became fascinated with the idea of how technology could play a role to make life easier,” says Bai. “I wanted to explore how to help facilitate this very intimate bonding from day one between parents and their kids.”

Bai and Hall began collaborating on a project called the Tabletop Interactive Play System (TIPS) to help parents learn ASL in a natural setting. The system uses a camera and microphone to observe the parent and child interacting, and then uses a projector to present videos of relevant signs retrieved via artificial intelligence from multiple ASL libraries.

In addition to a tabletop version, Bai has been developing versions for tablets, smart watches, and smart glasses, together with her team of undergraduate and graduate students with backgrounds in computer science, data science and neuroscience. She has also collaborated with student fellows from the Rochester Bridges to the Doctorate program and other researchers from the Deaf community such as Dr. Athena Willis, a scholar in the Rochester Postdoctoral Partnership, from the Department of Neuroscience.

Rochester, reportedly home to the country’s largest population of DHH people per capita, is a uniquely rich setting for researching assistive technologies for the Deaf community. Hall says Bai’s willingness to learn from and collaborate with the Deaf community has helped improve the effectiveness of the tool.

“Often we’ve seen hearing people, hearing researchers become involved in Deaf-related things, they learn something interesting about deaf people and want to run with it for their own work. Even with the best of intentions, that can go awry very quickly if they are not collaborating with Deaf people and the community at all or in the right way,” says Hall. “My experience with Dr. Bai, though, she really started with a good foundation and kept collaborating with me in a very positive way, so it’s been a great partnership from the very beginning.”

Subscribe to the University of Rochester on YouTube: https://www.youtube.com/user/UniversityRochester?sub_confirmation=1