Researchers develop new smartwatch app that warns deaf users

Scientists, including one of Indian-origin, have developed 'Sound Watch', a smartwatch app for the deaf and hard-of-hearing people who want to be aware of nearby sounds.

Smartwatches offer people a private method for getting notifications about their surroundings -- such as a phone call, health alerts or an upcoming package delivery.

When the smartwatch picks up a sound the user is interested in -- examples include a siren, a microwave beeping or a bird chirping -- SoundWatch will identify it and send the user a friendly buzz along with information about the sound.

"This technology provides people with a way to experience sounds that require an action -- such as getting food from the microwave when it beeps," said study author Dhruv Jain from the University of Washington in the US.

"But these devices can also enhance people's experiences and help them feel more connected to the world," Jain added.

"I use the watch prototype to notice birds chirping and waterfall sounds when I am hiking. It makes me feel present in nature. My hope is that other deaf and hard-of-hearing people who are interested in sounds will also find SoundWatch helpful," he added.

The team started this project by designing a system for deaf and hard-of-hearing people who wanted to be able to know what was going on around their homes.

The first system, called HomeSound, uses Microsoft Surface tablets scattered throughout the home which act as a network of interconnected displays.

Each display provides a basic floor plan of the house and alerts a user to a sound and its source.

The displays also show the sound's waveforms, to help users identify the sound, and store a history of all the sounds a user might have missed when they were not home.

The researchers tested HomeSound in the Seattle-area homes of six deaf or hard-of-hearing participants for three weeks.

Participants were instructed to go about their lives as normal and complete weekly surveys. Based on feedback, a second prototype used machine learning to classify sounds in real-time.

The researchers created a dataset of over 31 hours of 19 common home-related sounds -- such as a dog bark or a cat meow, a baby crying and a door knock.

The researchers then pivoted to a smartwatch system, which allows users to get sound alerts wherever they are, even in places they might not have their phones, such as at the gym.

Because smartwatches have limited storage and processing abilities, the team needed a system that didn't eat the watch's battery and was also fast and accurate.

First, the researchers compared a compressed version of the HomeSound classifier against three other available sound classifiers.

Then, the researchers tested the SoundWatch app and found the app was useful for letting them know if there was something that they should pay attention to.

The study was presented on Wednesday at the ACM conference on computing and accessibility.