Seeing speech: New tech helps deaf individuals learn to speak using visual feedback

Deafness creates significant barriers to speech development as it prevents the brain from receiving essential auditory feedback needed for learning speech. While cochlear implants can be effective when implanted early in life, older children and adults often face limited options for oral speech development. Sign language and total communication methods are valuable, but they do not facilitate oral speech. Traditional speech therapies, which rely on tactile or visual cues, have had limited success, primarily because they lack real-time feedback. With these challenges in mind, there is an urgent need for innovative technologies that can empower the deaf to speak.

Published (DOI: 10.26599/JOTO.2025.9540003) on March 20 in the Journal of Otology, researchers from the Amrita Institute of Medical Sciences, CouthIT and SSN College of Engineering, and other institutions unveiled an innovative assistive technology. The study, funded by the Indian government, developed a smartphone app that transforms speech sounds into visual equivalents, allowing deaf users to practice and improve their pronunciation. The six-month trial with 72 participants showed promising improvements in speech development.

The core innovation of the study lies in its use of "Visual Equivalents" (VEs)—color-coded patterns that represent the spectral and temporal features of speech sounds. These VEs are displayed on the smartphone app to provide real-time feedback. The app captures the user’s vocal attempt, converts it into a VE, and compares it to a reference VE of the target sound, guiding adjustments through color indicators (green for a close match, red for poor alignment). By tapping into the brain's plasticity and the visual processing strengths of the deaf community, this technology effectively replaces auditory feedback with visual cues, enabling speech development where traditional methods fall short. Early trials showed promising results, with participants advancing from no vocalization to an average of 7 phonetic sounds, with some achieving up to 18 within six months. Its affordability and scalability, running on low-cost smartphones, make the technology particularly beneficial for resource-limited settings. Despite the positive results, challenges remain, such as optimizing the interface for long-term engagement, expanding the phonetic library for more comprehensive language applications, and validating long-term effectiveness across various age groups. Nonetheless, this study represents a significant leap toward accessible, technology-driven speech rehabilitation for the deaf.

Dr. Singh S, the study's lead author, commented, "This technology marks a new era in deaf rehabilitation. By using visual feedback, we are providing the deaf with the means to independently develop speech, something once considered impossible without auditory input. The next challenge is refining the system for long-term use and expanding it to support a wider range of languages."

This technology has the potential to revolutionize deaf education and rehabilitation, particularly for individuals who are ineligible for cochlear implants or those in underserved regions. Future applications could integrate with speech-to-text tools, creating a comprehensive communication platform. The approach might also benefit late cochlear implant recipients by offering a hybrid system combining both visual and auditory feedback. Further research will be necessary to refine the VE algorithm and assess its effectiveness across different languages and age groups. If successfully scaled, this innovation could significantly enhance the quality of life for millions of deaf individuals worldwide.

This research was supported by grants from Biotechnology Industry Research Assistance Council (BIRAC), Department of Biotechnology (DBT), Government of India.

About Journal of Otology

Journal of Otology is an open access, peer-reviewed journal that publishes research findings from disciplines related to both clinical and basic science aspects of auditory and vestibular system and diseases of the ear. This journal welcomes submissions describing original experimental research that may improve our understanding of the mechanisms underlying problems of basic or clinical significance and treatment of patients with disorders of the auditory and vestibular systems. In addition to original papers the journal also offers invited review articles on current topics written by leading experts in the field. The journal is of primary importance for all scientists and practitioners interested in audiology, otology and neurotology, auditory neurosciences and related disciplines. Journal of Otology welcomes contributions from scholars in all countries and regions across the world.