The technology that allows you to transcribe your work meetings might help people with paralysis speak again.
Researchers at UC Berkeley and UC San Francisco used generative AI to reduce the delay between when a person with severe paralysis attempts to speak and when the computer device plays the sound. Their work helped a woman named Ann, who suffered a brainstem stroke in 2005 at age 30, to communicate in close to real time. Ann spoke with a voice that sounded like her own because the model was trained on recordings of her from before her stroke.
The deployment of gen AI in a few different ways allowed researchers to make improvements in neuroprosthesis that might have taken far longer, said Cheol Jun Cho, a UC Berkeley Ph.D. student in electrical engineering and computer sciences and co-lead author of the study, which appeared in March in Nature Neuroscience.
It’s one example of how generative AI tools — using the same underlying technology that powers chatbots like OpenAI’s ChatGPT and Anthropic’s Claude or transcriptions in Google Meet — are helping medical and scientific researchers solve problems that might have taken much longer to solve, Cho told me. AI experts and backers have pointed to the technology’s use in medicine as an area with huge upside, whether in devising novel drugs or providing better testing and diagnosis.
«AI is accelerating the progress,» Cho said. «Sometimes we had imagined the timeline would be a decade or two. Now that pace is like three years.»
The technology that has helped Ann is a proof of concept, Cho said, but it shows a path toward tools that could be more plug and play in the future.
Speeding up speech
The problem with existing neuroprostheses is latency. There’s a time lag between when the person begins attempting to speak and when a sentence is actually generated and heard. Cho said the previous technology meant Ann had to wait until one sentence was finished before starting the next.
«The major breakthrough here is that she doesn’t need to wait until she finishes the sentence,» he said. «Now we can actually stream the decoding procedure whenever she intends to speak.»
The prosthesis includes a thing array of electrodes implanted on her brain’s surface and connected via a cable to a bank of computers. It decodes the control signals Ann’s brain sends to the muscles that control speech. After Ann has chosen the words she intends to say, an AI reads those signals from the motor cortex and gives them life.
To train the model, the team had Ann attempt to speak sentences shown on a prompt on a screen. They then used data on that activity to map the signals in the motor cortex, using gen AI to fill in the gaps.
Cho said the team hopes the breakthrough leads to devices that are scalable and more accessible.
«We are still in the ongoing efforts to make it more accurate and lower latency,» he said. «We’re trying to build something that can be more plug and play.»
Using AI to go from thought to speech
Cho said the team used gen AI in a few different ways. One was to replicate Ann’s pre-injury voice. They used recordings from before her injury to train a model that could produce the sound of her voice.
«She was very excited when she first heard her own voice again,» Cho said.
The big change was in the real-time transcription. Cho compared it to the tools that transcribe presentations or meetings as they happen.
The work built on a 2023 study that used AI tools to help Ann communicate. That work still had a significant delay between when Ann attempted to speak and when the words were produced. This research cut that delay significantly, and Ann told the team it felt more natural.
«She reported she felt embodied, that it was her own speech,» Cho said.
