Alexander Huth was on a Zoom call when he found out he could read people’s thoughts for the first time. He wasn’t completely surprised. An assistant professor of neuroscience and computer science at the University of Texas at Austin, Huth had been striving for this for years with his colleague and fellow neuroscientist Jerry Tang. When it finally happened, though, he told The Daily Beast that one thought crossed his mind, “Damn, that’s very impressive.”
Of course, the two weren’t capable of mind reading the way you might imagine like with stage magicians or carnival hucksters. Rather, they were able to develop an AI language decoder that could reconstruct entire thoughts from functional MRI (fMRI) data from three patients. In a paper published Monday in the journal Nature Neuroscience, Huth and his co-authors were able to decode words and short sentences using brain scans and the model.
“We're using fMRI data that we recorded while somebody's lying in an MRI scanner to try and decode the words that they’re hearing, or thinking, or even words related to something like a video that the person is watching from the brain activity that we record,” Huth said.
ADVERTISEMENT
The language decoder offers a non-invasive way of translating thoughts into text—a goal for many people working on brain-computer interfaces (BCI). These technologies offer hope for the millions of people living with paralysis caused by stroke, locked in syndrome, or injury. Unlike BCI ventures like Neuralink or the Stanford BCI lab, though, the findings from the UT Austin researchers are non-invasive, which means surgery is not necessary to implant a chip in a patient’s skull. There are, however, privacy issues that could snowball into something truly nefarious if left unchecked.
To create the language decoder, the team recorded fMRI data from three patients who each listened to 16 hours of podcasts. These podcasts were narrative focused, like The Moth Radio Hour, The Modern Love Podcast, and TED Radio Hour. The authors then used the data to train a model in order to map their brain activity with the words and sentences of the podcasts.
A very simplistic example: if you hear the sentence “I have a map,” specific types of brain activity occur that can be recorded. This data can then be used to train an encoder to understand that when those regions of your brain light up, you’re likely thinking of the sentence: “I have a map.”
“An encoding model will take the words that a person is hearing, like a transcript of the podcast that they're listening to, and use that to predict the brain responses,” Huth explained.
This is where the decoder comes in. Whereas the encoder takes sentences and connects them with fMRI data, the decoder model is the one that actually takes the fMRI data and translates it back into sentences. For this, the team actually utilized GPT-1 from OpenAI—the same folks behind ChatGPT—to create the model.
The authors were able to successfully reconstruct certain words and phrases from the initial set of recordings. Huth recalls that he was sitting in on a Zoom call with Tang at the time as they realized they were able to decode the fMRI data. After that initial set proved successful, they went back and conducted additional experiments to decode silent movies and even independent thoughts that the participants had.
For the latter, the subjects told five 1-minute stories while recorded by the fMRI. They then told the stories aloud to provide reference transcripts. The study’s authors were able to identify the story that the subjects were telling after decoding their brain scans. While the decoder wasn’t perfect and could only translate broader thoughts and ideas, it still matched the accuracy of the actual transcripts more closely than if things were left to pure chance. “When that just popped out and worked, it was a real ‘damn’ moment,” Huth said.
The implications of technology like this are frightening—and Huth is quick to acknowledge that. The authors write about the extensive privacy concerns that might arise when it comes to what essentially amounts to a mind-reading robot.
However, Huth emphasized the current technology is incredibly limited. The patient needs to be cooperative in order to properly decode someone’s thoughts. They can also easily disrupt it by silently counting numbers or thinking of random animals, among other things.
The encoder and decoder also don't work across all brains. It needs to be trained specifically to each individual person in order to work—which makes sense. After all, everyone’s minds are different, and so the models need to be different too.
However, Huth concedes that technology like this does open the doors part way to a potential future where it becomes sophisticated enough to create a sort of generalized brain decoder. In a press briefing on April 26, Huth and Tang said that it’s beholden on the policymakers and regulators to create effective guardrails for this technology before it becomes powerful enough to become a privacy crisis across society.
Of course, that’d be great… if there was any evidence that would ever happen. In the past few months alone, we’ve seen policymakers attempting to play catch up with the burgeoning world of generative AI and its potential to upend industries and proliferate fake news via deepfakes. While this is the latest example of the glacial pace of lawmakers to keep up with new technology, it isn’t the only one. We’ve seen it happen with social media in the 2000s, and the advent of the Internet before that.
That’s all to say that policymakers aren’t the best at anticipating the dangers of emerging technology—so there’s little reason to think it’d be the same with BCIs.
Still, the work that Huth and his colleagues are doing no doubt offers hope to millions who aren’t capable of speaking on their own. Of course, not everyone can lug around an enormous fMRI machine with them—but the authors are quick to say that isn’t the point. The decoder offers a proof of concept for potential BCI tech that doesn’t involve invasive brain surgery.
For now, Huth hopes to build off of the research in the future in order to apply the decoder to a more practical technology like functional near-infrared spectroscopy (fNIRS), which requires a much simpler set up of a skull cap with electrodes connected to it. The authors noted that fNIRS utilizes a similar process and system to fMRIs when it comes to the data gathered.
Huth also hopes to experiment with GPT-3 and GPT-4 in future versions of the decoder. Those models are orders of magnitude more powerful than GPT-1, so it could represent a quantum leap for the decoder once it happens.
“It’s a good feeling as a scientist,” Huth said. “I know we have something here.”
