While AI chatbots like ChatGPT and Gemini may impress with their ability to read and write, they’re not perfect. Along with their habit to hallucinate and their pernicious issues with bias, the technology also has a pretty basic problem holding it back: memory.
Anyone who has played around with ChatGPT for more than a few hours knows that, after a while, the bot begins acting strangely—forgetting earlier parts of the conversation and even crashing entirely on occasion. This creates a big hurdle for users who rely on the bots for work like coders, or for creative uses like writing or even playing Dungeons and Dragons.
Luckily, those issues might soon be nothing but bad memories. OpenAI announced in a blog post on Tuesday that it’s testing out a new feature that allows ChatGPT to remember specific details to “make future chats more helpful.” Users can “tell it to remember something, ask it what it remembers, and tell it to forget” via a chat conversation or the chatbot’s settings.
ADVERTISEMENT
“Remembering things you discuss across all chats saves you from having to repeat information and makes future conversations more helpful,” OpenAI wrote.
If you want ChatGPT to remember specific details, you’ll now be able to set parameters that the bot will remember in future conversations. For example, users can tell the bot that they prefer meeting notes to have headlines and bullet point summaries; or that they live in Chicago so it will give them restaurant recommendations closer to home; or that they typically code using Python.
OpenAI said that the feature is going to be available to “a small portion of ChatGPT free and Plus users” as part of an early testing period. Future plans for a wide-scale launch will follow “soon.”
The company isn’t the only one interested in boosting their bot’s memory either. On Tuesday, a team of AI researchers at MIT, Carnegie Mellon University, Meta, and NVIDIA released a pre-print detailing a new tweak that could allow most large language models (LLMs) like ChatGPT to retain memory and boost performance.
The process is called StreamingLLM and it allows for chatbots to perform optimally even after a conversation goes on for more than 4 million words. “Now, with this method, we can persistently deploy these large language models,” lead author Guangxuan Xiao, an AI researcher at MIT, said in a statement. “By making a chatbot that we can always chat with, and that can always respond to us based on our recent conversations, we could use these chatbots in some new applications.”
The tweak itself involves a change to the conversation memory of the LLM called a key-value cache. In some chatbot systems, when the cache holds too much memory, it gets rid of the oldest data in order to make room. In doing so, this destabilizes the system and can lead to performance issues.
It’s like when a friend is telling you a long story. You tend to forget the things that were told earlier because they were told further in the past. To combat this, the researchers used a framework called an “attention sink,” that acts essentially like a bookmark for the LLM to recall the earlier key-value cache. That way it stays within its memory.
This allows StreamingLLM to outperform models without the tweak. For example, if a chat contains 256 tokens (or words), it might take 63 milliseconds for a typical LLM to decode a new token. However, StreamingLLM takes just 31 milliseconds. The model is even faster than normal models the more tokens there are.
“The innovative approach of StreamingLLM, centered around the attention sink mechanism, ensures stable memory usage and performance, even when processing texts up to 4 million tokens in length,” Yang You, a computer scientist at the National University of Singapore who wasn’t involved in the study, said in a statement. “This capability is not just impressive; it's transformative, enabling StreamingLLM to be applied across a wide array of AI applications.”
So your chatbots are about to get a whole lot smarter and capable of remembering more. Soon, this will allow it to remember all manner of details about your life and work, recalling past conversations that it had with you better than actual people do.
It’s scary. However, it’s worth noting that OpenAI allows users to opt out of the memory feature, which can help alleviate at least some worries for those who are nervous about privacy and security (though, if that’s the case, you really shouldn’t be using chatbots in the first place).
Until then, it’s still a good idea to be careful with what you tell ChatGPT. After all, there’s no telling what it might remember.
