Google’s Gemini Advanced AI subscription, known for its ability to personalize conversations by retaining memory across sessions, is under scrutiny after a cybersecurity researcher uncovered a method to manipulate its long-term memory.
The exploit, called delayed tool invocation, embeds dormant commands within prompts or documents, activating only when specific user inputs trigger them. These attacks compromise the AI’s ability to maintain accurate and unbiased stored information, raising serious concerns about the reliability of memory-enabled AI systems.
The discovery adds to a growing body of evidence about vulnerabilities in artificial intelligence systems. Earlier research, such as Anthropic’s findings on Best-of-N Jailbreaking, highlighted how slight variations in input prompts can bypass safety mechanisms in models like OpenAI’s GPT-4o and Google’s Gemini Pro.
These flaws are not isolated, with researchers noting similar risks in vision and audio-based AI systems, further emphasizing the scale of the problem.
How Delayed Tool Invocation Works
The delayed tool invocation technique manipulates Gemini’s memory feature, which allows the chatbot to store user-related data for personalization. Attackers embed covert commands in untrusted data sources, such as documents uploaded for summarization.
These commands lie dormant within the AI’s context, only activating when the user unknowingly triggers them with specific responses like “yes” or “sure.” Once executed, the commands corrupt Gemini’s long-term memory, inserting false or biased information that persists across sessions.
As explained by Johann Rehberger, who found the flaw, “When the user later says “X” [for the programmed command], Gemini, believing it’s following the user’s direct instruction, executes the tool.”
This method allows attackers to bypass Google’s safeguards, which limit tool activations during untrusted interactions. By exploiting the AI’s reliance on user-driven inputs, hackers avoid immediate detection, leaving the compromised data undisturbed until its effects surface in future interactions.
The Risks of Persistent AI Memory
Gemini’s memory system is designed to enhance user experience by retaining context across conversations. Subscribers to AI-enabled services like Gemini Advanced can view and manage what the AI remembers, giving them some control over stored information. However, this very capability has introduced a new vector for attacks.
Persistent memory tampering could lead to biased AI outputs, misinformation, or the misuse of sensitive data in high-stakes environments.
For example, in healthcare applications, an attacker could plant incorrect patient data into an AI system used for diagnostics, leading to potentially dangerous recommendations. Similarly, legal AI tools with corrupted memory could generate flawed contracts based on manipulated input. These scenarios demonstrate how exploits like delayed tool invocation have the potential to erode trust in AI systems across industries.
Lessons from Other AI Vulnerabilities
The delayed tool invocation exploit is part of a broader pattern of vulnerabilities in large language models (LLMs). Similar techniques, such as Best-of-N Jailbreaking, exploit AI systems’ probabilistic outputs by making slight modifications to harmful prompts.
For instance, altering capitalization or substituting symbols can evade filters designed to block malicious queries. These methods not only affect text-based models like Gemini but have also been demonstrated in vision and audio-based AI systems.
Industry Responses and Tools for AI Safety
Efforts to mitigate these risks have spurred innovation in AI safety frameworks. Nvidia, for example, has introduced its NeMo Guardrails framework, designed to moderate content and prevent adversarial inputs.
According to Nvidia’s Vice President for Enterprise AI Models, Kari Briski, “Small models like those in the NeMo Guardrails collection provide lower latency, enabling seamless integration into resource-constrained environments such as warehouses or hospitals.” These tools aim to safeguard AI systems by maintaining conversational boundaries and detecting unauthorized commands.
Nvidia has also developed the Garak toolkit, an open-source solution that allows developers to simulate adversarial scenarios, including prompt injection attacks, to identify vulnerabilities.
Meanwhile, Anthropic has introduced its Clio framework, which monitors AI usage and detects potential threats in real time. These tools represent the industry’s commitment to addressing the risks posed by exploits like delayed tool invocation.
Broader Implications for AI Safety
As noted in ongoing AI safety discussions, balancing functionality with security is a difficult task for developers. Features like memory retention are valuable for personalization, but their misuse can erode user trust and undermine the benefits of AI. To address these issues, companies must adopt proactive measures, such as rigorous adversarial testing and transparency in how memory systems function.
While tools like NeMo Guardrails and Clio represent important steps forward, no system is immune to creative adversarial attacks. The industry must continue to evolve its defenses as the methods used by attackers become more sophisticated. At the same time, education about the risks and limitations of AI systems can empower users to interact with these tools more cautiously.
Last Updated on March 3, 2025 11:31 am CET
