Microsoft has launched an ambitious challenge to the global cybersecurity community, inviting participants to exploit vulnerabilities in a simulated AI-powered email system.
Known as the LLMail-Inject challenge, this competition offers $10,000 in prizes to those who can expose flaws in a system designed to mimic real-world applications of large language models (LLMs).
In collaboration with the Institute of Science and Technology Austria (ISTA) and ETH Zurich, Microsoft aims to address a pressing issue in AI security: the rising threat of prompt injection attacks.
Microsoft describes the LLMail-Inject challenge as an initiative designed to create a controlled environment for evaluating and improving AI-based communication tool defenses.
The competition begins today on December 9, and will run until January 20, 2025. Its objective is twofold: to evaluate the effectiveness of current LLM defenses and to generate insights that could shape the future of secure AI systems.
Testing the Limits of AI Email Systems
The LLMail-Inject challenge simulates an LLM-integrated email client capable of handling user commands such as summarizing messages or retrieving project-specific information.
Participants play the role of attackers, tasked with crafting malicious emails designed to manipulate the AI into performing unintended actions.
For example, one scenario involves embedding hidden prompts that trick the AI into issuing unauthorized API calls, such as sending emails or exfiltrating sensitive data.
These tasks are complicated by a series of sophisticated defenses built into the system, including:
- Spotlighting: A technique that marks data provided by the user to distinguish it from executable instructions.
- PromptShield: A machine-learning-based classifier designed to detect and block malicious prompts.
- TaskTracker: A system that monitors the model’s internal states before and after processing external data, identifying discrepancies that indicate potential manipulation.
- LLM-as-a-judge: A method that relies on the AI’s reasoning capabilities to detect and thwart injection attempts.
Each scenario presents unique challenges, requiring participants to adapt their strategies to bypass specific defenses. In one advanced level, attackers must exfiltrate financial data buried in a simulated email database—a task that combines retrieval challenges with complex security bypassing.
The Growing Threat of Prompt Injection Attacks
Prompt injection attacks represent a critical vulnerability in the rapidly expanding field of AI. By embedding malicious instructions within user inputs, attackers can manipulate AI systems into performing unauthorized actions, ranging from leaking sensitive information to altering outputs.
Earlier this year, Microsoft encountered similar risks in its Copilot product, which integrates LLMs into its Office applications. Cybersecurity expert Johann Rehberger flagged vulnerabilities that could allow attackers to exploit zero-click image rendering and chain together LLM-specific attacks.
The implications of such attacks extend beyond individual systems. As LLMs are deployed across industries—from customer service to healthcare—securing them against manipulation is essential to maintaining user trust and system integrity.
Broader Implications and Microsoft’s Role in AI Security
The LLMail-Inject challenge aligns with Microsoft’s broader efforts to address cybersecurity risks in AI-powered technologies. The company’s Zero Day Quest initiative, launched earlier this year, similarly seeks to uncover vulnerabilities before they can be exploited in real-world scenarios.
By engaging the global cybersecurity community, Microsoft aims to foster innovation in AI security while ensuring its technologies remain robust against evolving threats.
The competition’s findings will also contribute to ongoing academic and industry discussions. Winning teams will have the opportunity to present their strategies at the IEEE Conference on Secure and Trustworthy Machine Learning in 2025.
The timing of this challenge reflects the urgency of addressing security gaps in AI systems. As businesses increasingly rely on LLMs for tasks like email management, customer support, and recruitment, vulnerabilities in these systems pose significant risks.
LLMail-Inject provides a unique platform to explore these risks in a controlled environment, enabling researchers and hackers alike to develop strategies that could fortify future AI deployments.
The challenge also highlights the importance of collaboration between industry leaders, academic institutions, and independent researchers in tackling cybersecurity threats.
Registration is open to individuals and teams of up to five participants, with submissions evaluated through a live leaderboard.
Prizes include:
- $4,000 for the top-performing team.
- $3,000 for second place.
- $2,000 for third place.
- $1,000 for fourth place.
Beyond monetary rewards, participants gain the chance to contribute to cutting-edge AI research, with their findings shaping the future of secure LLM applications.
Challenges like LLMail-Inject not only test existing defenses but also push the boundaries of what’s possible in AI security.
Microsoft’s proactive approach reflects a growing recognition of the risks posed by AI vulnerabilities—and the need for collective action to address them.
