Mark Russinovich, Chief Technology Officer of Microsoft Azure, has highlighted the growing security concerns associated with generative AI. Speaking at the Microsoft Build 2024 conference in Seattle, Russinovich underscored the diverse array of threats that Chief Information Security Officers (CISOs) and developers must navigate as they integrate generative AI technologies. He emphasized the necessity for a multidisciplinary approach to AI security, which includes scrutinizing threats from various angles such as AI applications, underlying model code, API requests, training data, and potential backdoors.
Data Poisoning and Model Misclassification
One of the primary concerns Russinovich addressed is data poisoning. In these attacks, adversaries manipulate the datasets used to train AI or machine learning models, leading to corrupted outputs. He illustrated this with an example where digital noise added to an image caused the AI to misclassify a panda as a monkey. This type of attack can be particularly insidious because even a minor alteration, such as a backdoor insertion, can significantly impact the model's performance.
Russinovich also discussed the issue of backdoors within AI models. While often seen as a vulnerability, backdoors can also serve to verify a model's authenticity and integrity. He explained that backdoors could be used to fingerprint a model, enabling software to check for its authenticity. This involves adding unique questions to the code that are unlikely to be asked by real users, thereby ensuring the model's integrity.
Prompt Injection Techniques
Another significant threat Russinovich highlighted is prompt injection techniques. These involve inserting hidden texts into dialogues, which can lead to data leaks or influence AI behavior beyond its intended operations. We have seen how OpenAI's GPT-4 V is vulnerable to this type of attack. He demonstrated how a piece of hidden text injected into a dialogue could result in leaking private data, akin to cross-site scripting exploits in web security. This necessitates isolating users, sessions, and content from one another to prevent such attacks.
At the forefront of Microsoft concerns are issues related to the disclosure of sensitive data, jailbreaking techniques to overtake AI models, and forcing third-party applications and model plugins to bypass safety filters or produce restricted content. Russinovich mentioned a specific attack method, Crescendo, which can bypass content safety measures to induce a model into generating harmful content.
Holistic Approach to AI Security
Russinovich likened AI models to “really smart but junior or naive employees” who, despite their intelligence, are vulnerable to manipulation and can act against an organization's policies without strict supervision. He stressed the inherent security risks within large language models (LLMs) and the need for stringent guardrails to mitigate these vulnerabilities.
Russinovich has developed a generative AI threat map that outlines the relationships among these various elements. This map serves as a crucial tool for understanding and addressing the multifaceted nature of AI security threats. He provided an example of how planting poisoned data on a Wikipedia page, known to be a data source, could lead to long-term issues even if the data is later corrected. This makes tracking down the poisoned data challenging because it no longer exists in the original source.
