Debunking AI Security and Privacy Myths: Insights from Katharine Jarmul
At the InfoQ Dev Summit Munich 2025, Katharine Jarmul took the stage to challenge five prevalent myths in AI security and privacy. Her talk was a wake-up call, underscoring that conventional approaches may not be enough in an ever-evolving landscape. She argued that relying solely on technical solutions overlooks the deeper, systemic issues that demand our attention.
The Landscape of AI Automation
Jarmul opened her keynote by referencing Anthropic’s September 2025 Economic Index report, which marked a significant turning point: for the first time, AI automation had outpaced augmentation. This shift has delivered tasks into the hands of AI, leaving many privacy and security teams struggling to keep up. The rapid pace of change raises critical questions for users about the need for AI expertise and the reliability of various security measures. Unfortunately, amid this complexity, fearmongering and a blame culture are becoming common tactics.
Myth 1: Guardrails Will Save Us
The first myth Jarmul tackled was the assumption that guardrails alone can safeguard us in AI usage. While they serve to filter inputs and outputs, their effectiveness can be compromised. For instance, when users request translated code, such as instructions in French, they can bypass basic output guardrails. Similarly, presenting prompts in ASCII art can effectively deceive guardrails designed to block certain queries, like illicit activities. Techniques such as Reinforcement Learning from Human Feedback (RLHF) aren’t foolproof, and relying on them without broader measures can leave gaps in security.
Myth 2: Better Performance Solves Security
Next, Jarmul pointed out the myth that enhanced performance equates to improved security. While models with more parameters often yield better results, they can also inadvertently leak sensitive information contained within their training data. Copyrighted content, personal data, and medical information can all be extracted by malicious actors. Structured privacy approaches, like those offered by VaultGemma, aim to mitigate these risks but can sometimes fall short in practical scenarios. Thus, better performance does not inherently translate to better security.
Myth 3: Risk Taxonomies Are Enough
Jarmul then shifted gears to discuss risk taxonomies. While frameworks from esteemed organizations like MIT and NIST aim to outline potential risks, they often overwhelm organizations by listing hundreds of threats and possible mitigations. Jarmul advocates for an "interdisciplinary risk radar," which would involve collaboration among stakeholders across various domains, including security, privacy, product management, and data analysis. By doing so, organizations can pinpoint real, relevant threats and foster a culture of agile solutions—enhancing their risk preparedness.
More Read
Myth 4: One-Time Red Teaming Suffices
The fourth myth Jarmul addressed revolves around red teaming, the practice of simulating attacks to identify vulnerabilities. While it’s a valuable exercise, the reality is that cyber threats are constantly evolving. As systems change and new attack strategies emerge, a one-time red teaming assessment becomes obsolete. Jarmul recommends an ongoing, cyclical approach that combines threat modeling frameworks, continuous monitoring, and regular red teaming. By making these activities a regular part of security practices, organizations can adapt more fluidly to new challenges.
Myth 5: The Next Version Will Fix This
Lastly, Jarmul tackled the belief that the next version of a model will inherently resolve existing issues. Analyzing usage data from ChatGPT, she highlighted concerns about companies’ intentions with user information. For instance, some are developing means to track user behavior for hyper-personalized advertising, raising further privacy concerns. Jarmul urged teams to diversify their model providers, looking beyond mainstream options. Incorporating local models, such as Ollama, GPT4All, and Apertus, can provide better privacy control than traditional cloud-based services.
Through her insightful examination of these myths, Jarmul has laid the groundwork for a more nuanced understanding of AI security and privacy. By advocating for interdisciplinary collaboration, ongoing testing, and a holistic approach to risk, she emphasizes that addressing these challenges requires more than technical fixes; it mandates strategic, adaptable frameworks that keep pace with rapid technological advancements.
Inspired by: Source
