Recently, threat-modeling expert Adam Shostack faced an urgent request from a client who had integrated a newly vibe-coded app with customer data. This organization sought clarity on the potential risks posed by this application and needed insight into appropriate countermeasures without delay. In just 15 minutes, Shostack conducted a rapid assessment of the system.
During this brief analysis, he identified several significant threats to the app. Notably, issues like hallucination and biases arose, problems that wouldn't typically be captured through STRIDE, a common framework for application security threat modeling. "I found some interesting stuff that wasn’t obvious to me when I started," Shostack noted, highlighting the efficiency of focused sessions.
His methodology involved the use of PHANTOM-B, a threat modeling framework designed specifically for systems utilizing large language models (LLMs). The framework begins with the vital question, "What can go wrong?" but narrows its focus on aspects unique to LLM components within a system. Each letter in PHANTOM-B points to specific threats: Prompt injection, Hallucination, Anthropomorphization, Non-explainability, Training issues, Overreliance (inclusive of data quality concerns), Missing security engineering, and Bias.
PHANTOM-B is intended to work alongside STRIDE, complementing its broad applicability by addressing vulnerabilities associated strictly with LLM interactions, thereby streamlining the threat modeling process. The design focuses on delivering actionable insights promptly, making the usually cumbersome practice of threat modeling more feasible and efficient. Shostack emphasizes, "Let’s make it inexpensive to do this work," recognizing the pressing need for swift adaptations as organizations increasingly pivot towards AI technologies.
Traditional Threat Modeling Limitations
Threat modeling has long been a challenging endeavor for companies of all sizes. Many struggle with consistency due to the resource-intensive nature of the process, often relegating it to critical applications only. This limited approach leaves various software products vulnerable to threats. "AI didn’t break threat modeling. It exposed weaknesses that were already there," remarked Jeff Williams, founder of OWASP. He added that threat modeling’s lack of standardization and automation has hindered scalability.
The real challenge lies not only in identifying threats but also in understanding the systems themselves. Traditional methods involving surveys and diagrams often yield incomplete models that can rapidly become outdated as applications evolve. The unpredictable nature of generative AI compounds this problem, as these systems operate on probabilistic responses rather than deterministic rules, making their risks more complex to assess.
Brian Glas, a vice president at CODIFIC, emphasizes the need to revise risk assessment approaches to accommodate the non-deterministic behaviors of modern AI systems. This shift is echoed in Microsoft’s guidance on AI threat modeling, which reveals that the lines between data and commands can blur, creating new vulnerabilities and complications. Faster development cycles mean teams have less time to adequately secure systems, contributing further to the inadequacies of conventional threat modeling.
Transforming Threat Modeling Practices
Engaging in brief, structured threat-modeling sessions can seem trivial but offers substantial benefits. Shostack explains that these exercises need not be exhaustive; their aim is to uncover enough meaningful risks to inform immediate decisions regarding system adjustments or deeper reviews. By adopting an agile mindset, teams can iterate more effectively to articulate and address potential issues. “You make the experiments cheap, and when the experiment is cheap, you can run it repeatedly,” Shostack points out.
The goal is to generate a set of concrete scenarios illustrating possible system failures. These scenarios empower CISOs to grasp the risks associated with their systems and to prioritize solutions, whether that’s limiting data access, refining tool permissions, or re-evaluating the necessity of an LLM in their applications.
PHANTOM-B also introduces critical concepts such as anthropomorphization, questioning whether users wrongly assume that models understand human intent and can reliably follow instructions. Non-explainability raises significant challenges, especially when AI systems need to justify their outputs in fields such as hiring or healthcare. Users might receive logically plausible responses that don’t accurately reflect the model’s reasoning path, complicating accountability and auditing.
Shostack warns against assuming that embedding an LLM is a panacea for existing software risks. The introduction of such technology can exacerbate pre-existing vulnerabilities, particularly when organizations hastily deploy applications that haven't been thoroughly understood. It is crucial to employ resources like PHANTOM-B alongside ongoing contributions from industry groups to navigate emerging risks effectively.
Avoiding Pitfalls in AI Threat Modeling
Despite their novelty, LLM-powered applications exhibit characteristics typical of traditional software, such as reliance on external libraries, handling sensitive data, and requiring strict access controls. Ignoring foundational security principles while focusing solely on new AI-specific threats can leave organizations vulnerable. Williams cautions that overlooking established risks is a severe misstep—“Traditional application security has to be the foundation,” he asserts.
Understanding the context of an LLM within the broader application framework is vital. If teams don’t grasp how inputs, trust boundaries, and data flows integrate, they risk missing significant vulnerabilities. Attempting to catalog every potential attack is futile, given the plethora of possibilities. Instead, Glas advocates for a focus on protecting crucial assets and analyzing current preventive measures to identify gaps in safeguards.
In adapting threat modeling practices to the complexities of AI, organizations can better fortify their systems against evolving security challenges, ensuring preparedness in an increasingly digital landscape.
