Recent findings reveal a staggering 71% of organizations grant AI access to core business systems, yet only 16% manage that access with any real effectiveness, as highlighted in the 2026 CISO AI Risk Report. Leaders in incident response (IR) must start by asking three critical questions: Where is your AI system inventory? What protocols exist if an AI model begins to produce harmful outcomes? Who holds the authority to deactivate it?
Having spent over a decade scrutinizing the security domain across various sectors—including energy, banking, telecom, and manufacturing—one observation stands out: Organizations might have AI operating in production environments alongside an IR playbook, yet these are not intrinsically linked. They should be, but sadly, they're often treated as separate entities.
Chief Information Security Officers (CISOs) who assume their existing IR protocols are adequate for handling AI-related incidents are often mistaken. Those who have actually tested their response plans recognize the substantial gaps.
Dissecting AI Incidents: Understanding the Two Fundamental Types
Data indicates that from 2023 to 2024, AI incidents surged by 56.4%, documenting a total of 233 cases. Most IR frameworks, including NIST SP 800-61, MITRE ATLAS, and the GLACIS AI Incident Response Playbook, outline various types of incidents but often overlook a critical classification: distinguishing between failures originating from the AI model itself and those instigated by human actions. This distinction is crucial as it impacts detection, containment strategies, and legal implications.
Model-originated failures include issues such as degradation, bias, and hallucinations and occur when the AI operates as intended but poorly. For instance, the Epic Sepsis Model utilized in numerous U.S. hospitals only achieved a sensitivity of 33% during validation trials, missing a significant volume of actual sepsis cases while inundating healthcare providers with erroneous alerts, as noted in a 2021 JAMA Internal Medicine study. No external attacks were involved; instead, the system simply failed to perform effectively.
On the other hand, externally induced failures—such as adversarial attacks, data poisoning, and privacy breaches—emerge when a malicious entity compromises the AI's inputs or its training processes. A notable example involves Tesla’s Autopilot and its issues with phantom braking, which has drawn the scrutiny of NHTSA across countless vehicles. These two incident categories necessitate distinct defense strategies and playbooks.
Another nuanced hybrid case blends both categories: hallucinations, where model-originated errors end up creating substantial legal risks. When Air Canada’s chatbot fabricated a policy regarding bereavement fares, the airline faced repercussions, demonstrating a legal liability that primarily stemmed from a model error. Similarly, the case of Mobley v. Workday established that AI hiring systems might be seen as an agent of the employer, bearing direct legal implications. Complex situations like these underscore the necessity of involving legal teams in IR discussions.
Limitations of Traditional IR Frameworks: The CIA Triad
Fundamental principles of security, such as the CIA triad—confidentiality, integrity, and availability—fail to encompass the complexities of most AI incidents. This was evident when Air Canada’s chatbot fabricated policies: no sensitive information was compromised, nor were there unauthorized alterations. Accordingly, conventional IR metrics failed to identify any issues.
Moreover, established IR principles are built on deterministic models, which falter when faced with the probabilistic nature of AI systems. A viewpoint presented by Microsoft’s Security Blog reflects the challenge: a model might generate harmful outputs today while delivering completely different results tomorrow based on the same input. The crux of the problem doesn't lie in specific lines of code but rather in inherent probability distributions, which cannot be "patched."
Statistics further underline this disconnect: the average time to detect AI incidents stands at about 4.5 days, with a significant 67% of such incidents arising from model errors rather than external attacks. Yet, security budgets predominantly support perimeter defenses aimed at the latter, leaving organizations ill-equipped to respond to the real threats.
Establishing Mature AI Incident Response Capabilities
When I address industry conferences, inquiries often revolve around the characteristics of an advanced AI incident response capability. The answer usually boils down to three foundational elements that should be established before any incident surfaces.
First, organizations need an AI Bill of Materials (AIBOM) for every production system. Think of it as akin to a software SBOM, encompassing essential details like the base model, training datasets, third-party dependencies, and other components. Without this foundational knowledge, investigating issues like data poisoning becomes nearly impossible. The OWASP GenAI Security Project provides an open-source AIBOM generator that produces outputs compliant with CycloneDX format and SPDX standards, making adoption straightforward.
Second, each production AI system should have an accessible model card—this is not just a document lost in a shared drive. It needs to be a resource that can be quickly referenced during incident response. Critical information such as the origin of training data, version details, performance limitations, and access controls should be readily available, especially since many organizations currently possess documentation that’s geared toward data scientists instead of the incident response team. That kind of material is not documentation; it’s potential liability.
Third, it’s vital to have a designated data scientist within your incident response call tree. This individual should possess the authority to analyze model behavior in real time, akin to how traditional IR teams have network engineers on call. In this context, AI IR also requires accessible expertise to navigate the intricacies of the system under distress.
One additional aspect that remains a rarity involves establishing documented rollback thresholds for deployed AI models. Having a pre-defined criterion outlining the anomaly rate or fairness deviation that triggers immediate containment or system rollback can streamline an organization’s response in a crisis scenario. Organizations that lack this clarity often find themselves caught in debates over whether a situation constitutes a genuine problem—while those with established thresholds can focus on active incident management.
Essential Preparatory Actions Ahead of AI Incidents
Organizations need to revisit their detection protocols. Implementing anomaly scoring, monitoring data distributions for drift, and tracking behavioral data regarding model API usage should become integral parts of the detection framework. Relying solely on traditional SIEM tools won't suffice; investment at the AI system level is required.
Next, redefining the containment strategy is crucial. In most AI incidents, simply isolating a system may not be the best first step. Often, opting for a rule-based fallback while keeping the service operational may inflict less damage than taking the system offline and escalating operational risks. It’s crucial to document pre-established rollback criteria and designate a fallback solution.
Incorporating legal expertise into incident preparedness is also vital. As seen in Mobley v. Workday, legal liabilities can rest with both the AI vendor and the operating organization when bias incidents occur. Furthermore, the Air Canada incident highlights that disclaimers cannot absolve responsibility for an AI’s external communication. If legal teams become aware of an AI issue only through press inquiries, something has already gone awry.
Lastly, organizations should compile an AI inventory and treat it like a key asset register. Begin with the highest-risk systems, those accessing critical personal or financial data. The GenAI-IRF framework provides a structured approach, while the GLACIS AI Incident Response Playbook aligns these practices with established procedures like NIST SP 800-61 and MITRE ATLAS.
With 42% of organizations having encountered a suspicious or verified AI incident, according to recent statistics, and over half admitting their security measures are either reactive or inconsistent, refreshing your incident response playbook is critical. Proactive preparation is not just advisable; it’s essential.
This article is published as part of the Foundry Expert Contributor Network.
Want to join?
