In the crowded arena of orchestration frameworks such as LangChain, CrewAI, and AutoGen, most evaluations focus on developer experience and ecosystem maturity. However, a more pressing question often remains unaddressed: how significantly does the choice of framework influence the susceptibility of the agent to compromise?
My recent testing reveals a clear answer: it matters—a lot. The discrepancies in security performance across different orchestration frameworks are substantial and not reflected in popular comparison guides.
The Evaluation Framework
To assess the security implications of these frameworks, I engineered an evaluation harness designed to conduct adversarial payload tests. This included methods like tool call hijacking, cross-tool injection, and memory poisoning, among other attack types. The complete methodology, including the dataset used, is available on GitHub for those interested in the details. Importantly, throughout this process, I maintained a consistent AI model to ensure that any variations in compromise rates would solely stem from the orchestration frameworks—CrewAI, LangChain, AutoGen, and SmolAgents—being tested.
If these frameworks were just interchangeable options around a single model, one would expect compromise rates to remain relatively uniform across the board. However, that was not the case.
Results of the Testing
The data tells a compelling story: across thousands of tests, the compromise rate varied dramatically, ranging from 11.9% on the most secure framework to a staggering 31.1% on the least secure—essentially a 2.6x difference, solely attributable to framework choice. The model and the attacks remained constant; only the orchestration layer changed.
This divergence is not trivial. It highlights a fundamental difference in security postures, raising critical questions that teams should address before deployment.
To illustrate, the variance is depicted in Figure 1, where the same AI model interfaces with different frameworks, resulting in a notable difference in compromise rates.
Julie Brunias
Understanding the Underlying Factors
It’s crucial to recognize that orchestration frameworks aren’t merely neutral connectors; each one introduces specific architectural decisions regarding tool call validation, memory management, and the autonomy of agents. The design choices inherent to these frameworks directly influence the potential attack surface.
A framework that rigorously validates tool requests or conservatively manages memory reduces potential vulnerabilities far more effectively than one that operates with a more permissive approach. This difference can be identified at a granular level. For instance, requiring tool calls to undergo a strict schema check before execution closes off many potential attack vectors that a more lenient framework would leave open.
The Market's Security Oversight
The prevailing trend in framework comparisons tends to place security as a minor bullet point among various other features. Resources from sites like Bestarion, Atlan, and Moxo tend to focus on aspects such as ecosystem maturity and human-in-the-loop capacities rather than on actual adversarial defense capabilities. Surprisingly, they rarely investigate measured security differences through critical testing.
Practical Considerations for Teams
When your team evaluates orchestration frameworks for AI systems, security concerns should hold equal weight with performance and developer experience. Here are several practical recommendations before finalizing your choice:
- Approach security claims with skepticism: Don't take claims at face value. Documentation may outline a framework's capabilities but will likely fall short of explaining its resistance to specific attack vectors like memory poisoning or tool hijacking. Running your own adversarial tests is vital for meaningful insights.
- Assume model safety doesn't guarantee frame security: A securely trained model wrapped in a permissive framework might suffer from poor security outcomes compared to a less robust model in a tighter framework.
- Evaluate deployed systems proactively: If your infrastructure is already in place, test its security rather than waiting for a planned migration to a different framework. Identifying vulnerabilities in your current system is key for timely improvements.
The Larger Perspective
Conversations around the security of AI agents have predominantly focused on the models themselves—primarily concerning which are most resistant to external manipulations. This perspective is limiting. The orchestration framework, which dictates how the model operates and interacts with its environment, plays a significant role in the overall security architecture. Publicly available evaluation guides often fail to provide adequate data regarding how these layers respond to various forms of attack.
It’s clear that the model is merely one aspect of an agent's security; the orchestration framework could very well represent the most variable factor in its vulnerability. If your organization is currently evaluating frameworks or has implemented one without thorough security testing, addressing these issues promptly could be vital for safeguarding against potential threats.
