As AI systems move from experimental tools to operational infrastructure, the nature of risk has changed dramatically. What was once a question of model accuracy is now a broader challenge of system integrity, adversarial resilience, and real-world impact.
In a recent TrojAI webinar, “AI Red Teaming: What a Year of Breaking Models Taught Us”, CEO Lee Weiner sat down with AI red teaming experts John Vaina and Gavin Klondike to revisit how the threat landscape has evolved over the past year.
The discussion reveals a clear shift. AI security is no longer theoretical. It is operational, complex, and deeply intertwined with enterprise and national security concerns.
This blog distills the key themes and insights from that conversation.
From model testing to full-stack adversarial simulation
A year ago, much of AI security focused narrowly on model-level vulnerabilities. That scope has expanded.
Today’s red teaming efforts simulate attacks across the entire AI stack, including:
- Models and prompts
- Retrieval systems and data pipelines
- Agentic workflows and tool integrations
- Infrastructure and deployment environments
As John Vaina described, modern red teaming is no longer about probing isolated systems. It is about understanding how interconnected components behave under adversarial pressure.
This shift reflects a fundamental truth: risk emerges at the seams. Vulnerabilities often appear not within a single model, but in how systems are composed, orchestrated, and exposed.
The rise of agentic systems and new attack surfaces
One of the most significant changes over the past year is the rapid adoption of agentic AI systems.
These systems are designed to take autonomous actions, interact with external tools and APIs, and operate across multiple steps and decision points. While this capability unlocks powerful new use cases, it also introduces entirely new categories of risk.
Traditional security assumptions begin to break down when AI systems can execute unintended actions, chain decisions across environments, and amplify even minor prompt manipulations into real-world consequences. What might begin as a subtle input can cascade into a sequence of actions with material impact.
As a result, the attack surface has expanded beyond static inputs to include dynamic, evolving behaviors across interconnected systems.
Adversarial AI is now a real-world discipline
AI red teaming is no longer experimental or confined to academic research. It has become an active, high-demand discipline across government agencies, frontier AI labs, and large enterprises.
Organizations are increasingly conducting continuous adversarial simulations to identify weaknesses before they are exploited in real-world environments. This shift reflects a broader recognition that AI systems are now mission-critical, that failures can have material consequences, and that threat actors are actively probing these systems for vulnerabilities.
Security is no longer optional. It is foundational.
The convergence of AI security and national security
Another defining shift over the past year is the growing convergence of AI security and national security, elevating AI risk from a technical concern to a matter of strategic importance. As AI systems are embedded into critical functions like intelligence analysis, cyber defense, and infrastructure operations, a single vulnerability can have consequences that extend far beyond a single organization.
Experts in AI red teaming are now contributing to policy and strategic initiatives through organizations like the Institute for Security and Technology. This convergence underscores a critical shift: AI systems are increasingly treated as infrastructure, and their vulnerabilities carry potential geopolitical implications.
As a result, the conversation is no longer limited to engineering teams. It now includes policymakers, regulators, and national security stakeholders.
Key challenges organizations face today
Across the discussion, several recurring challenges emerged that need to be addressed to secure AI systems:
- Visibility gaps: Organizations often lack a clear understanding of how their AI systems behave under adversarial conditions.
- Rapid adoption outpacing security: AI capabilities are being deployed faster than security practices can mature.
- Complexity of multi-component systems: Modern AI applications are ecosystems, not single models. Securing them requires system-level thinking.
- Talent and expertise shortage: Experienced AI red teamers remain scarce, making it difficult for organizations to build internal capabilities.
Addressing these challenges requires more than incremental process improvements.
Organizations need purpose-built tools that can simulate adversarial behavior, provide continuous visibility into system performance, and evaluate risk across the full AI stack. By automating key aspects of red teaming and security testing, these platforms help bridge the talent gap, enabling existing teams to operate with the depth and consistency of specialized experts. Just as importantly, they make it possible to embed security into the development lifecycle, transforming AI security from a reactive exercise into a scalable, proactive capability.
Why traditional security approaches fall short
Conventional security frameworks were not designed for AI-driven systems. They were not designed to assess model intent, conversational context, or autonomous tool orchestration. They typically assume:
- Deterministic behavior
- Predictable inputs and outputs
- Static attack surfaces
AI systems violate all three assumptions. They are:
- Probabilistic
- Context-sensitive
- Continuously evolving
AI systems require a new security mindset. This mindset must combine adversarial thinking with behavioral testing and continuous monitoring. It must also be flexible enough to evolve as quickly as AI capabilities are.
The path forward: continuous, system-level AI security
The key takeaway from our red team experts is clear: AI security must be continuous, not episodic.
To achieve this, organizations need to test systems regularly under adversarial conditions. This should include an evaluation of the full AI stack - not just models but also the applications and agents now in widespread use. In addition, the behavior and performance of AI systems should be monitored in production environments. This is particularly important because in AI systems, the same inputs can elicit different outputs.
The key here is that this is not a one-time audit. Security needs to be integrated into the AI development lifecycle as an ongoing discipline.
Conclusion
The past year has transformed AI security from a niche concern into a core operational requirement.
As AI systems become more autonomous, interconnected, and impactful, the stakes continue to rise. The organizations that succeed will be those that:
- Treat AI security as a first-class priority
- Invest in adversarial testing and red teaming
- Build resilience into every layer of their AI systems
At TrojAI, we believe that understanding how AI systems fail is the first step toward making them trustworthy.
Learn more
TrojAI's mission is to enable the secure rollout of AI in the enterprise. TrojAI delivers a comprehensive security platform for AI. The best-in-class platform empowers enterprises to safeguard AI models, applications and agents, both at build time and run time. TrojAI Detect automatically red teams AI models, safeguarding model behavior and delivering remediation guidance at build time. TrojAI Defend is an AI application and agent firewall that protects enterprises from real-time threats at run time. TrojAI Defend for MCP monitors and protects agentic AI workflows.
By assessing AI risk during the development lifecycle and protecting AI systems at run time, TrojAI delivers end-to-end security across agents, applications, and models.
To learn more, please visit us at www.troj.ai.
