The evolution of agentic AI systems
Agentic systems - software systems composed of multiple interacting agents - have been used to solve complex problems for many years. The field of AI safety has grown around agentic systems to study the risks inherent in these systems.
With the advancements in Large Language Models (LLMs), the evolution of these agentic systems, commonly referred to as agentic AI, is gaining much more attention. However, adopting these agent-based systems, which incorporate generative AI, brings new complexities and risks. As these technologies continue to advance, new paradigms and strategies must be tailored to understand and address the risks associated with agentic AI. This blog provides insight into the emergence of these tools and considers the unique threats posed by agentic AI use.
What is agentic AI
Foundationally, agentic AI is rooted in agents and agentic systems. An agent is an autonomous unit programmed to perform tasks, make decisions, and interact with the environment. Agentic systems are composed of multiple interacting agents. These systems remained largely theoretical until recent technological advances enabled the creation of functional implementations through software.
Agentic AI is a category of artificial intelligence and agentic systems, enhanced by large language models and generative AI. It is designed to operate with a high degree of autonomy by leveraging the language model to make decisions, execute tasks, and adapt to changing environments with minimal human intervention or oversight. Agentic AI systems are capable of handling complex, multiple-step tasks by perceiving their surroundings, reasoning about the best courses of action, acting on them, and learning from their experiences to improve performance over time.
An example of this is an autonomous security agent that handles routine security monitoring and initial response while maintaining human oversight for critical actions to dramatically reduce time-to-decision and human analyst workload.
Differences between agentic AI and traditional AI
Traditional AI systems excel at specific, predefined tasks, but lack the flexibility to adapt beyond their programming. Agentic AI systems leverage reinforcement learning and language models to independently perceive, reason, act, learn, and adapt dynamically in complex environments.
Agentic AI systems differ from traditional AI in the following key ways:
- Autonomy: Agentic AI can make decisions and take actions independently, whereas traditional AI typically follows predefined instructions.
- Adaptability: Agentic AI continuously learns and adjusts its behavior, while traditional AI often requires manual updates and retraining.
- Problem-solving: Agentic AI can plan and execute complex sequences of tasks, whereas traditional AI focuses on single-task execution.
- Proactivity: Agentic AI actively pursues goals and solves problems, unlike traditional AI, which reacts to inputs when triggered.
- Interaction: Agentic AI engages with its surroundings dynamically, whereas traditional AI operates within predefined parameters.
- Reasoning: Agentic AI uses multi-step reasoning and planning to achieve objectives. In contrast, traditional AI processes information based on pre-set algorithms.
Components of an agentic AI system
The following are the components of an agentic AI system:
- Large language models: Agents leverage the language capabilities of LLMs to understand instructions and context, reason, and generate responses based on received input.
- Integrated services/tools: The components or external services that the agent interacts with to gather information and take action toward completing the task.
- Memory: Provides temporal context from the history of interactions, data retrieval from long-term memory, and general knowledge functions.
- Communication interface: Facilitates interactions with human users or other agents, ensuring effective collaboration and information exchange.
At its core, agentic AI blends large language models and software components to form complex workflows and solve problems efficiently and autonomously at scale. The large language model provides the necessary mechanisms for reasoning, adaptability, and autonomous decision-making within the system. The software underpins the system architecture and facilitates the integration and coordination of various applications within the deployment.
Agentic AI security versus traditional AI security
With its combination of machine learning capabilities and software components, agentic AI becomes an important security consideration. Agentic AI security refers to the practices implemented to mitigate the risks of using agentic systems. Agentic safety and agentic security are two different but related concerns. Agent safety refers to preventing agents from autonomously taking actions that can cause harm, while agent security is focused on preventing malicious users from exploiting vulnerabilities in the system. Given the components involved, these can be existing threats or new attack vectors arising from the capabilities and architecture of agentic systems.
An example of an existing threat is the use of large language models (LLMs) within agentic systems, which make them vulnerable to excessive agency. Excessive agency is a known risk in LLM applications in which the LLM is inadvertently given excessive functionality or permissions that could be exploited or could negatively impact the system goal through hallucination. Automation within an agentic system can further exacerbate this risk.
Examples of new vulnerabilities specific to agentic AI include the following:
- Intent breaking or goal manipulation: Techniques such as prompt injection could exploit weaknesses in the agent’s perception of its objectives.
- Memory poisoning: The memory mechanisms of the system history and any stored data in any datastores are exploited, leading to compromised decision-making and system integrity.
- Cascading hallucinations: The LLM within the agent (inadvertently or through interaction with a malicious user) produces false information that propagates and becomes reinforced through interconnected processes within the system. This leads to systemic misinformation and impairs decision-making across the automated workflow.
Why is agentic AI security important
The adoption of AI, in general, continues to rise, and agentic AI is expected to be the next step in the evolution of AI. The progress made through agentic systems will be significant and transformative.
As agentic AI becomes more commonplace and spur innovation, it will lead to new risks for the enterprise:
- Data integrity and system exploitation: AI agents often require access to data and critical systems to function effectively. If agentic AI systems are not properly secured, malicious actors could manipulate or hijack them to execute harmful tasks.
- Unintended consequences: While implemented through code, working with AI agents creates a paradigm where the results have nondeterministic outcomes through the use of the models within the system.
- Autonomy risks: The autonomous nature of these systems gives agentic AI more excessive agency. This, combined with less human interaction - and the added threat of user complacency - means that things can quickly go awry.
- Regulatory and compliance needs: Implementing safety guardrails and ensuring the system is secure is crucial for meeting legal and ethical standards. Agentic AI will eventually have its own regulatory and compliance requirements.
How agentic AI changes security approach
New, robust governance standards and frameworks around these agentic systems will be needed. Recent publications, such as the OWASP Top 10 for LLM Apps and Generative Agentic AI threats and mitigations and the MITRE OCCULT framework, exemplify this. These frameworks highlight the importance of blending traditional cybersecurity with new AI security standards specific to agentic AI.
These frameworks include understanding the system's architecture and design considerations, like how the agent integrates and interacts with the broader system, to identify unique risks, such as agent authentication and authorization. They also include continuously monitoring agent interactions and behaviors within the system through logging and implementing any necessary controls to ensure the agents operate within defined boundaries. This oversight is essential as the stochastic nature of LLMs can result in unexpected outcomes and misaligned goals.
For agentic AI, it is essential to adopt a proactive approach to security rather than a reactive one. As teams build out their agentic AI use cases, they should start with a security-first approach.
How TrojAI protects AI
As Agentic AI continues to evolve, it brings both transformative potential and significant security challenges. Its ability to operate autonomously, adapt dynamically, and execute complex tasks makes it a powerful tool, but also introduces new risks. Ensuring the security of agentic AI requires a proactive approach. By addressing these challenges early, organizations can harness the benefits of agentic AI while minimizing potential threats, paving the way for safer and more responsible AI adoption.
Our mission at TrojAI is to enable the secure rollout of AI in the enterprise. We are a comprehensive AI security platform that protects AI models and applications. Our best-in-class platform empowers enterprises to safeguard AI applications and models both at build time and run time. TrojAI Detect automatically red teams AI models, safeguarding model behavior and delivering remediation guidance at build time. TrojAI Defend is an AI application firewall that protects enterprises from real-time threats at run time.
By assessing the risk of AI model behaviors during the model development lifecycle and protecting model behavior at run time, we deliver comprehensive security for your AI models and applications.
Want to learn more about how TrojAI secures the largest enterprises globally with a highly scalable, performant, and extensible solution?
Visit us at troj.ai now.
