TL;DR: Almost every product shipping this year is adding an LLM or an AI agent, and most are bolting it on faster than they are thinking about how it can be abused. AI does not just behave differently from traditional software. It fails differently. It is probabilistic, so the same input can produce different results, and it can be talked into things, because plain language is now part of the attack surface. When an agent can also take actions, query a database, call an API, send a message, those failures stop being cosmetic and start touching your real systems. The good news: this is testable. It just takes a different kind of testing than the scanners most teams are relying on.
Software quietly became something we depend on
Not long ago, software mostly helped people do their jobs. Now it runs the things those jobs depend on. Cities manage utilities and traffic through connected sensors. Cars make driving decisions in real time. The networks that carry a country's calls and messages are defined in software, which is precisely why state-linked attackers have spent recent years living quietly inside them.
As software moved from the back office into the systems we physically rely on, the cost of a failure moved with it. A bug used to mean downtime and an apology. Increasingly it means something with weight in the real world.
AI is the newest layer of that shift, and the least understood. The reason this matters is not that AI is uniquely dangerous. It is that AI changed the shape of the risk, and most testing has not caught up.
Why AI fails differently
Traditional software follows fixed paths. Give it the same input and you get the same output, and you can reason about where it might break. AI systems are not like that, in two ways that matter for security.
First, they are probabilistic. The same prompt can produce different responses, which means the attack surface is non-deterministic. A weakness that does not show up on the first attempt may surface on the fortieth, which is exactly the kind of thing a static scanner is built to miss.
Second, and more importantly, they can be persuaded. With traditional software, the attacker manipulates inputs. With an LLM, plain language is the input, and the line between data and instruction is blurry. An attacker does not need an exploit in the classic sense. They need to talk the system into something. That is a genuinely new category of weakness, and it gets serious the moment your AI can do more than talk.
The two halves of the problem
It helps to split modern AI risk into two halves that deserve equal attention.
The model as something that leaks. Even a read-only assistant holds and handles information you would rather it did not give away. Prompt injection lets an attacker override the instructions you gave the model, often through content the model reads rather than anything the user types directly. From there the failure modes stack up: the model can be coaxed into revealing its system prompt, surfacing another user's data, or, in a retrieval setup, pulling documents out of your private knowledge base that the user was never meant to see. There is also the model itself as an asset. Through patient querying, an attacker can clone its behavior and walk off with the intellectual property, or, if they can reach the training pipeline, plant a quiet backdoor that only triggers on inputs they choose.
The model as something that acts. This is the half teams underestimate. An AI agent with tool-use privileges is no longer answering questions, it is doing things on your behalf: running queries, calling APIs, moving data, triggering workflows. Give it those privileges and a persuasion-based weakness stops being embarrassing and becomes operational. An agent that can be talked into running an unintended query, calling a tool with the wrong arguments, or chaining one action into another is a new kind of actor inside your system, one that follows instructions and does not always know whose instructions it is following.
The point of laying it out this way is not to alarm you. It is to make the new failure modes legible, because you cannot test for what you have not named.
Why a scanner, or your own team, usually misses it
Most AI weaknesses do not look like a line of vulnerable code. They look like a conversation. They emerge over multiple turns, under adaptive pressure, across thousands of slightly different phrasings, exactly the behavior a real adversary uses and exactly what automated scanning is not built to reproduce. Internal teams hit a different wall: they know their product intimately, which makes it hard to think like someone trying to break it, and AI red-teaming is a specialism most are still building.
This is where Cyrex's pair hacking model fits the problem unusually well. We pair senior security engineers with proprietary AI agents: the agents generate and iterate through huge volumes of adversarial prompts to find the jailbreaks and injection paths a human alone would never have time to reach, while the engineers do the work only people can, orchestrating the cascading, multi-step exploits that define real AI risk. In practice one engineer pressures the model's context while another targets the APIs and backend the model is wired into, so there is no gap between what the AI says and what the infrastructure behind it does. Using AI to test AI is not a gimmick here. It is the only way to match the speed and persistence of the threat.
Doing this properly means exercising the kind of adversarial techniques against a model that most AI providers block by default, for good reason. Cyrex is verified under Anthropic's Cyber Verification Program, a vetting process that permits approved security professionals to carry out legitimate offensive and defensive work that standard safeguards would otherwise interrupt. It is a small detail with a real implication: we are trusted by a frontier AI lab to conduct exactly this kind of testing, and we can do it without hitting the walls that stop most teams.
We have spent years finding what others miss for some of the most demanding studios and enterprises in the world. You can see the broader track record in our portfolio. The discipline is the same. The target is new.
When it makes sense to test
The honest answer is before the moment you would most regret skipping it. In practice that means before an AI feature goes public, before it ships to enterprise clients, after wiring up a retrieval pipeline, when you give an agent the ability to take actions, and after any fine-tuning or retraining that changes how the model behaves. If your AI touches user data or internal systems, it has earned a structured test.
A quick word on regulation
This is not only a technical question. AI features that process personal data sit squarely inside existing data-protection law, and newer rules such as the EU AI Act are layering obligations on top, especially for higher-risk uses. Regulated sectors like finance and healthcare inherit AI risk directly, which means an untested AI feature can be a compliance exposure as much as a security one. None of this is legal advice, but it is worth having on the radar before launch rather than after.
The takeaway
AI is not the enemy here. It expands what your product can do, and that is worth building toward. It also expands how your product can fail, in ways traditional testing was never designed to catch. The teams that come out of this period well are not the ones that move slowest. They are the ones that understand the new failure modes and test for them deliberately, before someone with worse intentions does it for them.
That is a solvable problem, and it is what we do.
Frequently asked questions
What is AI security testing? It is an offensive security audit built for AI systems: large language models, the applications around them, and the agents that act on their behalf. It looks for weaknesses that traditional penetration testing and automated scanners are not designed to find, such as prompt injection, data leakage through the model, and abuse of agents that have execution privileges.
How is testing an LLM different from a normal penetration test? Traditional testing assumes fixed, repeatable behavior. AI systems are probabilistic and respond to natural language, so the same input can yield different results and the model can be manipulated through conversation rather than code. Effective testing is adversarial, multi-turn and adaptive, pressuring the system across many attempts rather than running a one-time scan.
When should we test our AI features? Before any public or enterprise release, after connecting a retrieval (RAG) pipeline, when enabling agents or automation that can take actions, and after fine-tuning or retraining. If your AI interacts with user data or internal systems, it should be tested before it goes live.
Cyrex is an offensive security firm. We find the things other people's tests miss, for the biggest names in games and enterprise. Our AI security service brings the same adversarial discipline to LLMs, RAG systems and AI agents.
Explore AI security testing · Book an AI security assessment · hello@cyrex.tech · Cyrex on LinkedIn
Written by Ian Matthews
Ian is Head of Marketing at Cyrex, responsible for the full marketing picture - from brand strategy and campaigns to the copy and content that brings Cyrex's work to life. With 8+ years across B2B and B2C, his speciality is bridging the technical and the human with sharp, results-driven messaging.
