On March 24th of this year, two versions of LiteLLM were published to PyPI carrying a credential stealer. They were live for about 40 minutes. In that window, according to LiteLLM's own security advisory, the package that sits between enterprise applications and more than a hundred model providers shipped code designed to harvest cloud credentials, SSH keys, and Kubernetes secrets.
Datadog Security Research advised DevSecOps professionals to treat any host or CI job that installed those versions as a full-credential exposure event. Now here's the part that should reorganize your enterprise AI security review. The attack didn't start at LiteLLM. It started with Trivy, an open source vulnerability scanner running inside LiteLLM's own build pipeline.
Yes, ironically enough a compromised security tool exfiltrated the publishing token, and the poisoned package went out from a trusted release path.
However, the source code on GitHub was never modified. No prompt was injected. No model was jailbroken. No model card was inaccurate. Every control your enterprise AI security questionnaire asks about would have passed. And the credentials would still be gone.
The questionnaire has settled into a familiar shape. Which model, trained on what, evaluated against which benchmarks, and red teamed by whom. So the vendor returns a model card, a set of jailbreak resistance scores, and a SOC 2 report. The reviewer signs.
But what nobody pays attention to is that every question in that exchange is about the model. And almost none of the incidents are. For that reason, we’ll explore the importance of auditing the pipeline, not the model.
What does a typical AI security review actually cover?
A typical review covers model provenance and model behavior, which is roughly a fifth of the system. Reviewers ask which foundation model sits underneath, whether it was fine-tuned, what the vendor's evaluation results look like, and how the model responds to adversarial prompts.
What the review usually doesn't cover is everything the model is wired into:
- The retrieval index and its permission model.
- The embedding store and who can query across it.
- The tools the system can call and what those tools are authorized to do.
The dependency chain that assembles the whole thing at deploy time. That's where the system meets your data. And it's where the attack surface actually lives.
Why doesn't model-level assurance transfer to your deployment?
Model-level assurance doesn't transfer because the model is the one component you didn't configure. A foundation model's safety evaluation describes how it behaves in the vendor's own evaluation setup. With the vendor's system prompt and on the vendor's own test set.
Your deployment changes the system prompt, adds retrieval over your documents, grants tool access to your systems, and passes the output into your applications. None of that was in the evaluation.
OWASP's guidance has moved decisively in this direction. The OWASP GenAI LLM Top 10 published its 2026 edition in August, and the headline change was Excessive Agency climbing from sixth place to third. Prompt Injection and Sensitive Information Disclosure held the top two positions.
Read that ranking as an argument about where to spend review time. The framing has shifted from preventing the model from being fooled to containing what happens when it is. And containment is an identity and authorization question rather than a model question.
That shift isn't a footnote. It means a review that certifies the model and skips the permission boundary has audited the part that was never going to hold. And it skipped the part that decides the blast radius.
Where does the real attack surface sit?
The real attack surface sits in four places the model questionnaire never reaches:
- The retrieval layer
Similarity search frequently runs across the whole index before the app applies access control. So an over-shared drive indexed into a knowledge base returns exactly what it was built to return. The document was never supposed to be in the index. The model did nothing wrong.
- Tool and agent permissions
OWASP breaks excessive agency into excessive functionality, excessive permissions, and excessive autonomy. A document-reading tool that can also delete. A read-only feature connecting through an account that holds UPDATE rights. An irreversible action with no human approval.
- Output handling
Model output reaches browsers, databases, shell commands, and APIs. Downstream code executes it because the AI produced it. These are the same injection classes AppSec cataloged 20 years ago, arriving through a new door.
- The dependency chain
Model weights, adapters, orchestration frameworks, and inference libraries, assemble at deploy time from sources the security review never enumerated.
None of these are model defects. All of them are configuration and architecture decisions made inside your environment. Which is precisely why vendor attestation can't answer for them.
What does the retrieval layer expose that a model review misses?
The retrieval layer exposes whatever was indexed, to whoever can query it, regardless of the source system's permissions. This is the single most common gap we see in enterprise AI security reviews, and it's structural rather than accidental.
Index a document library into a vector store and its permission model usually does not travel with the content. The index holds chunks and embeddings, not the knowledge that chunk 4,812 came from a restricted folder.
Teams solve this two ways. Post-filtering searches everything, then removes results the user should not see, which leaks through result counts, score distributions, and timing. Pre-filtering constrains the search space to what the user can access before the search runs. Only the second is a control.
Ask whether document-level permissions are enforced before or after retrieval. And have it demonstrated with two accounts of different seniority. Teams on a proper abstraction layer answer quickly. Teams that indexed a shared drive in a hackathon can’t.
How should agent permissions be reviewed?
Agent permissions should be reviewed as identities, not as features. Every agent that can act holds an effective permission set, usually inherited from a service account provisioned for convenience, not scoped for the task.
Three questions do most of the work:
- What can it reach, as systems and data classes rather than product names
- What can it change, separating read from write from delete
- And what can it do without a human in the loop
The last one determines whether an error is recoverable. An agent scoped for a narrow task gets a broad credential because narrowing it was fiddly, then gets extended to new tasks without anyone revisiting it. The permission outlives its justification, quietly.
Auditability is a security concern here, not just compliance. Without logs tying the identity, the input, and the resulting change together, an incident review has nothing to reconstruct from.
What should replace the model questionnaire?
Replace it with a trace, not a longer questionnaire. Follow one real request end to end. What the user asked, what the system retrieved and under whose access rights. Don’t forget that understanding which model served inference, what tools ran with which credentials, and what the output touched on the way back is also important.
That takes an afternoon and surfaces what no document review finds. That’s because it examines the system as deployed rather than described.
Keep the model questions in the enterprise AI security review. Provenance still matters, and a model agnostic platform gives you options a welded-in deployment doesn’t. They’re just not the review. The model is the component you control least and have the most documentation about. Which makes it a comfortable place to spend attention that belongs elsewhere.
Ask where your data goes, who the system can act as, and what it can reach when it gets something wrong. If your review doesn’t answer those three, it’s auditing the model and missing the pipeline. If you want a second pair of eyes on that trace, we should connect.
What else do security teams ask about enterprise AI security?
Is a vendor's SOC 2 report enough for an AI security review?
No. A SOC 2 report covers the vendor's organisational controls, not the configuration of your deployment. It says nothing about which of your documents were indexed, what permissions your agents hold, or what your retrieval layer returns to an unprivileged user. Those are properties of your environment and have to be tested there.
Does prompt injection testing cover the main AI security risks?
Only partly. Prompt injection remains the most exploited category, but OWASP's 2026 list moved Excessive Agency to third place specifically because the damage depends on what the manipulated system is permitted to do. Testing inputs without reviewing tool permissions measures the likelihood of an incident while ignoring its size.
How do you check whether retrieval respects document permissions?
Ask whether access control is applied before or after the similarity search, then test with two accounts at different permission levels and compare results, result counts, and response times. Post-filtering can leak through metadata even when the restricted text is never returned, so a demonstration is worth more than a policy statement.
.svg)