Ask a room how many AI systems the organization runs in production. Somebody will give a number. Somebody else will give a different one. And that gap is the distance between a model that works and production-ready AI. It isn't a failure of attention. Counting is genuinely hard. And the best evidence comes from the one set of organizations legally required to do it.
US federal agencies have been required to inventory their AI use cases annually since 2020. In July 2025 the Government Accountability Office reported that across 11 agencies it reviewed, the number of reported use cases nearly doubled from 571 in 2023 to 1,110 in 2024. While generative AI use cases rose about ninefold from 32 to 282.
The 2025 government-wide inventory, published by the Office of Management and Budget, recorded 3,611 use cases across 56 agencies, more than double the previous year. Some of that’s real growth. A large share is agencies finding systems that were already running.
Reporting on the 2024 inventory alone produced totals of 1,757 and later 2,133 depending on when you looked. That’s because components that had never been reported at all were pulled in by a more rigorous data call.
These are organizations with a statutory obligation to count, a central body collecting the returns, and public scrutiny of the results. Enterprises have none of those and generally assume they're in better shape.
That assumption is the problem this article highlights. The federal numbers aren't a story about government inefficiency. They're the best available measurement of how far an inventory drifts from reality when the obligation to maintain it is strong. Produced by organizations under far more scrutiny than yours.
Which means your own count is a guess. And the direction of the error is knowable. It's low. None of which is an argument for slowing down. Shipping models is the point, since the organizations getting value out of AI are shipping more of them rather than fewer. My argument is actually that the inventory is a control rather than a reporting exercise. And that everything else worth having sits on top of it.
What follows is the three questions that reveal whether a live system is genuinely production-ready. I’ll also cover why a governance policy stops working the moment it meets a deployment pipeline. We’ll close with how to build a gate nobody wants to bypass, and what proportionate regulation is now asking you to prove.
Can you answer three questions about every model you're running?
There are three simple questions that reveal whether your system is genuinely production-ready. Try these on whatever went live last quarter:
- Who owns this system, and what was it approved to do? Not who built it. Who answers for it now. In a lot of organizations the answer is a data scientist with no authority over the production environment, which means the named owner and the actual owner are different people and neither knows it.
- How does it perform today compared to the day it was approved? This needs a stored evaluation against a versioned test set with a date attached. A number in a go-live slide deck isn't a baseline, and monitoring without a baseline detects nothing, which is why so many monitoring dashboards are quietly decorative.
- What happened the last time it changed, and who signed off? Delivery tooling usually captures most of this already. It just isn't connected to anything that governs the model, so the deployment record and the approval record live in different systems and never meet.
An organization that can answer all three for every live system has production-ready AI. One that can't has models that happen to be running. Which is a different thing wearing similar clothes.
Why doesn't a governance policy survive contact with a pipeline?
Because a policy describes what should happen and a control stops something from happening until a condition is met. That distinction sounds academic right up until the first model ships.
Take a requirement that every AI system touching customers gets a documented risk assessment. As a policy it lives in a framework, appears in onboarding, and gets communicated internally. Whether it actually happens depends on whether individual teams know about it and choose to comply. You’ll find out during an audit.
As a control it's a check in the deployment pipeline. The model can't be promoted unless a risk classification record exists and links to the artifact being promoted. Missing record, failed promotion, specific error message naming what's required. The failure gets logged, and the log is the evidence.
The added engineering is smaller than most teams expect. The classification form already existed in the policy version. What's new is a registry entry and a pipeline check. Which is days of work rather than a program.
The difference in outcome is absolute because a control can't be forgotten and generates its own audit trail. Any requirement that can't be attached to an artifact and checked at a transition gets complied with unevenly. That isn't a criticism of anyone. It's what happens when you ask people to remember a rarely invoked procedure that carries no consequence when skipped.
Which records make a system operable by someone who didn't build it?
There are four. And they should hang off things your delivery process already produces. Just know if you build a parallel bureaucracy, teams will route around it.
- A registry entry gives every deployed system an owner, a purpose, a risk classification, a data inventory and links to its evaluations. Make registration a precondition of deployment and the inventory problem solves itself structurally. Run it as a periodic survey instead and it's stale within a quarter.
- A stored evaluation gives you the baseline, tied to a versioned test set with a date and the results. Without it, drift is undetectable and every performance conversation is an argument about memory.
- A deployment record captures what shipped, by whom, with which approvals, at what time, against which registry entry.
- A monitoring configuration defines what's measured, which thresholds trigger action, and who gets the alert. Monitoring with no defined action on breach is telemetry, not control. And the difference matters the moment an examiner asks what happens when a threshold is crossed.
The NIST AI Risk Management Framework backs this directly. Its four functions are govern, map, measure, and manage. With governance built as a cross cutting function infused through the others rather than sitting above them as a stage. And with risk management meant to run continuously across the lifecycle. Attaching requirements to artifacts is what turns cross cutting from a diagram into something operational.
What does a gate look like that nobody wants to bypass?
Automatic, proportionate, and fast enough that going around it would take longer than going through it. Gates that add weeks get bypassed. And a bypassed gate is worse than no gate because it leaves a record suggesting control that wasn't there.
Proportionality does the work. One gate applying the heaviest requirements to every model produces either paralysis or a steady stream of exceptions. And exceptions become the norm within two quarters.
Tier by impact instead. Low impact internal systems need registration and an owner. Systems touching customers or feeding regulated decisions need classification, a documented evaluation, and an oversight design with named approval. Systems inside a specific regulation need whatever that regulation says.
Run the check in the pipeline, not in a meeting. It queries the registry, confirms the records exist and are current for the tier, and passes or fails with a reason. Where approval is required, make it a recorded decision rather than a standing review board. Which removes the largest single source of delay in most designs.
The failure behavior is what gets negotiated away, and it shouldn't be. If the gate can be overridden without a recorded exception and a named approver, it's advice. Advisory gates decay steadily because every individual override is reasonable. And the aggregate has no control at all. Teams designing secure AI deployment patterns tend to find the exception path deserves as much design attention as the primary one.
What is proportionate regulation actually asking you to prove?
That your judgment was sound, which is a heavier ask than following a checklist. The EU AI Act entered into force in August 2024 and became applicable in August of this year. Obligations for general purpose AI models and the governance rules went into effect in August of 2025.
Under the AI Omnibus, in force starting in July 2026, rules for high risk systems in the sensitive areas listed in Annex III start to apply in December 2027. And rules for high risk AI embedded in regulated products under Annex I begin the following year in August 2028.
Those obligations cover:
- Risk assessment and mitigation
- Data quality
- Human oversight
- Automatic logging for traceability
- Technical documentation
- And accuracy and cybersecurity
In United States banking, the Federal Reserve, the Office of the Comptroller of the Currency and the Federal Deposit Insurance Corporation issued revised model risk management guidance in April 2026 under SR 26-2. Which supersedes SR 11-7 from 2011, and the 2021 interagency statement on models supporting anti money laundering compliance. It emphasizes a risk based approach tailored to each institution's profile and operational complexity.
Both movements traded prescription for proportionality. Argue that a production-ready AI system warrants lighter treatment, and you need a documented classification, a recorded rationale, and monitoring that would have caught you if the judgment were wrong. That's the same four records described above. Which means the deferred deadlines bought time to build controls rather than permission to skip them.
Teams working through enterprise AI data security will recognize the shift. And anyone still weighing build versus buy should treat control infrastructure as something a platform supplies rather than something every project reinvents. It rests on the same discipline that separates AI ready data teams from the rest. If you want a helping hand as you deliberate, don’t hesitate to reach out.
What else do enterprise teams ask about production readiness?
How long does it take to put these records in place?
With an existing delivery pipeline, the registry, the gate and the evaluation record take roughly a quarter of focused engineering. Agreeing the tiering takes longer, because that needs decisions about risk appetite a document can leave vague and a control can't.
What should be built first when nothing exists?
The inventory. Nothing else can be prioritized without knowing what's running, and the federal experience shows the first honest count is always higher than expected. Making registration a precondition of deployment stops the list decaying the day after it's compiled.
Do the deferred EU AI Act deadlines reduce urgency?
They move the deadline without changing what the obligations require. Technical documentation describes design decisions being made right now, and reconstructing it later from a live system costs considerably more than capturing it as you go.
Who should own a production system, the model owner or the business?
The business owner carries accountability, the model owner answers for performance, and technology runs the enforcement points. Put accountability entirely in a central function and you get a queue, which is the most common reason gates get bypassed in the first place.
.svg)