AI Pilot to Production: The MLOps Checklist That Ships

AI pilot to production, a lab prototype connected by a bridge to the same AI core running inside a production facility

AI pilots rarely stall because the model is bad. They stall because the data pipeline was a one-off, nobody owns the model after the demo, nothing watches it in production and nobody priced what it costs to run. Score your pilot against Google’s three MLOps levels, close the gaps on the checklist below, and buy the generic tooling instead of building it. Small teams ship by narrowing scope, not by adding headcount.

Most AI pilots stall on data pipelines, ownership, monitoring and unclear value, not on the model. Reaching production takes versioned data and models, automated deployment, drift and quality monitoring, a named owner and a cost-per-prediction budget.

The demo went well. Leadership clapped. Six months later the model still lives in a notebook on one data scientist’s laptop, and the business process it was meant to change hasn’t changed at all.

That gap between a pilot and a working system is what this post is about. It’s the gap our machine learning engineering team gets called in to close, and it’s rarely about the algorithm. It’s about everything around it, the plumbing that turns a clever result into something the business can rely on every morning.

We’ve written this for CTOs and IT leaders at mid-market companies, where the AI team is two people and a contractor, not a platform group of twenty. You’ll get honest numbers on how much of this stalls, a way to score where yours is stuck, and the checklist we use before anything goes live.

Do most AI pilots actually reach production?

Fewer than half, by most credible counts. IDC found only 4 of every 33 AI proofs of concept reached wide deployment. S&P Global found 46% of proofs of concept were scrapped before production. The exact figure depends on what counts as failure.

You’ve probably seen a scary percentage quoted with no source. The real numbers are scary enough, and they measure different things, which matters when you’re explaining this to a board.

SourceWhat it measuredFinding
IDC with Lenovo, 2025Proofs of concept reaching wide deployment88% didn’t. About 4 of every 33 graduated
S&P Global, 2025Initiatives abandoned42% of companies abandoned most AI initiatives, up from 17% a year earlier
Gartner, 2024GenAI projects dropped after proof of conceptAt least 30% abandoned by the end of 2025
MIT NANDA, 2025Pilots producing rapid revenue gainsAbout 5% did

Four studies, four definitions of failure.

Notice the spread. Abandoned, never deployed and no revenue impact are three different problems with three different fixes. The MIT figure gets quoted as 95% of AI fails. It doesn’t say that. It says 95% didn’t move revenue quickly, which is a value problem as much as an engineering one.

Five reasons pilots die between the demo and production

Gartner’s own explanation for abandoned GenAI projects lists poor data quality, inadequate risk controls, escalating costs and unclear business value. Not one of those is the model. Google makes the same point in its MLOps architecture guide, noting that only a small fraction of a real ML system is the ML code itself. The rest is data collection, validation, serving, monitoring and configuration. That observation goes back to Google’s 2015 paper on hidden technical debt in machine learning systems, which warned that quick ML wins don’t come for free.

Small AI core surrounded by a large network of pipes and gauges, showing the model is a small part of an ML system

Here’s how that plays out in mid-market companies.

  • The pilot ran on a CSV someone exported by hand. Production needs that data every day, cleaned the same way, and no pipeline exists.
  • Nobody owns it after the demo. The data scientist moves to the next idea and the model quietly ages.
  • What happens when the model is wrong? If nobody can answer that, legal and operations won’t sign off, and they’re right not to.
  • Cost shows up late. A GenAI pilot that cost $300 in API calls for 50 testers can cost far more for 5,000 users, and nobody modeled it.
  • And the success metric was never agreed. A demo can impress without proving anything.

Which pilot should you ship first?

Ship the pilot with the clearest business owner, data that already flows reliably and a cost of being wrong you can live with. A modest use case that goes live beats an ambitious one that stays a demo.

Most mid-market companies we talk to have three to six AI ideas in flight. That’s the problem. Small teams spread across six pilots ship zero. Score each one quickly on four questions.

QuestionStrong answerWeak answer
Who owns the outcome?A named manager whose numbers improve if it worksThe innovation team, or IT
Does the data already flow?It lives in an ERP, CRM or warehouse with a reliable daily loadSpreadsheets, email inboxes, a vendor portal export
What does a wrong answer cost?A person catches it in review, or the cost is smallA customer, a regulator or a payment sees it first
Can you measure the gain?Hours saved, forecast error, collection days, against a recorded baselineBetter insights, improved experience

Four-question pilot scorecard.

Back-office use cases tend to win this scoring. That fits the MIT research, which found the biggest returns in back-office automation rather than the customer-facing projects that get the most budget. Invoice matching, demand forecasting and support ticket routing aren’t glamorous. They ship.

Where is your pilot today? A three-level self-check

Google defines three MLOps maturity levels. Level 0 is manual, Level 1 automates the training pipeline, and Level 2 automates testing and deployment of the pipeline itself. Most stalled pilots sit at Level 0, and most mid-market use cases only need Level 1.

Three platforms from manual handling to full automation, representing MLOps maturity levels 0, 1 and 2
LevelWhat it looks likeSigns you’re here
0. ManualNotebooks, hand-run training, a model file handed to ITRetraining means a person remembers to do it. Nobody would notice if accuracy slipped
1. Pipeline automationTraining runs as an automated pipeline on fresh data, with validation and a model registryNew data triggers retraining. Every model version is tracked and can be rolled back
2. CI/CD automationChanges to the pipeline itself are tested and deployed automaticallySeveral models, several people, frequent changes, and releases are routine

MLOps maturity levels as described in Google Cloud’s MLOps guide.

Google’s description of Level 0 problems reads like a stalled-pilot diagnosis. Models degrade in production, nothing actively monitors them, retraining is rare, and manual handoffs create a mismatch between how the model was trained and how it’s served. Sound familiar?

The useful insight for a small team is the target. You don’t need Level 2 to ship one forecasting model or one document classifier. You need Level 1 done properly. Chasing Level 2 too early is how a two-person team builds a platform instead of shipping a product.

The pilot-to-production checklist

Before a model goes live, confirm seven things. Reliable data, a versioned model, automated deployment with rollback, monitoring for drift and quality, a named owner, a cost budget, and documented risk controls. Missing any one is a common reason pilots stall.

A model passing seven inspection gates before production, representing the pilot-to-production checklist

This is the list we walk through before a model touches a real business process. Print it. Argue about it in a meeting. That argument is half the value.

Data

  • An automated pipeline feeds the model, not a manual export
  • Schema and value checks run on every load, and bad data stops the run instead of reaching the model
  • Training data is versioned, so you can say exactly what a model learned from

Model

  • Every model version sits in a registry with its metrics, data version and code version
  • A baseline exists. What did the process achieve before the model? If the model can’t beat it, stop here.

Deployment

  • Deployment is scripted and repeatable, run by a pipeline, not a person
  • Rollback to the previous version takes minutes and has been tested once on purpose
  • Integration tests cover the systems the model reads from and writes to

Monitoring

  • Input drift alerts when incoming data stops looking like training data
  • Output quality is measured against real outcomes on a schedule, even if that’s a weekly sample checked by a person
  • Latency, errors and uptime are on the same dashboard as the rest of production

Ownership

  • One named business owner decides whether the model is doing its job
  • One named technical owner gets paged when it breaks. A shared mailbox doesn’t count.

Cost

  • Cost per prediction or per conversation is known and budgeted at full production volume
  • Alerts fire when daily spend passes a set limit

Governance

  • Risks are mapped and owned using the four functions of the NIST AI Risk Management Framework, which are Govern, Map, Measure and Manage
  • For GenAI, NIST’s 2024 Generative AI Profile covers risks like made-up answers and data leakage
  • A human review step exists wherever a wrong answer costs real money or harms a customer

Testing deserves a special mention. Models are software, and they fail like software, plus in a few new ways. Our QA automation team builds the integration and regression tests that sit around the model, and we explain why that matters in QA vs quality engineering.

Classic ML vs GenAI: what changes with LLMOps

Most of the checklist applies to both. A few things shift when the model is a large language model you call through an API instead of one you trained.

Two parallel tracks for a classic model and a language model, representing MLOps vs LLMOps
ConcernClassic ML (MLOps)GenAI and agents (LLMOps)
What you versionTraining data, features, model weightsPrompts, retrieval sources, model and API version
How you test qualityAccuracy against labeled dataEvaluation sets of real questions with graded answers
What driftsInput data and real-world patternsThe vendor’s model updates, your documents, user behavior
Main running costCompute for training and servingTokens per call, multiplied by users and retries
Main new riskSilent accuracy decayConfident wrong answers, prompt injection, data leakage

MLOps vs LLMOps, the practical differences.

The evaluation set is the piece GenAI pilots skip most. Fifty real questions with agreed good answers, rerun every time a prompt or model version changes, will catch more problems than any amount of demo polish. And agents raise the stakes further. Gartner expects over 40% of agentic AI projects to be canceled by the end of 2027, citing costs, unclear value and weak risk controls. The same three gaps, louder.

Build or buy the MLOps stack?

Buy the generic layers, such as experiment tracking, model registry, deployment and monitoring, from the cloud you already use. Build only what’s specific to your business, which is usually the data pipelines and integrations.

The MIT research Fortune reported on found that tools bought from specialized vendors succeeded about 67% of the time, while internal builds succeeded at about one-third that rate. That’s GenAI pilots, not MLOps platforms specifically, but the lesson carries over. A mid-market team that builds its own model registry has spent its budget on something it could have rented.

Which platform? Follow your cloud.

  • On Azure and Microsoft 365, Azure Machine Learning fits naturally beside your identity and data.
  • Google Cloud shops get Vertex AI next to BigQuery.
  • AWS customers use SageMaker.
  • Already running Databricks for data engineering? Its built-in MLflow covers tracking and the registry without adding another vendor.

Bias disclosed. We’re a services firm, so you might expect us to push custom builds. We don’t. Custom platform code is the part we’d most like to delete from a client’s estate, because someone has to maintain it after we leave.

A realistic 90-day path for a small team

  1. Weeks 1 to 2. Pick one pilot. Agree the business metric, the baseline and the owner. Kill the others for now, politely.
  2. Weeks 3 to 6. Build the production data pipeline with validation. Our data engineering team usually finds this is half the total effort.
  3. Weeks 7 to 9. Register the model, script deployment and rollback with your DevOps pipeline, and write the integration tests.
  4. Weeks 10 to 11. Turn on monitoring and cost alerts. Run in shadow mode beside the current process and compare.
  5. Week 12. Go live for a limited group, with the human review step in place, and schedule the first monthly model review.

A 90-day pilot-to-production plan.

Ninety days is tight but realistic for one well-scoped use case at Level 1. It’s not realistic for three pilots at once. Narrow first. Then ship.

Short on hands? That’s what our engagement models are for, from a single embedded ML engineer to a team that owns the whole path to production.

Who we are

Upen Group is a digital transformation and engineering company headquartered in Irving, Texas. Our machine learning engineering, data engineering, DevOps and quality engineering teams work together to move mid-market AI pilots into production, from the data pipeline through deployment, monitoring and testing.

The short answer

Pilots stall on plumbing and ownership, not models. Score yours against Google’s three levels, aim for a solid Level 1, and work through the checklist before go-live. Buy generic MLOps tooling from the cloud you already use, build only your pipelines and integrations, and ship one use case before starting the next.

Got a pilot stuck at the demo stage? Send us a short description of the use case and where it lives today, and we’ll score it against the checklist and tell you what stands between it and production.

What CTOs ask about stalled pilots

So why do most AI pilots never make it to production?

Because the pilot proved the model, not the system around it. Data pipelines, ownership, monitoring, cost and risk controls are usually missing, and Gartner’s own list of reasons for abandoned GenAI projects names data quality, risk controls, cost and unclear value.

What share of AI projects actually reach production?

About 4 of every 33 proofs of concept in IDC’s research. S&P Global found 46% scrapped before production. Figures vary because studies define failure differently.

MLOps vs DevOps, do we need both?

MLOps is DevOps discipline applied to machine learning, plus the parts DevOps never had to handle. Data versioning, retraining and drift monitoring are the big additions. If you already run DevOps well, you’re halfway there, and the same pipelines can deploy models.

Is LLMOps really different from MLOps?

Somewhat. You version prompts and retrieval sources instead of training data, test with evaluation sets instead of labeled accuracy, and watch token costs closely. Ownership, deployment, rollback and monitoring stay the same.

Should a mid-size company build its own MLOps platform?

Almost never. Use Azure Machine Learning, Vertex AI, SageMaker or Databricks with MLflow, depending on the cloud you already run. Spend your engineering time on data pipelines and integrations, which no vendor can build for you.

How do we know when a production model is getting worse?

Two signals. Input drift, meaning incoming data stops looking like training data, and output quality, meaning predictions stop matching real outcomes. Monitor both. The second one usually needs a person checking a sample every week or month, and that’s fine.

Realistically, how long does it take to get one use case live?

Around 90 days for a single, well-scoped use case at MLOps Level 1 with a clear owner. Longer if the data pipeline doesn’t exist yet or several pilots compete for the same people.