All notes

Why AI Pilots Stall Before Production, and What Gets Them Over the Line (2026)

Most AI pilots stall because they prove a model can work, not that the business can run it; here is a readiness scorecard, the real cost drivers, and the red flags.

Most AI pilots stall because they answer the wrong question. A pilot shows that a model can produce a good answer on curated inputs; production needs proof that the business can run the whole workflow on real data, inside real permissions, at a known cost per completed task, with someone accountable when it goes wrong. Pilots get over the line when those requirements are written into the pilot from the first week: a named owner with a decision metric, an evaluation set drawn from real traffic, the integration and access model, a unit cost at volume, and an agreed risk sign-off.

What counts as a “stalled pilot”?

A pilot is a time-boxed trial of an AI system on a limited slice of work, usually with hand-picked users and data. Production means the system runs on the real flow of work, connected to the systems of record, with monitoring, an owner, and a support path. A stalled pilot met its demo goals but has no approved date, budget, or owner for that move, and survives on goodwill.

The stall is common enough to measure. MIT NANDA’s 2025 GenAI Divide report found that for embedded or task-specific generative AI tools, about 60% of organisations investigated them, 20% piloted them, and 5% implemented them successfully (MIT NANDA, 2025, p. 6). The authors call these figures “directionally accurate” because they come from interviews, so read them as a shape rather than a census. Gartner predicted in 2024 that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025, citing poor data quality, inadequate risk controls, escalating costs, or unclear business value (Gartner, 2024). A year later it predicted that over 40% of agentic AI projects will be canceled by the end of 2027 for the same reasons of cost, value, and risk (Gartner, 2025).

Bar chart comparing general-purpose LLMs and task-specific GenAI across investigated, piloted and successfully implemented stages, from MIT NANDA 2025.
Figure 1. Share of organisations at each stage, redrawn from MIT NANDA (2025), p. 6. Task-specific tools fall from 20% piloted to 5% implemented. Interview-based and directional, per the authors.

Newer data says the gap persists even in companies that do ship. Surveying 100 senior leaders at enterprises with revenue above US$1 billion, HFS Research found that on average 26% of agentic AI initiatives reach production, the journey from proof of concept takes about 12 months, 53% say it took longer than planned, and 43% of spend sits in stalled work (HFS Research, November 2026). Forrester’s 2026 survey finds 82% of respondents with at least one generative AI application in production, yet only 11% with many applications scaled at the business level (Forrester, November 2026). Getting one thing live is achievable. Getting it live on purpose, on time, and with a measurable effect is where the money leaks.

Why do pilots that “worked” fail to ship?

Think of a pilot as a test kitchen. The chef has the best ingredients, no queue, and unlimited time per plate. Production is the same dish on a Friday night, with whatever the supplier delivered and a full room. The recipe did not change; the conditions did. In AI terms, a pilot optimises for possibility on a small, clean sample, while production demands repeatability across the messy distribution of real work. The gap between the two is engineering and organisation, not model choice.

Deloitte’s January 2026 State of AI in the Enterprise report describes the same mismatch: a pilot “can run with a small team in a few months using cleansed data in an isolated environment,” whereas production needs integration, security reviews, compliance checks, monitoring, and maintenance, and use cases estimated at three months “can stretch to 18 months when integration complexities emerge” (Deloitte, 2026, p. 9). These are the specific breaks we look for.

Line diagram of the path from idea to pilot to production gate to live operation, with seven stall points marked between pilot and production.
Figure 2. Where pilots stall. The demo stage hides seven requirements that surface at the production gate. Modelshifts schematic, not survey data.

No owner, no decision metric

The most common cause sits upstream of any code. RAND interviewed 65 experienced data scientists and engineers and found that misunderstanding or miscommunicating the problem to be solved was the most frequent root cause of AI project failure; 84% of its industry interviewees named a leadership-driven cause as the primary reason projects fail (RAND, 2024). In practice: a pilot sponsored by an innovation team, measured by “users liked it,” with no operations manager who owns the throughput number and would sign the go-live. HFS sees the mirror image: 41% of organisations in production name executive leadership as the most important stakeholder, against 24% of those still piloting.

Demo data is not production data

Pilot data is usually chosen by the people who want the pilot to succeed. A retail leader quoted by HFS put it plainly: “What worked on curated inputs failed badly on real customer chaos.” Real traffic contains scanned attachments, mixed languages, forwarded threads, and the awkward minority of cases that eat most of the effort. If the pilot never saw those, its accuracy number describes a different population from the one you are about to serve.

Nobody defined “good enough”

An evaluation set is a fixed collection of real, labelled examples, including hard ones, that the system must pass before and after every change. Without it, the go-live decision becomes a debate about anecdotes. With it, the question becomes concrete: on 300 stratified real cases, what share is accepted unchanged, what share needs edits, and what share is wrong in a way that costs money? IMDA’s agentic AI framework asks for the same, recommending pre-deployment tests of “overall task execution, policy adherence and tool use accuracy” across varied datasets (IMDA, 2026).

Integration, identity and permissions

A pilot reads a folder someone exported. Production reads the live system as a specific identity and often writes back. Which service account, which permissions, what happens when a write times out and the agent retries, and where is the audit trail? HFS respondents rank integration as the top enabler of reaching production, at 50%. RAND’s interviewees describe models that could not move from test to production because the environments were incompatible. We cover the retry and duplicate problem in AI workflow reliability.

Cost at volume was never modelled

Pilot costs are small and absorbed. At volume, the bill is model calls plus retrieval plus retries plus, above all, human review time. McKinsey’s 2026 survey reports that about one in five organisations now say AI operating costs, including tokens, constrain their AI use (McKinsey, 2026). The number that matters is cost per accepted outcome, covered below.

Risk sign-off arrives at the end

If legal, security, or data protection first see the system a week before launch, the launch moves. HFS respondents cite governance, compliance, and risk controls as a production barrier at 46%. The fix is not more paperwork. It is deciding early which actions the system may take alone, which need human approval, and who signs the risk.

The workflow was not redesigned

Dropping a model into an unchanged process rarely moves a business metric. McKinsey’s high performers, about 6% of respondents, are far more likely to have fundamentally redesigned workflows: nearly three quarters have, against one quarter of others. MIT NANDA’s barrier survey lists change management and unwillingness to adopt new tools alongside output quality as top reasons pilots fail to scale.

How do you know a pilot is ready for production?

Use a gate, not a vibe. We score each dimension 0 (absent), 1 (partly evidenced) or 2 (evidenced with an artefact someone can open). This is our working heuristic, not an industry standard, and the artefact column matters more than the score.

DimensionA score of 2 looks likeArtefact to ask for
Owner and decision metricNamed business owner, measured baseline, target, stop conditionOne-page decision memo signed by the owner
Production-representative dataEvaluation sample drawn from live traffic, stratified, with edge casesSampling note and the labelled set
Evaluation thresholdPass rate on that set, threshold agreed before resultsEvaluation report and regression run
Integration and accessScoped service identity, idempotent write-backs, logged actionsAccess matrix and audit log sample
Unit economics at volumeCost per accepted outcome at expected volume, review time includedSpreadsheet with visible assumptions
Risk and governance sign-offRisk tier set, approval rules for high-stakes actions, data protection review doneRisk register entry and sign-off
Operations and adoptionMonitoring, manual fallback, runbook, trained users, redesigned stepsRunbook, dashboard, training record
Readiness scorecard with seven dimensions scored 0 to 2, showing an illustrative pilot at 7 of 14 with two zero scores, against a gate threshold of 10.
Figure 3. Production-readiness scorecard applied to the illustrative quotation pilot below. Gate rule: no zeros and at least 10 of 14 before a limited rollout. Illustrative example, not client data.

The gate rule is simple: no dimension at zero, and at least 10 of 14 before a limited rollout on a subset of real traffic. A high total with a zero on ownership or sign-off is not ready, because those zeros are what stop launches in the final week. HFS recommends the portfolio version: track the share of initiatives reaching production and the share of spend tied up in stalled work, and make an explicit stop or recommit decision on each stalled one. Our note on which AI projects to keep, fix, or kill pairs with this.

What drives the cost and timeline of going to production?

A vendor who quotes one number for “taking it to production” without seeing your systems is guessing. The quote is driven by a few variables you can reason about before anyone writes a proposal.

Integration count and difficulty. Each system the workflow reads or writes adds authentication, data mapping, error handling, and tests. Read-only access to one document store is a different job from writing to an ERP and a CRM. Read-only versus action-taking. Once the system sends, books, posts, or updates, you need idempotency, approval rules, rollback, and audit; this is usually the largest single jump in effort. Evaluation and failure handling. The labelled set, regression harness, and exception queue are real work, and they make the system safe to change later. Security and governance. Identity, data residency, retention, personal data, sign-off cycles; calendar time here often exceeds engineering time. Operations and handover. Monitoring, ownership, updates, and documentation your team can run without the builder.

Model API cost is rarely the dominant line. The useful formula is:

Cost per accepted outcome = (model and infrastructure cost + human review and correction time + exception handling + monthly share of build and operations) ÷ outcomes accepted into the system of record.

An accepted outcome is a unit of finished business work, such as a quote sent or an invoice posted, that passed the checks you require. Tokens and drafts are not outcomes. Our AI ROI note works through break-even volume with this denominator.

Illustrative example: a quotation assistant (not a client case)

A 60-person B2B distributor pilots an assistant that reads request-for-quote emails and drafts quotes from the price list. The sales lead picks 50 recent emails and judges 45 drafts “right”: 90%. Then the team draws a stratified sample of 300 emails from last quarter, including PDFs, forwarded chains, and customers with negotiated discounts held in the ERP. On that set, 62% of drafts are accepted unchanged, 23% need edits, and 15% are unusable.

Now the economics at 1,200 requests a month, with assumed figures. Manual handling at 15 minutes each and a loaded cost of S$40 an hour is 300 hours, about S$12,000. With the assistant, accepted drafts take 2 minutes to check, edited ones 6, and failed ones the full 15: roughly 25 + 28 + 45 = 98 hours, about S$3,900. Add S$200 for model and hosting and S$1,500 for monitoring, evaluation upkeep, and support: about S$5,600 a month, or roughly S$5.50 per usable quote across 1,020 accepted or edited drafts, against S$10 manually. That is a monthly saving of about S$6,400. Had the pilot’s 90% held on real traffic (90% accepted, 10% redone by hand), the same cost model gives about 66 hours, roughly S$4,300 a month all-in, and a saving of about S$7,700. So the real saving is still large, but about 16% smaller than the pilot implied, and it only holds if the ERP discount lookup is built, which the pilot skipped. On the scorecard the pilot scores 7 of 14, with zeros on representative data and unit economics. The right call is a four to six week production rehearsal, not a launch and not a cancellation.

Timelines follow the same drivers. HFS measured about 12 months on average for large enterprises. A smaller company with one workflow, one or two integrations, and a decisive owner can move much faster, but only if the scorecard gaps are closed in parallel rather than discovered in sequence.

Should you build on a platform or go custom?

Most pilots start on a platform: an office-suite copilot, a CRM or helpdesk vendor’s agent builder, or an automation tool. That is often the right production answer as well. The question is where the platform’s boundary sits relative to your workflow.

SituationPlatform-native usually winsCustom usually wins
Data lives mainly in one vendor’s suitePermissions and identity come with itOnly for cross-system reasoning
Actions span several systemsIf the connectors are matureFor control of retries, approvals, audit
Evaluation needsLight; vendor monitoring is enoughStrict thresholds and regression tests
Volume and unit costLow to moderate; per-seat pricing worksHigh volume where per-outcome cost matters
Team to run itNo in-house engineersSomeone can own code and on-call

Two data points pull in opposite directions. MIT NANDA found externally partnered deployments reached deployment about 67% of the time against about 33% for internal builds, while cautioning the gap “may reflect organizational capabilities rather than implementation approach alone” (p. 19). McKinsey reports that 32% of respondents have decided against buying at least one software product because they could build it with agentic coding tools. Building has become cheaper; running what you built has not. Decide who will operate the system, then choose the stack.

What are the red flags that a pilot will never ship?

A few patterns predict a stall with uncomfortable reliability. The success metric is satisfaction rather than a business number with a baseline. Every accuracy figure comes from examples the builder selected. Nobody can say which identity the system will run as, or whether it will write to anything. The pilot has been extended twice with no change in scope. The cost estimate covers model calls but not review time. Risk and data protection colleagues have not seen it. The vendor’s “phase two” is a bigger pilot rather than a rehearsal on live traffic. Add Gartner’s warning about “agent washing,” the rebranding of assistants, RPA, and chatbots as agents; it estimates only about 130 of the thousands of agentic AI vendors are real (Gartner, June 2025).

What changes for companies in Singapore?

Singapore’s AI frameworks are voluntary guidance rather than an AI statute, but they are practical, and following them early removes the late sign-off delay.

IMDA’s Model AI Governance Framework for Agentic AI (version 1.5, published 20 May 2026, updated 5 June 2026) groups agent governance into four dimensions: assess and bound risks upfront, make humans meaningfully accountable, implement technical controls and processes, and enable end-user responsibility. Two recommendations translate directly into production work: enforce human approval “through system-level controls where possible” rather than prompt guardrails that “may be bypassed,” and roll agents out gradually with real-time monitoring.

The PDPC’s Advisory Guidelines on Use of Personal Data in AI Recommendation and Decision Systems (issued 1 March 2024) explain when the Business Improvement and Research Exceptions can apply to using personal data to develop and test AI systems (PDPC, 2024). If your evaluation set is built from customer emails, your data protection officer will want this referenced. The AI Verify Testing Framework assesses a system against 11 governance principles, covers generative AI since May 2025, and is mapped to the NIST AI RMF and ISO/IEC 42001 (AI Verify Foundation); it makes a good checklist for the risk dimension.

On funding, Enterprise Singapore’s EDGE Grant replaced EDG, MRA, and PSG for new applications from 30 September 2026 (Enterprise Singapore). The headline is up to 70% for SMEs, but support varies by activity. Large-scale Digital Transformation, which explicitly lists AI/ML, offers up to 50% for SMEs and 30% for non-SMEs, excludes “automation of isolated tasks without broader business transformation,” and requires that you have not started work or paid a vendor before applying (Enterprise Singapore). The point is timing: if a grant matters, scope the production phase before work starts, not after the pilot has quietly become the build.

How we approach it

When we are asked to look at a stalled pilot, the first step is usually not more engineering. It is a short, fixed-price assessment that scores the pilot against the seven dimensions, rebuilds the evaluation set from real traffic, and puts a cost per accepted outcome on the table. Sometimes the honest answer is to stop. More often it is a focused production rehearsal with two or three gaps to close. Companies without an internal AI lead sometimes keep us on as a fractional AI head to own that gate across several initiatives; others want the build and handover done end to end.

If you have a pilot everyone likes and nobody has scheduled, a 30-minute conversation is usually enough to see which dimension is holding it back.

FAQ

How long should an AI pilot run before a go or no-go decision?

Long enough to measure the agreed metric on real traffic, and no longer. Set the decision date and threshold before the pilot starts. A pilot extended twice without new evidence has stalled.

What accuracy is good enough for production?

There is no universal number. Derive the threshold from the cost of an error and the cost of review in your workflow, then measure on a stratified sample of real cases. A drafting assistant with human review can tolerate far more errors than an agent that posts payments.

Is the model usually the reason a pilot stalls?

Rarely on its own. RAND’s interviewees ranked problem definition and data first, and the surveys above point to ownership, integration, cost, and governance. Switching models seldom rescues a pilot that lacks an owner or an evaluation set.

Do we have to follow IMDA’s framework?

The Model AI Governance Frameworks are voluntary. The PDPA, which the PDPC guidelines interpret, is law. Using the frameworks early mainly makes the risk sign-off predictable.

Should the vendor who built the pilot build production?

Only if they can show the production artefacts in the scorecard, not just the demo. Ask who will operate the system after handover and what your team can change without them.