An AI proof of concept has one job: to answer whether the organisation should invest in building the system for production. Accuracy on a held-out test set does not answer that, because it does not tell you whether the business outcome improves at the cost, latency and reliability production requires. To reach a real decision, an AI POC needs three to five numeric success criteria agreed in writing before it starts, and a named authority who makes the go/no-go call. The 85% enterprise AI failure figure is real, but the usual reaction to it is wrong: the model is almost never where the failure starts.

What should an AI proof of concept prove before we commit to a full build?

An AI POC should prove that the core technical approach meets a success bar agreed with the sponsor on a fixed, representative test set, that its failure modes are understood and none of them blocks the use case, and that the cost per interaction is viable.

The POC should test the highest-risk assumptions:

1. The data exists in usable form. A feasibility check has to cover the availability, size and quality of the training data, because missing data is a major source of project delay and a small sample risks poor performance on data the model has never seen.
2. The approach clears the bar on representative data. A POC runs under conditions that represent production and ends in a go/no-go decision against criteria fixed in advance, which is what separates it from a sandbox experiment.
3. The failures are survivable. The evaluation should show which failure categories occur, and the gate fails if any one of them is a blocker (Cloud Soft Solutions gives inventing a policy that does not exist as an example).
4. The economics hold. Unit cost has to be viable to the sponsor before anyone talks about scale.

What the POC should not try to prove matters as much. Scalability, user adoption and production security belong to later stages. The POC should still write down the production constraints it was run against, because CRISP-ML(Q) treats robustness, scalability, explainability and resource demand in the target domain as requirements that later development and verification must check. Our view is that a POC which names these as open risks is honest, and one which ignores them is overclaiming its own scope.

On time-boxing, Microsoft’s engineering playbook puts feasibility studies at 4 to 16 weeks depending on the problem and the state of the data, and suggests starting with a 4-week milestone to decide how much more time is needed. Superkind recommends running the first four weeks in shadow mode, with the AI and the existing human process making the same decisions independently so their results can be compared without exposing the business to AI errors. If the data side looks shaky before any of this starts, our guide on how to evaluate GenAI use case feasibility before you build covers that earlier check.

What is the difference between a demo, a prototype and a POC, and why does each fail at a different stage?

Demos, POCs and pilots answer different questions.

	What it answers	What it produces	Where it fails
Demo	“Can AI do something impressive here?”	Stakeholder conviction	At the investment decision: it convinces people without giving them the information to decide
Proof of concept	“Can this approach meet the agreed quality bar on this data?”	A go/no-go against predefined criteria	When it has no measurable question: the results get read three different ways by data science, the business and the sponsor
Prototype or pilot	“Does it help real users doing real work?”	Evidence of acceptance, usage and workflow fit	At user acceptance: a model that passes every evaluation can still be hard to understand or miss corner cases users hit

Superkind and Cloud Soft Solutions do not use the words the same way. Superkind’s definition puts a PoC on a real business process with real data, which overlaps with what Cloud Soft Solutions calls a pilot, while Cloud Soft Solutions keeps the POC on a fixed sample, usually without real users. We would not argue over the labels. The useful discipline is to say, in the kickoff document, which question this stage answers and which it leaves for the next one. A demo dressed up as a POC is the expensive version of the confusion, because it creates confidence the evidence does not support.

How often AI POCs fail, and which scoping choices drive it

The figures people quote count different things. IDC research with Lenovo, reported by CIO.com, found that of every 33 AI POCs a company launched, only four reached production, which is where the 88% figure comes from. A 2026 paper by McClure and Gerdau cites S&P Global’s 451 Research finding that the share of companies abandoning most of their AI initiatives before production rose from 17% to 42% in a year, and BCG’s finding that only 26% of companies have moved beyond proof of concept to tangible value. Gartner predicted in July 2024 that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025. That last figure is a forecast, not a measurement.

Before repeating any of these figures, ask what it counts, on what unit, over what window, and whether it was measured or forecast; the often-quoted 80% counts opinions.

Four are worth checking before the POC starts:

• No defined question. Most AI POCs never reach production because nobody defined what was being tested, not because the algorithm failed. Our guide to AI POC requirements covers what to pin down first.
• No business case or funding. IDC’s Ashish Nadkarni describes POCs that are badly underfunded or not funded at all, often started without a strong business case.
• Low readiness in data, process and infrastructure. IDC reads the low conversion rate as a sign of low organisational readiness in exactly those areas, and Gartner names poor data quality, inadequate risk controls, escalating costs and unclear business value as reasons for abandonment. An enterprise AI readiness assessment belongs before the POC, not after it.
• Leadership and problem choice. RAND’s interviews with 65 data scientists and engineers identified five root causes: leadership-driven failures, data quality limits, chasing technology from the bottom up, underinvestment in deployment infrastructure, and applying AI to problems beyond the current state of the art. Only the last of these is a purely technical limit, and it is worth checking early whether you are facing an engineering task or a research question.

How do we write AI POC success criteria so evidence decides the go/no-go?

Write them before any tool is chosen, get every stakeholder to agree to them in writing, and make each one numeric, measurable within the POC window and tied to a business outcome the organisation already tracks. Superkind’s example shows the difference: “the AI performs well” is not a criterion, while “the AI resolves 70 percent of incoming exception emails without human intervention, with an error rate below 3 percent, within 4 minutes per item” is.

CRISP-ML(Q) adds a structural point. Success criteria should be set at three levels (business, machine learning and economic), each one measurable and each aligned with the others so the objectives do not contradict each other. The model’s training objective is only a proxy, and it has to be tuned in line with the business criteria. A criteria document with only an ML metric on it has skipped two of the three levels.

A checklist for one success criterion

A criterion is ready to sign when it passes every line:

• It is a number, with a minimum threshold (pass) and a stretch target (strong pass).
• Its measurement method is written down: which data source, which time period, and who pulls the number.
• It is measured against a baseline taken before the POC, for example human cycle time per item.
• The error rate is measured independently, not reported by the AI system itself.
• It is scored on a fixed, versioned test set kept separate from tuning examples, so a better score means a better system rather than easier questions.
• A further blind test set, disjoint from training and validation data and never shipped to any partner, is held back for the final evaluation.
• It names who makes the go/no-go call and what information they need to make it.

Two governance points stop the criteria from being renegotiated later. The vendor can configure and support the system, but it should not own the measurement or the decision, because a PoC where the vendor presents results with no independent internal measurement is a sales demonstration. At the end, the evaluation results are documented and compared against the business and ML criteria set at the start. How much tuning fits inside that window is a scoping question in its own right, covered in how to scope a hyperparameter sweep in an AI POC.

When does an AI POC need a clean kill criterion?

Always: no POC should start without written criteria and a named decision authority. Without numeric criteria agreed in advance, the end-of-POC review turns into a negotiation between optimists and pessimists, the POC runs another quarter, and the pilot becomes a permanent experiment. A stage-gate model makes each decision to invest further explicit, with three possible outcomes at each gate (proceed, loop back or stop), and stopping early is a legitimate, cheap result.

The kill conditions follow the assumptions the POC tests:

• At data feasibility: if the sample size is not enough, the project should be stopped or put on hold at that stage.
• At evaluation: if the success criteria are not met, the team backtracks to modelling or data preparation, or stops.
• On failure modes: any failure category that is a blocker fails the gate.
• On economics: a unit cost the sponsor will not accept fails the gate.

One kill condition comes before all of these. If stakeholders cannot agree on success criteria, Superkind’s point is that the disagreement is about business objectives rather than technology, and settling it is worth more than running the POC. We would also put a date next to each condition, because a threshold with no deadline can be renegotiated indefinitely.

A clean kill criterion also makes failure useful. When a POC misses, the diagnosis is whether the shortfall came from data quality (the process was not ready), scope (the wrong process was chosen), technology (the approach does not fit) or the criteria themselves (unrealistic targets for a first deployment), and each cause has a different fix. Only a POC with clear criteria produces a failure you can diagnose; one without them produces only ambiguity.

Measuring the return of an AI POC while it runs and after it ends

CRISP-ML(Q) makes economic success one of its three levels of success criteria, alongside business and ML criteria. While the POC runs, the primary business metrics decide the go/no-go, and operational metrics explain why the results look the way they do. Superkind lists automation rate, error rate on AI-handled items, cycle time against the pre-POC human baseline, and the human time spent on items the AI escalates.

For the investment case, Superkind compares the total cost of the POC (internal time, vendor fees, infrastructure, data preparation) with the projected annual value of a scaled deployment. In its example, an EUR 80,000 POC validates a deployment projected at EUR 400,000 a year. The projection is still a projection. CRISP-ML(Q) recommends evaluating the model under step-by-step closer-to-production conditions, and notes that this gives a first indication of whether the business and economic criteria could be met.

Two cautions apply. Gartner’s 2024 view is that generative AI needs more tolerance for indirect, future financial value than for immediate return, which many CFOs are uncomfortable with. IDC’s Nadkarni also says ROI calculations are being influenced by urgency, with people “willing to bend the rules on what ROI means”. Both point the same way: agree the ROI method with finance before the POC, so the result cannot be reinterpreted afterwards. After the POC, the test Cloud Soft Solutions applies before expansion is whether the ROI review shows a result the sponsor is willing to defend to their own leadership.

What the POC should leave behind if the project stops early

Each POC milestone should produce a usable result, even if the project never reaches its final goal. If all the value arrives at the very end, with a perfect final output, the project loses momentum and is likely to fail. That makes the artefact list part of the POC design rather than a by-product of it.

Milestone	Artefact	Why it still has value if the project stops
Scoping	Written problem statement, signed success criteria, named decision authority	Records what was asked and who decided, which any later attempt can reuse
Data feasibility	Data quality verification report, scope document, data dictionary or entity relationship diagram	Documents what data exists, how it joins and where the gaps are
Evaluation set-up	Versioned fixed test set plus a held-back blind test set	Lets the next approach or vendor be compared fairly on the same data
Shadow run	Side-by-side AI and human decisions, with the edge cases that came up	Gives measured error rates and process insight that were not visible in the initial mapping
Close	Feasibility report with a recommendation: the gaps, a narrower scope, a reframed problem or a data-collection plan	Turns a no-go into a plan rather than a dead end
Every gate	Decision log of questions, options, risks and rationale	Lets a later reviewer see the constraints that held when the decision was made

The commercial side of this is a matter of accountability. When an AI initiative fails, the blame does not stay with the model: it falls on the person who chose the vendor, signed the scope and defended the decisions internally. Milestone artefacts are how that person can show, at any review, what was known, what was scoped, what each milestone produced and how the decision was made. We hold this as a position, not a measured result: a POC with nothing to hand over gives the buyer nothing to defend.

Which evaluation evidence (data lineage, performance envelope, integration risk) must a POC hand to production engineering?

We group the evidence production engineering needs from the POC into three kinds.

Data lineage. This covers the data sources and how they join, the labels or proxy labels available, what data is in or out of scope, and access requirements such as retention and role-based access.

Performance envelope. This means the measured results against each criterion, the failure categories, and the conditions the model was tested under, along with the requirements still to verify: robustness, scalability, explainability and resource demand. The envelope should state its own limits, because production data may not match the training data and the hardware that collects it may be different.

Integration risk. This is the picture of the existing architecture, the results of any infrastructure spikes, and the risk assessment. Where evaluation exposed limits in robustness or explainability, CRISP-ML(Q) calls for an updated risk assessment such as a Failure Mode and Effects Analysis.

How does an AI POC connect to the downstream production engineering work?

A passed POC is one gate in a sequence, not permission to scale. In Cloud Soft Solutions’ stage-gate model, the POC is followed by a pilot, hardening, production launch and an operate-and-expand stage, with a measurable gate and a named artefact between each one. The pilot gate asks whether users come back without being chased and whether the metric moves against the baseline. Hardening asks for security sign-off, agreed SLOs, an exercised runbook and regression evaluations passing in CI.

CRISP-ML(Q) notes that once a PoC exists, a software project focused directly on deployment is more efficient. That deployment then needs an incremental rollout strategy, monitoring of usage and performance, and an automatic or human-controlled fallback to an earlier model. For LLM systems, the POC’s fixed, versioned test set is what lets later versions be compared fairly, which is the job of a task-specific evaluation harness for deployment decisions.

Production readiness is normally signed off by the customer: security approves the security review, operations accepts the SLOs and runbook, the sponsor makes the go or no-go call, and the engineering team supplies the evidence. Design the POC with that division of roles in mind, so the next stage is a handoff rather than a restart.