The GenAI Divide: State of AI in Business 2025, Read Critically

The GenAI Divide: State of AI in Business 2025 behind the 95% GenAI pilot figure: what it measured, where it is misread, and what to settle before budget.

The GenAI Divide: State of AI in Business 2025, Read Critically
Written by TechnoLynx Published on 28 Jul 2025

MIT NANDA’s report The GenAI Divide: State of AI in Business 2025 is where the “95% of generative AI pilots are failing” line comes from, and it does not say what the headline says. By 80,000 Hours’ reading, the report found that 60% of surveyed organisations had looked at custom enterprise AI tools, 20% had piloted one, and 5% of the total had taken those tools into production successfully. It counted a tool as a success only if users or executives said it had a “marked and sustained productivity and/or P&L impact”, which the Marketing AI Institute says was measured six months after the pilot. So the 95% counts all surveyed organisations without a successful deployment, and 80% of them never piloted at all. It is a contested business-outcome figure from a small sample. It measures productivity or P&L impact, not technical failure of GenAI pilots.

That distinction matters if your own pilot has just stalled. The headline alone does not explain why it stalled. The report’s own findings, the critiques of it and some independent research can tell you a fair amount, provided you keep the general causes separate from the ones that only generative AI brings.

What The GenAI Divide: State of AI in Business 2025 actually counted

The report describes its evidence as interviews, surveys and an analysis of 300 public implementations. Fortune’s article, which launched the study into the mainstream, gave the sample as 150 interviews with leaders, a survey of 350 employees and 300 public deployments. A close reading by 80,000 Hours found that the paper itself rests on 52 interviews and a paper survey of 153 people. The Marketing AI Institute’s critique added that the “zero return” finding came from those 52 interviews, which the report itself calls “directionally accurate based on individual interviews rather than official company reporting”. The same critique notes that the report never explains how the 300 public initiatives were analysed or fed into the findings.

The report frames its headline this way: despite $30–40 billion of enterprise investment in GenAI, 95% of organisations are getting zero return, and just 5% of integrated AI pilots are extracting millions in value. Look at how those two clauses are worded. One counts organisations, the other integrated pilots, yet journalists reported that 95% of pilots fail.

Is it true that 95% of generative AI pilots at companies are failing?

No. That reading is wrong, and the funnel 80,000 Hours draws from the report shows why. On that reading, the 5% that succeeded is 5% of all surveyed organisations, and only 20% of those organisations ran a pilot. Among the companies that actually piloted a custom tool, about a quarter succeeded (5 ÷ 20 = 25%). 80,000 Hours points out that this is five times the rate that was reported, achieved against an extremely high bar set by the researchers.

Two further weaknesses limit how much weight the number can carry:

  • Sample size. With 52 interviews, the 5% figure probably rests on two or three companies, so the uncertainty around it is very wide.
  • Author interest. The authors are working on, or selling, the agentic AI frameworks the report recommends as the fix.

The figure also has company. Fortune noted that Capgemini found in 2023 that 88% of AI pilots failed to reach production. Gartner predicted in July 2024 that at least 30% of GenAI projects would be abandoned after proof of concept by the end of 2025. Those numbers measure different things (reaching production, abandonment after PoC, six-month P&L impact). Lining them up shows how much the answer depends on where you set the bar. It does not confirm any one of them.

A failed pilot versus a pilot with no measurable outcome yet

Under the report’s definition, a project that breaks even counts as a failure. So does a project that has helped the company without yet moving productivity or profit markedly, and so does a project on track to pay off next year. The Marketing AI Institute argues that the method leaves out efficiency gains, cost reductions, reduced churn, better lead conversion and faster pipelines. Gartner makes a related point: GenAI productivity gains are hard to turn directly into financial benefit, and GenAI needs more tolerance for indirect, future value than an immediate-ROI test allows.

Here, a pilot that failed technically means one that never produced reliable outputs on the real task. A pilot with no measurable outcome may work, but nobody defined what it should move, or it was judged before it had time to move it. The fixes are different. The first needs engineering. The second needs success criteria, and those should have been set before the pilot started.

What the headline implies What the report measured Source of the gap
95% of GenAI pilots fail 95% of surveyed organisations had no successful deployment, and 80% never piloted Organisations and pilots merged
Failure means the technology did not work Success meant marked, sustained P&L or productivity impact within about six months A business bar read as a technical one
Large, robust sample 52 interviews and 153 survey responses Larger sample reported by Fortune (150 interviews, 350 employees)
Pilots fail regardless of approach About a quarter of actual pilots succeeded Funnel denominator ignored

Which GenAI pilot failure causes are specific to generative AI?

Some of what goes wrong is ordinary AI project failure, and some comes from how generative models behave in workflows. RAND interviewed 65 experienced data scientists and engineers and identified five leading root causes of AI project failure. It found that misunderstanding the project’s intent and purpose is the most common cause, and that some projects fail because the problem is too hard for AI to solve. RAND studied AI projects as a whole, and its causes are covered on our page about the root causes behind most enterprise AI projects failing.

The NANDA report’s central finding concerns GenAI specifically. It says the core barrier to scaling is learning, not infrastructure, regulation or talent: most GenAI systems do not retain feedback, adapt to context or improve over time. It explains that chatbots succeed because they are easy to try and flexible, but fail in critical workflows because they lack memory and customisation. Users prefer ChatGPT for simple tasks and drop it for mission-critical work because it has no memory. For mission-critical work, 90% of users preferred humans. Fortune summarised the report’s argument as saying the problem was not the technology but how companies were using it.

Two more of its patterns concern GenAI budgets and deployments. About half of GenAI budgets go to sales and marketing, while back-office automation often returns more, and the report puts this bias down to easier metric attribution rather than actual value. External partnerships with learning-capable, customised tools reached deployment about 67% of the time, against about 33% for internal builds. The report itself warns that this may reflect the organisations’ own capabilities rather than the approach, and that correlation does not prove causation.

Cause General AI or GenAI-specific Evidence
Misunderstood project purpose General RAND
Problem too hard for AI General RAND
Tools that do not retain feedback or context GenAI-specific MIT NANDA
Chatbots lacking memory in critical workflows GenAI-specific MIT NANDA
Budget skewed to visible top-line uses Reported for GenAI budgets MIT NANDA
Data not ready for AI General Gartner
Multi-agent failure modes GenAI-specific Berkeley

The sections below cover the causes a team can act on before spending. For the scope, data, architecture and success-criteria patterns read against NANDA’s methodology in more depth, see GenAI-specific failure patterns beyond the headline number.

Why a pilot that worked on curated demo data stalls in production

A demo works on data someone selected. Production brings the data nobody selected, inside a workflow the demo never had to fit. NANDA says most GenAI tools fail because of brittle workflows, a lack of contextual learning and misalignment with day-to-day operations.

Gartner’s data work points the same way. In a July 2024 survey of 1,203 data management leaders, 63% of organisations either lacked the right data management practices for AI or were unsure whether they had them. Gartner predicts that through 2026 organisations will abandon 60% of AI projects unsupported by AI-ready data. Its guidance separates the dataset built for training from the live feed into production systems, and it recommends data observability to track how the data changes. A demo built on a one-off curated extract tests neither. On the infrastructure side, our guide to MLPerf Storage benchmarks explains what they measure and where GenAI data pipelines break.

The symptoms are recognisable. Hamel Husain describes an LLM product that improved fast with prompt engineering and then plateaued as its surface area grew. Fixing one failure mode produced others, nobody could see effectiveness across tasks beyond vibe checks, and the prompts grew long and unwieldy. That is what a pilot looks like when production variety reaches it with nothing in place to measure it. Our explainer on the data-centric approach covers why GenAI fails on production data, and the companion on fixing data-quality failures covers what that work involves.

Scoping a GenAI pilot to replace human judgement

Some pilots are set up to fail because the task is too difficult for AI. RAND states plainly that AI is not a magic wand, and that even the most advanced models cannot automate away some difficult tasks. NANDA’s own survey draws a clear line: 70% of respondents preferred AI for drafting emails and 65% for basic analysis, but for complex or long-term work humans dominated by 9-to-1 margins.

Our position is that the first question in any generative AI engagement is technical feasibility: whether the expected outcome is plausible at today’s capability. “We want an AI to be smarter than us” is a very different proposition from “we want AI to do this repeatable, menial task instead of us”, and the second is far more likely to show measurable return. A pilot that needs AI to be smarter than its experts falls on the wrong side of that line.

In our view, accountability sits with whoever approved that scope and accepted the data. We hold that clients should be told the failure risk of AI R&D at the start. Risky work can still go ahead, but the risk should be named before the budget is committed. When a pilot was never feasible, the scoping decision failed, not the model.

When multi-agent architecture is the failure, not the fix

Multi-agent designs bring failure modes of their own. A UC Berkeley team analysed five popular multi-agent frameworks across more than 150 tasks and identified 14 failure modes in three categories: specification and system design failures, inter-agent misalignment, and task verification and termination. They report that performance gains over single-agent frameworks remain minimal on popular benchmarks, and that the correctness of ChatDev, an open-source multi-agent system, can be as low as 25%. Their best-effort interventions on agent specification, conversation management and verification improved task completion by 14%, but did not fully fix the failures. The authors argue that many failures come from interactions between agents rather than from the limits of individual agents.

A budget-controlled study across the Qwen3, DeepSeek and Gemini model families found that single-agent systems matched or beat five multi-agent architectures on multi-hop reasoning once reasoning tokens were held constant. Its conclusion is that many reported multi-agent gains come from unaccounted computation and context effects rather than from the architecture. The study sets a fair boundary: multi-agent structure becomes competitive when a single agent’s effective use of its context degrades far enough.

Anthropic’s engineering guidance also favours simplicity. It recommends the simplest solution possible, notes that this may mean not building an agentic system at all, and says that for many applications optimising single LLM calls with retrieval and in-context examples is usually enough. It adds that frameworks can hide the underlying prompts and responses and make it tempting to add complexity a simpler setup would not need.

These findings sit uneasily with the NANDA report, which presents agentic AI with persistent memory as the answer to the learning gap. We give more weight to the two measured studies than to that recommendation, not least because 80,000 Hours says NANDA’s authors sell the remedy. In our judgement, a GenAI pilot should start with one model call and a retrieval step, and an orchestration layer should be added only when evaluation shows the simpler design falling short. For choosing that single model, our guide to 32B models explains when a 32B LLM fits a GenAI project.

Why so many GenAI pilots start without measurable success criteria

A fluent answer looks like a working system, and a vibe check feels like a test. Hamel Husain writes that unsuccessful AI products almost always share one root cause, a failure to build robust evaluation systems. His point is that teams focus on changing the system (prompts, fine-tuning, code) and skip evaluating quality and debugging issues, which keeps the product stuck at demo level. He argues that product evaluations have to be tailored to users’ tasks, and that they measure something different from general model benchmarks.

If a pilot never defined what it was meant to move, then under a test like NANDA’s it can only count as a failure, whatever it did. The Marketing AI Institute’s advice is to decide how success will be measured before launch, including efficiency and productivity measures, and it notes that a pilot’s success is rarely settled within six months. NANDA reports that buyers who succeed evaluate tools on business outcomes rather than software benchmarks.

Husain describes evaluation at three levels: unit tests, model and human evaluation, and A/B testing. Each level costs more than the one before, and he runs the costlier levels less often. He also advises checking how closely model-based evaluation agrees with human judgement before relying on the automatic version.

A pilot whose only value appears at the end, with a perfect final output, loses momentum. Every milestone should produce something usable even when the end goal is still out of reach. NANDA’s successful startups worked the same way, landing small, visible wins in narrow workflows and then expanding.

Decisions to settle before committing budget to a GenAI pilot

These are the decisions to settle before the first sprint.

  1. Feasibility at current capability. State the task in one sentence and ask whether it is a repeatable task done instead of a person, or judgement better than a person’s. If it is the second, rescope it or treat it openly as research; our piece on why projects fail before launch covers this scoping.
  2. Name the failure risk. Set honest expectations about the pilot’s failure risk before it starts.
  3. Data readiness on production data. Check that the data is AI-ready, and plan the live feed to production alongside the training set.
  4. Architecture matched to the problem. Start with single calls plus retrieval, and add agents only when the evaluation shows a measured shortfall.
  5. Success criteria before any build. Decide how success will be measured, including efficiency and productivity measures beyond direct P&L.
  6. Packageable intermediate value. Each milestone should leave behind something the business can use on its own.
  7. Build, buy or partner. Weigh NANDA’s 67% against 33% deployment finding for partnerships over internal builds alongside its warning that the gap may reflect the organisations, not the approach.

TechnoLynx starts generative AI engagements with this feasibility question: whether the outcome is plausible at current capability and where measurable value will appear first. For a team that wants that assessment done before committing spend, our AI engineering services cover the scoping and the engineering that follows.

What the headline figure cannot settle

The headline figure leaves open whether your pilot was set up to be judged at all. That means a feasible task, AI-ready data, the simplest architecture that works and a measure defined in advance.

Back See Blogs
arrow icon