Ask an analytics agent “what drove the revenue dip last quarter?” and you get a clean chart and a confident sentence. The temptation is to check whether the number reads plausibly and move on. That check tells you almost nothing about whether the agent picked the right table, applied the right filter, or would produce the same answer once the underlying data grew by a quarter’s worth of rows. That gap — between a plausible-looking answer and a verifiable path to it — is the whole story of evaluating AI agents for analytics. Teams searching for the term usually picture a chatbot bolted onto a BI tool: type a question in English, receive a chart. What they are actually buying is a multi-step system that plans, queries data, runs tools, and composes a result. Each of those steps is a place the reasoning can silently go wrong, and none of them is visible in the final chart. How do AI agents for analytics work in practice? Strip away the marketing and an analytics agent is a loop, not a single call. A large language model receives a natural-language question, decides what it needs to know, and then acts: it inspects a schema, drafts a SQL query, runs it against a warehouse, reads the result, and either answers or decides it needs another step. Frameworks like LangChain, LlamaIndex, and the various “text-to-SQL” agents built on top of them formalize this into a plan-act-observe cycle. The model is the reasoning engine; the tools — a SQL executor, a Python sandbox, a charting call — are how it touches the world. The important detail is that the model does not see your data. It sees a schema description, a few sample rows if you are lucky, and the results of queries it has already run. It is reasoning about your business from a keyhole view. When it chooses orders instead of orders_fact, or filters on created_at instead of closed_at, the mistake is invisible in the answer because the query still returns a number, and the number still renders as a chart. This is why the naive evaluation fails. Scoring the agent the way you would score a single-turn LLM — feed a prompt, check whether the output looks right — measures the one layer that is easiest to fake and skips every layer where the reasoning actually lives. A confidently wrong quarterly revenue figure that reached a board deck did not fail at the “does the sentence read well” layer. It failed three steps earlier, at a join the reviewer never saw. Why a multi-step agent is not a single-turn LLM answering a data question The difference is not academic, and it changes what “correct” even means. A single-turn LLM answering “what is 15% of 2.4 million” is right or wrong on one axis: the arithmetic. An analytics agent answering “how did enterprise churn trend last quarter?” can be wrong on a dozen axes while still producing a number that survives a plausibility check. Consider where a single answer can diverge from the truth: It queried the right table but joined on the wrong key, inflating counts through row duplication. It applied the correct filter but on a timezone-naive timestamp, quietly shifting the quarter boundary. It picked a defensible metric definition that happens to disagree with the one finance uses. It hit a query timeout, fell back to a cached partial result, and never flagged that the answer is incomplete. Every one of those produces a chart. None of them produces an error. This is the same reasoning that governs agentic AI benchmarks and when to trust them: the unit under test is a trajectory, and a trajectory has failure surfaces a final-output check cannot reach. In our experience across production AI reviews, the answers that erode trust fastest are not the obviously broken ones — those get caught. It is the plausible-but-wrong answers that flow into downstream reports and surface as an escalation weeks later, when someone reconciles two numbers that should have matched. Which evaluation-framework layers change for an analytics agent? If you already run task-specific LLM evaluations, you have the right skeleton. The layers do not get replaced; they get extended to cover a trajectory instead of a single response. The evaluation spec that links task, dataset, scoring, and run conditions is the same spec — but each layer now has to account for intermediate steps. What changes at each layer Layer Single-turn LLM eval Analytics agent eval Task definition One question → one expected answer One question → an expected trajectory (tables, filters, tools) plus the answer Dataset Prompt/answer pairs Questions paired with a known-correct query path and a schema snapshot the answer was valid against Scoring Output match / rubric on final text Per-step scoring: right table, right filter, right tool call — and final answer Run conditions Model, temperature, prompt All of that plus warehouse state, data volume, query timeouts, tool availability Evidence capture The final response Full step log: every query issued, every result observed, every decision made The row that most teams skip is dataset construction. To score a trajectory you need a ground-truth trajectory, which means someone who knows the warehouse has to author the correct query path for each evaluation question. That is more work than writing prompt/answer pairs, and it is exactly the work that determines whether the eval predicts production behaviour. This is the same discipline behind a well-built retrieval-augmented pipeline example, where knowing which document should have been retrieved is what separates a real eval from a vibe check. How do you score an agent’s trajectory instead of its final output? Trajectory scoring means grading the path, step by step, against a known-good path — not just comparing the endpoint. The measure that matters is trajectory-level accuracy: at each step, did the agent select the correct table, apply the correct filter, and invoke the correct tool? A run can land on the right final number by luck — two errors that cancel — and still be a broken trajectory you cannot trust on the next question. A practical scoring rubric for a text-to-SQL analytics agent looks like this (illustrative structure; the exact weights depend on your task): Table/source selection — did it query the intended tables? A wrong table is an automatic trajectory failure regardless of the final number. Filter and join correctness — right predicates, right join keys, right time boundaries. This is where silent duplication and timezone drift live. Metric definition — does the computed metric match the organization’s canonical definition, not just a defensible one? Tool sequencing — did it call tools in a valid order, and did it handle a failed or empty result rather than papering over it? Final answer — only meaningful once the four above pass. A correct answer on a broken path is scored as a near-miss, not a success. This is a benchmark-class measurement when the ground-truth trajectories are authored and the scoring is reproducible — it is a named test run against a fixed dataset, not a feel for whether the tool “seems good.” That distinction is what makes the result something a procurement decision can rest on. The broader question of how to compare agents against each other on a level field is the same problem framed for agentic benchmarks for tool-using LLM agents. Why can a plausible answer still fail a procurement-grade evaluation? Because procurement is not asking “did this one answer look right?” It is asking “will this agent produce trustworthy answers across the questions our analysts actually ask, under the data conditions we actually run?” Those are different questions, and the gap between them is where run conditions and dataset construction earn their keep. Run conditions decide whether eval results transfer to production. An agent evaluated against a 50,000-row sample warehouse can pass cleanly and then fail on a 50-million-row production warehouse — because a query that returned in two seconds in the eval now times out, and the agent’s timeout-handling path (untested, because it never fired) returns a partial result as if it were complete. Data volume, query latency, timeout behaviour, and tool availability are not incidental; they are part of the specification, in the same way serving configuration is part of any honest inference benchmark. Freeze them, or the number does not mean what you think it means. Dataset construction decides whether the eval covers the failure surface. If every evaluation question maps to a single well-behaved table, the eval never exercises the multi-table joins where trajectory errors concentrate. The dataset has to include the awkward questions — the ones that require a join across a fact and a dimension table, the ones with ambiguous metric definitions, the ones where the right answer is “the data doesn’t support this comparison.” An agent that produces a plausible chart for an unanswerable question is failing, and only a dataset built to include unanswerable questions will catch it. This layered posture — evidence over plausibility — is the same one behind measuring cost-per-task when one request fans out to many, where the fan-out itself is the thing under measurement. What evidence should the evaluation capture? Enough that a reviewer can reconstruct — and challenge — any specific answer after the fact. The final chart is the least useful artifact; the useful artifact is the full step log: every query the agent issued, every result it observed, every decision it made about what to do next. When a wrong number surfaces in a downstream report, the step log is what turns “the agent is unreliable” into “the agent joined on customer_id when it should have used account_id, here on step three.” This is where the evaluation stops being a quality gate and starts being an operational asset. A [production AI monitoring harness](Production AI Monitoring Harness) instantiates these layers against a candidate analytics agent, extending the scoring and run-conditions to cover multi-step trajectories rather than single-turn outputs — so the trajectory log that graded the agent in evaluation is the same artifact you inspect when a live answer is challenged. The same evidence discipline serves teams building agentic tooling on AI infrastructure and SaaS platforms, where the difference between a demo and a deployable feature is exactly whether you can defend a specific answer weeks later. FAQ What matters most about AI agents for analytics in practice? An analytics agent is a loop, not a single model call. It receives a natural-language question, plans what it needs, then queries data, runs tools, and composes a result — inspecting a schema, drafting and running SQL, reading the output, and deciding whether to answer or take another step. In practice it reasons about your business from a keyhole view: it sees a schema and prior query results, not your data, which is why a wrong table choice still returns a plausible number. How is a multi-step analytics agent different from a single-turn LLM answering a data question, and why does that matter for evaluation? A single-turn LLM is right or wrong on one axis; a multi-step agent can be wrong on many axes — wrong join key, wrong time boundary, wrong metric definition, silent partial result — while still returning a number that survives a plausibility check. That matters because scoring only the final answer measures the one layer easiest to fake and skips every layer where the reasoning actually lives. Which evaluation-framework layers change when the unit under test is an analytics agent? The layers are the same — task definition, dataset, scoring, run conditions, evidence capture — but each now accounts for intermediate steps. Task definition expects a trajectory not just an answer; the dataset pairs questions with a known-correct query path; scoring grades each step; run conditions include warehouse state and timeouts; evidence capture logs every query and decision. How do you score an agent’s trajectory rather than just its final output? You grade the path against a known-good path: correct table/source selection, correct filters and join keys, correct metric definition, correct tool sequencing, and only then the final answer. A correct answer reached on a broken path is scored as a near-miss, not a success, because it will not hold on the next question. How do run conditions and dataset construction determine whether eval results match production behaviour? Run conditions — data volume, query latency, timeout behaviour, tool availability — decide whether results transfer; an agent that passes on a 50,000-row sample can fail on a 50-million-row production warehouse where queries time out and untested fallback paths fire. Dataset construction decides coverage: unless the eval includes multi-table joins, ambiguous metrics, and unanswerable questions, it never exercises the surfaces where trajectory errors concentrate. Why can an analytics agent that produces a plausible-looking answer still fail a procurement-grade evaluation? Because procurement asks whether the agent produces trustworthy answers across real questions under real data conditions, not whether one answer looked right. A plausible chart for an unanswerable question, or a correct number reached through cancelling errors, is a failure a final-output check cannot see but a trajectory-and-run-conditions evaluation catches. What evidence should an analytics agent eval capture so a reviewer can trust — or challenge — a specific answer after the fact? The full step log: every query issued, every result observed, and every decision made about the next step. This is what turns “the agent is unreliable” into “it joined on customer_id when it should have used account_id on step three,” and it is the same artifact you inspect when a live answer is later challenged. Where this leaves a team evaluating agentic analytics tooling is a single question, and it is not “does the answer look right?” It is: can you reconstruct the path that produced any given number, score whether each step was correct, and show that the conditions you tested under match the ones you run under? The trajectory-level evidence that answers it is also what a governance-grade procurement evidence pack is built to structure — the difference between a tool your analysts trust and one they quietly stop believing.
