The evaluation problem Choosing an AI consulting firm is a decision made with significant information asymmetry. The buyer typically has less technical AI expertise than the seller — that is why they are buying consulting in the first place — which means the buyer cannot independently verify the seller’s technical claims. The result is predictable: purchasing decisions get anchored to signals that correlate weakly with delivery quality. Brand recognition. Partnership badges. Slide deck polish. The number of logos on the “clients” page. The firms that deliver are not always the ones that present best. And the firms that present best are not always the ones that deliver. The evaluation needs structure — a set of criteria a non-technical buyer can assess that actually correlate with delivery outcomes. What follows is the rubric we recommend when evaluating any AI consulting engagement, whether the candidate is a boutique, a Big Four practice, or a regional integrator. The framing matters. Most procurement processes for AI work were inherited from staff augmentation contracts, where the firm rents engineers and the buyer owns the direction. That structure absorbs technical risk into the buyer’s organisation. If you are not equipped to direct the work — and most enterprises buying AI consulting are not — the firm needs to own the outcome, not just the hours. The five criteria below are designed to surface which model you are actually buying. How do you assess technical depth without being technical yourself? Every AI consulting firm claims expertise in machine learning, deep learning, NLP, computer vision, generative AI, and MLOps. The claims are not differentiating because everyone makes them. What differentiates is the depth behind the claim. Ask the firm to describe a specific technical decision they made on a recent project and why they made it. Not “we built a computer vision model” — but something concrete: “we chose a YOLOv8 architecture over Faster R-CNN because the client’s latency requirement was 40ms per frame on a Jetson Orin, and YOLOv8-nano hit that budget at INT8 quantisation while Faster R-CNN exceeded 80ms even after kernel-level optimisation through TensorRT.” The specificity of the answer reveals whether the team has hands-on implementation experience or whether they are reselling subcontracted work under a slide deck overlay. Red flag. The firm cannot name the specific architectures, tools, or runtime targets used on their projects. Answers stay at the level of “we used advanced machine learning techniques.” When you ask about deployment, no one mentions ONNX, TensorRT, Triton, or CUDA versions. Green flag. The technical team discusses trade-offs unprompted — why they chose one approach over another, what alternatives they considered, where the chosen approach has limits, and what they would do differently next time. PyTorch versus JAX comes up. Quantisation strategy comes up. Inference batching, KV cache layout, attention kernel choice — these are not exotic topics for a team that has shipped production systems. What sophisticated buyers systematically miss. Depth signals are easy to rehearse. We have seen firms deliver impressive technical walk-throughs of their best project during evaluation, then staff the actual engagement with junior engineers who had no involvement in that project. The anti-gaming check is not just “can they describe technical depth?” but “can the specific people proposed for your project describe it on demand, unrehearsed, for their own recent work?” Criterion 2: Delivery evidence, not capability claims A firm’s capability deck describes what they can do. Delivery evidence shows what they have done. The gap between the two is often substantial. Request case studies that include specific, measurable outcomes — not “we improved accuracy” but, as an illustrative example of operational measurement from a deployment (an observed-pattern from named work, not a benchmarked industry rate), “we reduced false-positive rates from 12% to 3.2% on the client’s production defect detection system, measured over 90 days of production operation.” Then request references from clients who can speak to the delivery experience, not just the headline outcome. Was the team responsive? Did they meet timelines? Did they communicate problems early, or were surprises sprung at milestone reviews? Red flag. All case studies describe pilot projects and proofs of concept. None describe production deployments that operated for months or years. This pattern is consistent with a firm that is good at demos but has not solved the production engineering problems — monitoring, retraining, drift detection, infrastructure cost management. Green flag. Case studies describe production systems with operational metrics — uptime, accuracy over time, maintenance burden — and explicitly cover the transition from pilot to production. The work is still running, not sitting in a report. The same patterns show up in our own assessments of why most enterprise AI projects fail: the firms that have wrestled with productionisation talk differently about it than the firms that have only built POCs. Criterion 3: Knowledge transfer, not dependency creation An AI consulting firm that delivers a model but does not transfer the knowledge to operate and maintain it has created a dependency. The client must return to the firm for every update, every retraining cycle, every debugging session. This dependency is profitable for the firm and expensive for the client. Ask what the firm’s delivery includes beyond the model itself. Documentation — architecture decisions, training procedures, evaluation criteria, monitoring setup? Training for the client’s team — how to retrain, how to evaluate, how to debug? Does the delivery include the complete codebase with clear documentation, or is it a deployed model with opaque configuration the client cannot inspect? Red flag. The firm’s engagement model is ongoing managed service with no defined path for the client to take over operation. The “deliverable” is access to a running system, not the system itself. When you ask about disengagement, the answer is vague. Green flag. The firm explicitly plans for disengagement. The engagement includes knowledge transfer milestones, the client’s team is involved in development from the start, and the firm’s stated goal is to make itself unnecessary for ongoing operations. This is how engagements scoped to your problem should work — long-term client dependency is not a sustainable model for either party, and a firm that builds its book of business on locked-in clients eventually loses the ones who notice. Criterion 4: Honest scoping, not optimistic estimation The proposal should reflect realistic effort estimates based on the project’s actual complexity, data readiness, and integration requirements. A proposal that is significantly cheaper or faster than competitors is usually underestimating something — and the project will either blow past the estimate or deliver a cut-scope version that does not meet the original requirements. Compare the proposal against the predictable failure patterns: does it include data readiness assessment, clear success criteria, integration scoping, and risk identification? Or does it jump directly to model development without addressing the prerequisites? Red flag. The proposal does not mention data assessment, does not define success criteria, and estimates a problem that obviously needs data engineering, model development, integration, and production deployment at 6–8 weeks. Either the firm plans to deliver a POC and call it done, or the estimate is unrealistic. Both outcomes hurt the buyer. Green flag. The proposal includes a paid scoping phase before committing to the full project, identifies specific risks and mitigation strategies, and provides a range of effort estimates with the factors that determine where in the range the project will land. Honest scoping looks more expensive on paper and is almost always cheaper in execution. Criterion 5: Team composition, not firm size The quality of the engagement depends on the people who do the work, not the firm’s headcount. A 500-person firm that assigns junior consultants will deliver worse results than a 20-person firm that assigns senior engineers with relevant domain experience. Ask who specifically will work on the project. Request CVs or profiles. Ask about their relevant project experience — not in general, but on projects similar to yours: same industry, same technical approach, same scale. Ask whether the proposed team will remain assigned for the project’s duration, or whether members may be rotated to other accounts mid-engagement. Red flag. The firm cannot name the specific people who will work on the project until after the contract is signed. The proposal showcases senior people who disappear after the kick-off meeting and are replaced by less experienced staff for the actual delivery. Green flag. The proposed team is named, their relevant experience is documented, and the firm commits to team continuity for the engagement duration in writing. Weighted scoring rubric with anti-gaming checks Not all criteria matter equally. Technical depth and delivery evidence carry more weight because they are harder to fake and correlate most strongly with project outcomes. Use the rubric below to score each firm on a 1–5 scale per criterion, then multiply by the weight to get a weighted score. Criterion Weight Score 1 Score 3 Score 5 Anti-gaming check Technical depth 3 Buzzword-level answers (“advanced ML techniques”) with no architecture or trade-off detail Names specific tools and architectures but cannot explain why they were chosen over alternatives Describes architecture decisions, quantified trade-offs, and limitations on a recent project unprompted Ask the proposed team to walk through a real technical decision live — not from slides. Probe with “why not X?” follow-ups to test whether depth is rehearsed or genuine Delivery evidence 3 Only capability decks and pilot-stage case studies; no production metrics Production case studies exist but metrics are vague (“improved accuracy”) or unverified Case studies include quantified production outcomes (e.g., “false-positive rate from 12% to 3.2% over 90 days”) with referenceable clients Request a reference call with a client whose project is still in production. Ask whether the system is still running and what maintenance looks like Knowledge transfer 2 Deliverable is access to a running system with no documentation, code, or training plan Documentation and code are included but no structured training or disengagement plan Engagement includes architecture docs, retraining procedures, client team training, and explicit disengagement milestones Ask to see a sample deliverable package from a past engagement. Verify it includes runnable code, not just a deployed endpoint Scoping honesty 2 Proposal jumps to model development with no data assessment, no success criteria, and an unrealistically short timeline Proposal mentions data readiness and success criteria but does not include a scoping phase or risk identification Proposal includes a paid scoping phase, named risks with mitigations, and effort ranges tied to specific contingencies Compare the timeline against at least two other firms. If one estimate is half the others, ask what is excluded — data engineering, integration, or production deployment Team composition 2 Firm cannot name who will work on the project; team is “to be assigned” Team is named but relevant experience is generic or unverifiable Named individuals with documented experience on similar projects (same domain, scale, technical approach), with a continuity commitment Ask for named individuals, not roles. Request profiles. Confirm in writing that the same people will remain through delivery Score each firm 1–5 per criterion, multiply by the weight, and sum. Maximum possible score is 60. As a planning heuristic from our consulting engagements (an observed pattern, not a benchmarked industry rate), a firm scoring below 36 — 60% of maximum — has gaps significant enough that they should be addressed before contracting. Pay particular attention to any criterion where the anti-gaming check exposes a discrepancy between the firm’s claims and verifiable evidence. The total score matters more than any single criterion, and the process itself forces the evaluation to become evidence-based rather than impression-based. FAQ What should I look for when evaluating AI consulting firms, and what should I screen out? Look for technical depth that survives unrehearsed probing, delivery evidence from production systems (not POCs), explicit knowledge transfer plans, honest scoping with a paid discovery phase, and a named team with relevant experience and continuity commitments. Screen out firms whose differentiation rests on brand recognition, partnership badges, or logo walls without specific production case studies, and firms that cannot name the individuals who will actually do the work. How do boutique AI consultants differ from Big Four consulting firms in scope, methodology, and accountability? Boutiques typically staff engagements with the same senior engineers throughout delivery, own the technical outcome end-to-end, and have less internal handoff between sales and delivery. Big Four practices have broader change-management and procurement integration but often staff with junior consultants after a senior-led pitch, and the accountability sits at the firm level rather than with named individuals. Neither model is universally better — the right choice depends on whether you need outcome ownership on a defined technical problem (boutique) or organisational transformation alongside the build (Big Four). Which evidence genuinely separates capable firms from rebranded ones? Production case studies with quantified operational metrics, referenceable clients whose systems are still running, named team members who can discuss technical trade-offs unprompted on their own recent work, and sample deliverable packages that include runnable code and documentation rather than just deployed endpoints. Capability decks, partnership badges, and pilot-only references do not separate capable from rebranded firms. How much does an AI consultant cost, and what determines the price band for a serious engagement? Serious AI engagements price against scope, data readiness, integration complexity, and the seniority of the delivery team — not against hourly rates in isolation. A proposal that is significantly cheaper than the field is almost always excluding something material: data engineering, integration work, or production deployment. The price band is set by what the firm is actually delivering: a POC, a pilot, or a production system with knowledge transfer. Which contractual structures protect the buyer in AI work? Outcome-based and milestone-gated structures with explicit pivot points protect the buyer when the firm has the expertise to own the result. Time-and-materials shifts technical risk onto the buyer, which only works if the buyer’s team can direct the work credibly. Fixed-scope contracts work for well-defined problems but break down when the data turns out to be messier than the proposal assumed — which is most of the time in AI work. A paid scoping phase before the main contract is the single most useful protection: it lets both sides re-price honestly once the real problem is visible. How do I evaluate a consulting firm’s ability to hand off to my internal team rather than create dependency? Ask for the disengagement plan in writing before signing. A capable firm will describe knowledge transfer milestones, name the artifacts they will deliver (architecture docs, retraining procedures, monitoring runbooks, complete codebase), and specify how your team will be involved in development from the start rather than receiving a finished black box. If the engagement model is open-ended managed service with no defined handover, you are buying dependency, not capability transfer. Ask your consulting partner for a risk-structured engagement plan with named pivot points, a paid scoping phase, and an explicit disengagement path. If they cannot produce one, that is your answer.
