Your inference endpoint is slow, or the monthly GPU bill is climbing faster than request volume. The first instinct on most teams is to look at the data path, because that is the part with the most instrumentation already wired up. The Airflow DAG runs green every night. The dbt models build. The Fivetran connector reports zero row-level errors. So the pipeline is healthy — and yet the p95 latency on the serving endpoint is still 900 milliseconds, and nobody can explain why.

This is the trap. An automated ETL tool tells you the data arrived on time. It does not tell you whether inference latency comes from the runtime, the batching policy, or the model itself. Acting on ETL job metrics alone leaves the actual inference bottleneck untouched — you tune the part you can see instead of the part that costs money.

The confusion is understandable. “Pipeline” is one word that covers two very different regions of a production AI system, and the boundary between them is exactly where most cost and latency debugging goes wrong.

What does an automated ETL tool actually do?

An automated ETL tool orchestrates the extract, transform, and load stages of a data pipeline. In practice, that means it schedules jobs, tracks their dependencies, moves data between systems, applies transformation logic, and reports whether each step succeeded or failed. Apache Airflow schedules and monitors DAGs of tasks. dbt handles the transform layer as versioned SQL. Fivetran and Airbyte manage the extract-and-load connectors that pull from source systems into a warehouse. Prefect and Dagster cover similar orchestration ground with different opinions about state and observability.

What these tools have in common is their unit of observation: the job. They instrument data movement, transformation steps, job scheduling, and pipeline-level success or failure. That instrumentation is genuinely useful — it is how you know a feature table refreshed before the model that reads it woke up, and how you catch a schema drift that would otherwise corrupt a batch of predictions.

But the unit of observation is also the boundary of the tool’s vision. An ETL orchestrator sees the world in terms of tasks that run and complete. When data crosses into the model-serving boundary, it stops being a task and starts being a request — and requests are not what these tools were built to watch.

Where does the ETL tool stop seeing?

Once data has landed and a request hits your inference endpoint, a different set of things determine latency and cost, and none of them are visible to the orchestrator that got the data there.

Consider what actually happens inside the serving boundary. A request arrives. The serving runtime — Triton Inference Server, vLLM, TorchServe, whatever you run — decides whether to batch it with other in-flight requests. That batching policy has a first-order effect on both throughput and tail latency. The batched tensor then executes on the GPU, where kernel time, memory-bandwidth pressure, and precision format govern how long the forward pass takes. The result comes back, and somewhere in there a per-request cost accrues that depends on how well the GPU was utilised during that window.

None of that shows up in an Airflow task duration or a dbt build log. The ETL tool’s job succeeded. It has no concept of batching behaviour, no view of GPU kernel time, and no way to compute cost-per-request. To see any of that you have to profile inside the serving boundary — which is a different discipline with different tools. We map that boundary explicitly in our note on reading a machine learning architecture diagram for the serving path, because the diagram is usually where teams first realise the two regions were never the same thing.

Here is the split in a form you can extract:

Concern	Automated ETL tool sees it?	Where it actually lives
Data landed on schedule	Yes	ETL orchestration
Transformation logic correct	Yes	ETL / transform layer
Job succeeded or failed	Yes	ETL orchestration
Schema drift on ingest	Yes	ETL / extract layer
Batching policy at inference	No	Serving runtime
GPU kernel time per request	No	Serving boundary / hardware
Cost-per-request	No	Serving boundary + unit economics
p95 / p99 tail latency under load	No	Serving runtime + hardware

The rule of thumb: everything above the double line is a job property; everything below is a request property. ETL tools are excellent at the first and blind to the second.

Why does a healthy ETL job not explain my inference latency?

Because a green DAG and a slow endpoint are answering different questions. The DAG answers “did the data arrive correctly and on time?” The endpoint latency answers “how long does the model take to serve a request, and how efficiently?” These can be completely independent. Your ETL can be flawless while your inference is slow, and — the part teams forget — your ETL can be a mess while your inference is fast, because the two live on opposite sides of a boundary that most monitoring dashboards blur together.

We see this pattern regularly in cost-audit engagements (observed across TechnoLynx work; not a benchmarked rate): a team spends weeks optimising warehouse queries and connector schedules to shave latency off an endpoint whose real bottleneck is a batching policy set to a static batch size of one. The data path was never the constraint. The rework was avoidable, and the signal that would have prevented it — GPU utilisation during request windows — was sitting one layer down, uninstrumented.

There is a genuine reason the misdiagnosis is common. ETL observability is mature and mostly free with the tools you already run; serving-boundary profiling is neither. So teams reach for the flashlight that is already switched on, even though it points the wrong way. The LLM orchestration frameworks note explains where drift enters the pipeline once you move past ingestion — a related boundary problem that has the same root cause.

How do ETL metrics relate to cost-per-request and p95 latency?

Mostly, they don’t translate at all — and that gap is the point. Job-success metrics are binary and coarse: the run took eleven minutes and moved four million rows. Cost-per-request and p95 latency are continuous, per-request, and load-dependent. There is no arithmetic that gets you from “the DAG finished in eleven minutes” to “each inference request costs $0.0004 and the 95th-percentile latency under peak load is 340 ms.”

To get cost-per-request you need the serving-side unit economics: how many requests a GPU-hour serves at your target latency, and what that GPU-hour costs. Our unit-economics framework for AI infrastructureInference Cost-Cut Pack defines the cost-per-request KPI precisely because ETL job-success metrics can’t be translated into inference spend. The framework starts where the orchestrator’s vision ends.

A worked example makes the disconnect concrete. Suppose your nightly feature pipeline is billed at roughly $40 of warehouse compute per run, and it runs once a day — call it $1,200 a month, a figure you can read straight off the ETL bill. Now suppose the same product serves two million inference requests a day on GPU instances. At even a modest cost-per-request, the serving side dwarfs the pipeline: for example, if each request cost $0.0003, that is $600 a day and roughly $18,000 a month — an order of magnitude more than the ETL bill, and entirely invisible to the tool that produced the $1,200 number. Optimising the $1,200 while the $18,000 goes unprofiled is the expensive version of looking under the streetlight. (Illustrative figures; substitute your own bill.)

When should I stop tuning ETL and move to GPU-level profiling?

The transition is a scoping decision, and it is worth making deliberately rather than by exhaustion. The following checklist is the one we apply before recommending any serving-path work:

• The ETL path is proven not to be the constraint. Feature tables refresh before the model reads them, with margin. If the endpoint is slow even when the data is hours old and fully warm, the data path is not your problem.
• Latency is per-request, not per-batch-job. If your pain is measured in the p95 of a live endpoint rather than the runtime of a scheduled job, you are already past the ETL boundary.
• Cost scales with request volume, not run count. If doubling traffic doubles the bill but the ETL schedule is unchanged, the cost lives in serving.
• The batching policy is unknown or unexamined. If nobody on the team can state the current batch size, max batch delay, and how they were chosen, there is almost certainly latency and cost sitting there.
• GPU utilisation during request windows is uninstrumented. If you cannot produce a number for how busy the GPU actually was while serving, you are flying blind on the layer that matters most.

If three or more of these hold, stop tuning the data path. The next move is to profile inside the serving boundary — and that is a distinct engineering skill. GPU profiling resolves the serving-path detail that ETL instrumentation can’t reachInference Cost-Cut Pack once data crosses into model serving, using tools like Nsight Systems, the runtime’s own metrics endpoint, and per-request cost accounting rather than DAG dashboards.

How does a cost audit complement the ETL tooling you already run?

It does not replace the orchestrator — it profiles the region the orchestrator was never designed to see. An AI Inference Cost Audit confirms the ETL path is not the constraint, then measures inside the serving boundary: p95 latency before and after, cost-per-request, and GPU utilisation. Those are the same measurable outcomes the audit produces, and they are precisely the three numbers your automated ETL tool cannot compute. The complement is clean: keep the ETL tool watching the jobs; add profiling to watch the requests.

FAQ

What’s worth understanding about automated ETL tools first?

An automated ETL tool orchestrates the extract, transform, and load stages of a data pipeline: it schedules jobs, tracks their dependencies, moves data between systems, applies transformation logic, and reports whether each step succeeded or failed. In practice its unit of observation is the job — tools like Airflow, dbt, Fivetran, and Dagster tell you the data arrived on time and correctly, which is genuinely useful but is only half of a production AI system.

What do automated ETL tools actually instrument in an AI inference pipeline, and where do they stop?

They instrument data movement, transformation steps, job scheduling, and pipeline-level success or failure. They stop at the model-serving boundary: once data becomes a live inference request rather than a scheduled task, batching behaviour, GPU kernel time, and cost-per-request are all invisible to the orchestrator. Everything that is a job property is visible; everything that is a request property is not.

Why does a healthy, on-time ETL job not explain my inference latency or cost?

Because a green DAG and a slow endpoint answer different questions — one confirms the data arrived correctly, the other measures how long and how efficiently the model serves a request. The two live on opposite sides of a boundary, so your ETL can be flawless while inference is slow. Tuning the data path when the bottleneck is a batching policy or GPU utilisation leaves the actual cost untouched.

How do ETL pipeline metrics relate to cost-per-request and p95 latency for an inference deployment?

They mostly don’t translate. Job-success metrics are coarse and binary (the run finished in eleven minutes, moved four million rows), while cost-per-request and p95 latency are continuous, per-request, and load-dependent. Getting to cost-per-request requires serving-side unit economics — how many requests a GPU-hour serves at your target latency and what that hour costs — which the orchestrator’s job metrics cannot produce.

When should I stop tuning the ETL path and move to GPU-level profiling of the serving boundary?

When the ETL path is proven not to be the constraint: features refresh with margin, the pain is per-request p95 rather than job runtime, cost scales with request volume not run count, and the batching policy and GPU utilisation are unexamined. If three or more of those hold, stop tuning the data path and profile inside the serving boundary instead.

How does an inference cost audit complement the automated ETL tooling I already run?

It does not replace the orchestrator — it profiles the region the orchestrator was never designed to see. The audit first confirms the ETL path is not the constraint, then measures inside the serving boundary: p95 latency before and after, cost-per-request, and GPU utilisation. Keep the ETL tool watching the jobs; add profiling to watch the requests.

What are examples of automated ETL tools, and where do they share the same model-serving blind spot?

Common examples include Apache Airflow and Dagster for orchestration, dbt for the transform layer, and Fivetran, Airbyte, or Prefect for extract-and-load and workflow management. They differ in their opinions about state and observability, but they share one blind spot: their unit of observation is the job, so none of them can see batching behaviour, GPU kernel time, or cost-per-request once data crosses into model serving.

What a defensible choice looks like here

The healthiest ETL pipeline in the world still cannot tell you why your endpoint is slow, because the answer lives one boundary further down than the orchestrator can look. That is not a flaw in the tool — it is a category difference between watching jobs and watching requests, and confusing the two is the most common way we see inference-cost debugging go sideways.

If you are standing up an inference pipeline and cannot yet state your batch size, your GPU utilisation under load, and your cost-per-request, the [inference-cost-cut-pack](Inference Cost-Cut Pack) and our broader services are built to profile exactly the region your automated ETL tooling was never meant to reach. Confirm the data path is not the constraint first — then put the effort where the cost actually lives.