The ACM/SPEC International Conference on Performance Engineering — ICPE 2025 — reads, on first glance, like an academic event with nothing to say to a team still fighting to get one model out of a notebook. That read is wrong, and the gap it hides is expensive. The performance-engineering disciplines ICPE tracks — benchmarking, load characterisation, latency and throughput modelling — are precisely the ones a first MLOps deployment skips, and then pays for in production when the serving path buckles or the inference bill triples without explanation.

The point of this article is not to summarise a conference programme. It is to translate the threads at ICPE that actually touch inference serving into a short, honest list of measurements a first-deployment team can instrument without a mature data-engineering function behind them. A large organisation can chase every performance frontier the conference names. A team operationalising its first model does not need that. It needs the two or three numbers that keep the pipeline from lying to itself.

What is ICPE, and why should an MLOps team care?

ICPE is the ACM/SPEC International Conference on Performance Engineering — the venue where the performance-measurement community publishes on how software systems behave under load, how to benchmark them fairly, and how to model latency and throughput before a system reaches production. Historically its centre of gravity has been traditional application and systems performance: web services, databases, distributed middleware, cloud auto-scaling. The methods are mature and well-argued because that community has been arguing about them for decades.

The reason an MLOps team should care is not that ICPE is an “AI conference” — it mostly is not. It is that model serving is an application-performance problem wearing a machine-learning hat. An inference endpoint has a request queue, a tail-latency distribution, a resource-saturation curve, and a cost-per-unit-work, exactly like any other service. The vocabulary the performance-engineering field uses for those things — p95, saturation, load characterisation — is the vocabulary an MLOps team ends up reinventing, badly, when a deployment starts missing its latency target and nobody instrumented the thing that would explain why.

We see this pattern regularly. A team ships a model that passed every offline accuracy check, and then discovers in production that the accuracy was never the risk — the serving path was. The methods to catch that ahead of time are not new. They are the ones ICPE has been refining on non-ML systems all along.

Which ICPE 2025 themes apply to a first deployment — and which don’t?

The trap is treating the whole conference as relevant. Most of it is not, for a team at first deployment. The skill is separating the frontier work (for mature systems with dedicated performance teams) from the foundational measurement discipline (which every serving path needs on day one).

Here is the split we apply when advising a first-deployment team.

ICPE theme (as it maps to serving)	First deployment?	Why
Latency distribution & tail modelling (p95/p99, not mean)	Yes — day one	Mean latency hides the requests that break the SLA; the tail is where users and timeouts live
Load characterisation & saturation curves	Yes — day one	Tells you the request rate at which the endpoint degrades; needed to size a single node honestly
Cost-per-unit-work benchmarking	Yes — day one	Inference cost is the recurring bill; measuring cost-per-1k-predictions early exposes regressions cheaply
Auto-scaling policy optimisation	Later	Requires stable traffic patterns and a scaling substrate a first stack usually lacks
Micro-architectural / kernel-level profiling	Later	Real value only once the coarse serving-path measurements are stable
Formal performance modelling & prediction	Later / mature only	Pays off at scale; overkill when a load test answers the same question directly
Reproducibility & benchmark-fairness methodology	Partial	Adopt the discipline (fixed inputs, warm-up, repeated runs); skip the formal apparatus

(Evidence class for the “first deployment?” column: observed pattern across TechnoLynx MLOps engagements — a scoping heuristic, not a benchmarked cut-off.)

The line is scope, not sophistication. A first deployment needs the measurements that keep the pipeline honest. The frontier techniques buy real value later, once the coarse numbers are stable and someone owns them.

How do you translate ICPE-style benchmarking into serving-path metrics?

The translation from performance-engineering method to a minimal MLOps stack is more concrete than it sounds. Three measurements carry most of the weight, and none of them requires a data-engineering platform to capture.

p95 inference latency. The single most important lesson the performance-engineering field enforces is that the mean is a liar. If your endpoint serves a request in 40 ms on average but the 95th percentile is 900 ms, one in twenty users is having a bad time and your SLA is already broken — the mean will never show it. Capture the full latency distribution at the serving layer, not an average in a log line. Frameworks make this cheap: a request-timing middleware in front of a TorchServe or Triton Inference Server endpoint, exported to Prometheus, gives you a p95 histogram with no bespoke tooling.

Cost-per-1k-predictions. Inference cost is the recurring line item that quietly grows. Anchor it to work done, not to wall-clock. Cost-per-1k-predictions — GPU-hour or instance-hour price divided by throughput at your target latency — is a benchmark-class number you can compute the day the model ships, and it is the one that catches a regression when a model update doubles compute per request without anyone noticing.

Retraining trigger frequency. Drift-triggered retraining is a load source most first deployments forget to characterise. If your monitoring fires a retrain every few days because the drift threshold is set too tight, you are paying training cost on a schedule nobody budgeted. Counting how often the trigger fires — and against what threshold — turns an invisible cost into a dial you can turn.

These are exactly the kind of load-characterisation numbers ICPE’s methodology tracks, reduced to what a first stack can carry. The discipline that matters is the one the performance-engineering field is strict about: fix your inputs, warm up before you measure, and repeat the run. A single cold-start timing is noise, not a benchmark.

Where in the pipeline do these measurements belong?

The measurements are worthless if they live in a notebook nobody reruns. They have to land at the three points where a first MLOps pipeline can act on them.

• CI/CD gate. p95 latency and cost-per-1k-predictions belong as a gate in the deployment pipeline. A model that regresses either past a threshold should fail the build, not surprise you in production. This is the cheapest place to catch a serving-path regression, because it is caught before it ships.
• Monitoring dashboard. The live p95 latency and cost curves belong on the deployment’s monitoring dashboard. This is the evidence trail — the artifact that shows the deployment is meeting its latency and cost targets over time, not just at release. It is where the ICPE-style measurements become an operational record rather than a one-off test.
• Retraining trigger. Trigger frequency is monitored where drift detection lives, so the retraining cadence is a measured quantity rather than an assumption.

Landing the numbers at these three points is what makes them repeatable infrastructure instead of a heroic one-time analysis. It is also what makes the second deployment cheaper: the gate and dashboard you built once become the template you reuse, and inference latency and cost regressions get caught before they reach production instead of after. That reuse is the return on even a minimal performance-engineering investment.

If you want to see how this fits into a broader engagement — where the serving-path instrumentation sits alongside monitoring dashboards and CI/CD as the deliverables of a first MLOps build — our AI engineering and consulting services frame it as the difference between a demo and a system you can run twice.

How does inference-serving performance differ from the application performance ICPE has historically covered?

This is where the naive borrowing breaks. You cannot lift traditional application-performance assumptions onto model serving unchanged, and knowing where the analogy fails is as important as knowing where it holds.

Three differences matter most. First, the workload is not uniform: an LLM serving path has wildly variable input and output lengths, so a request’s cost depends on its content in a way a static web endpoint’s does not — which is why context window length feeds directly into inference cost. Second, the accuracy dimension has no analogue in traditional performance engineering: a serving optimisation that quietly degrades output quality is a failure the p95 latency chart will never show, so quality has to be measured alongside performance. Third, the resource profile is memory-bound more often than compute-bound — inference is frequently limited by memory bandwidth, not FLOPS, which inverts some of the classic scaling intuition.

The performance-engineering method transfers cleanly. The performance-engineering assumptions do not. That distinction is the whole value of reading a conference like ICPE with a serving lens rather than copying its case studies wholesale. The ICPE track that most directly addresses production ML is worth watching for exactly this reason — and the 2026 programme’s AI-readiness angle continues the thread for teams tracking where the research is heading.

FAQ

What matters most about ICPE 2025 in practice?

ICPE 2025 is the ACM/SPEC International Conference on Performance Engineering — a research venue where the software-performance community publishes on benchmarking, load characterisation, and latency/throughput modelling. In practice, for an MLOps team, it is a source of measurement methods rather than an event to attend: the disciplines it tracks map directly onto the serving-path metrics a first model deployment needs to instrument.

What is ICPE, and why should an MLOps team care about it?

ICPE is the international conference on performance engineering, historically focused on traditional application and systems performance. An MLOps team should care because model serving is an application-performance problem — request queues, tail latency, saturation, cost-per-unit-work — and ICPE’s vocabulary and methods are exactly what teams end up reinventing when a deployment misses its latency or cost targets.

Which ICPE 2025 themes actually apply to a first model deployment?

Latency-distribution modelling (p95/p99, not mean), load characterisation and saturation curves, and cost-per-unit-work benchmarking apply from day one. Auto-scaling optimisation, micro-architectural profiling, and formal performance modelling are for mature systems with dedicated performance teams. The dividing line is scope, not sophistication.

How do you translate ICPE-style benchmarking into concrete serving-path metrics?

Reduce it to three numbers a minimal stack can capture: p95 inference latency (the full distribution, not an average), cost-per-1k-predictions (instance-hour price over throughput at target latency), and retraining trigger frequency. Apply the field’s measurement discipline — fixed inputs, warm-up, repeated runs — so a single cold-start timing is never mistaken for a benchmark.

Where in a first MLOps pipeline do performance measurements belong?

p95 latency and cost-per-1k-predictions belong as a CI/CD gate that fails a regressing build; the live curves belong on the monitoring dashboard as the evidence trail that the deployment meets its targets; and retraining trigger frequency belongs where drift detection lives. Landing them at these three points turns them into repeatable infrastructure.

Should a team invest in performance engineering before or after their first model reaches production?

Before — but only the minimal shortlist. Instrumenting p95 latency, cost-per-prediction, and retraining frequency as a CI/CD gate and dashboard before production is cheap and catches regressions early; the return shows up on the second deployment, which reuses the same instrumentation. The frontier techniques can wait until the coarse numbers are stable.

How does inference-serving performance differ from traditional application performance engineering?

Three ways: the workload is non-uniform (request cost depends on input/output length), there is an accuracy dimension with no analogue in traditional performance work (an optimisation can silently degrade output quality), and the resource profile is often memory-bandwidth-bound rather than compute-bound. The ICPE method transfers; the assumptions behind its historical case studies do not.

The honest question to close on is not “should we track performance-engineering research” — it is which two or three serving-path measurements you can instrument this quarter, and whether they land on a dashboard that catches the regression before your users do.