A team commits to a fixed budget and a delivery date for a latency target that, it turns out, is still an open topic at a peer-reviewed performance venue. The schedule was never wrong because of poor execution. It was wrong because the problem was misclassified. What was booked as engineering — build it, ship it, measure it against a milestone — was actually investigation, the kind of work where you don’t know in advance whether the method exists. The International Conference on Performance Engineering (ICPE) is one place that distinction becomes visible. Treated as background noise — a list of academic papers with no bearing on a delivery plan — it tells you nothing. Read as a boundary signal, it answers a question that determines whether your project can even have a deliverable: is the performance behaviour you need a solved engineering task, or is it still being argued about by people whose job is to argue about it rigorously? What the ICPE conference is, and why it matters for scoping ICPE is the ACM/SPEC International Conference on Performance Engineering — an annual, peer-reviewed venue where researchers and practitioners present work on the measurement, modelling, and evaluation of software and system performance. Historically it grew out of the workshop tradition around software performance and benchmarking, and it sits close to organisations like SPEC (the Standard Performance Evaluation Corporation) that maintain the benchmark suites many teams already rely on. The attendees are a mix of academics, benchmark authors, and performance engineers from industry — the people who define how you are supposed to measure a system before anyone claims it is fast. That composition is the useful part. A venue like ICPE is not a product catalogue and not a tutorial series. It is a running record of which performance problems are considered solved enough to standardise and which are still open enough to publish a paper about. When a behaviour you need is sitting in the “still open” column of that record, you have learned something concrete about your own project — regardless of what your architecture diagram implies about how routine the work is. The naive reading treats the conference proceedings as noise: interesting, perhaps, but irrelevant to a delivery plan. The expert reading treats the same proceedings as a state-of-the-art check. If the reliable method for the performance behaviour you’re depending on is still being debated at a peer-reviewed performance venue, it is not yet a solved engineering task you can put on a milestone schedule. That is the reframe this article is built around, and everything below is a way of making it operational. How a performance venue signals engineering versus research The mechanism is simpler than it sounds. Peer-reviewed venues publish novelty. A paper gets accepted because it does something that was not already known to work — a new modelling technique, a new way to characterise a workload, a result that overturns a prior assumption about how a system behaves under load. By construction, the presence of an active research thread on a topic means the topic is not settled. If it were settled, it would be a benchmark, a textbook chapter, or a library, not a paper. So the signal runs in one direction with high confidence and the other direction with less. If a required behaviour is the subject of current research at a venue like ICPE, treat it as unsettled — that inference is strong. The reverse — “it’s not at ICPE this year, therefore it’s solved” — is weaker, because absence has many causes. Something can be absent from a venue because it’s genuinely routine, or because it’s out of scope for that community, or because nobody happened to publish it this cycle. Use the presence of research as a stop signal; do not use its absence as a green light on its own. This is the same logic that governs per-use-case generative AI feasibility assessment: reading a research venue to check whether a required capability is solved mirrors asking, for a given GenAI use case, whether the capability is demonstrated or merely plausible. In both cases you are checking the state of the art before you commit a budget, not after. What does it mean if the behaviour I need is still an active research topic? It means your project has an investigation embedded in it that you have not scoped as one. Concretely: you cannot write an acceptance test for a method that does not yet reliably exist, so you cannot tell, from a milestone chart, whether you are making progress or burning time. The failure mode here is a project with no deliverable and no way to evaluate progress — the classic open-ended spend against a fixed timeline. That is not a resourcing problem you can fix by adding engineers. It is a classification error made at scoping time. The correct response is not to abandon the goal. It is to split the project along the boundary the venue just revealed: the parts that are settled engineering get milestones and fixed estimates; the part that is still an open question gets treated as a bounded investigation with explicit termination criteria — a time box, a decision gate, and a defined “we stop and reassess if X is not true by date D.” A rubric for classifying a required performance behaviour Use this before you commit a fixed budget and deadline. It is a diagnostic, not a scoring system — one clear “research” answer is enough to change how you scope that slice of work. Question Leans “engineering” Leans “research” Is there a named library, runtime, or benchmark that already does this at your scale? Yes — CUDA/TensorRT/SGLang path exists and is documented No named artifact reliably delivers it When you search a performance venue like ICPE, is the behaviour the subject of recent papers? No — it’s assumed, not studied Yes — it’s actively being modelled or measured Can you write an acceptance test today that says “done”? Yes — target latency/throughput is measurable and reproducible No — success criteria are themselves uncertain Do independent teams report reproducing the result? Yes — multiple public reproductions No — one result, unreplicated, or contradicted Does the estimate depend on a method existing, or only on integrating known methods? Integration of known methods A method has to be discovered or invented first If the answers cluster in the right-hand column, you are not scoping engineering — you are scoping a bounded investigation, and pretending otherwise is where the schedule quietly detaches from reality. This is the same reasoning we apply when reading the year’s performance-engineering research for AI readiness: the venue is an input, not a verdict. How this feeds an A5 Risk Assessment’s research boundary An A5 Risk Assessment names research boundary identification as a core output — the line separating deliverable engineering from work that requires investigation. Reading a venue like ICPE is one concrete way to test where that line falls for a specific required behaviour. It is not the only input; internal reproduction attempts, vendor documentation, and the presence of a maintained runtime all count. But a peer-reviewed performance venue has one property those other sources lack: it is adversarially reviewed for novelty, so it is a comparatively honest map of what the field itself considers unsettled. The payoff is not academic. Correct classification protects the buyer from open-ended spend. Engineering work gets milestones and a fixed estimate; research gets a bounded investigation with explicit termination criteria — a defined stop condition instead of an unmanaged burn against a fixed timeline. In our experience the expensive projects are rarely the ones that were correctly scoped as hard. They are the ones scoped as easy that turned out to contain an unacknowledged research question in the middle. This is an observed pattern across R&D engagements, not a benchmarked failure rate — but the shape recurs often enough that a state-of-the-art check before commitment pays for itself. If you want to see where this classification sits inside a broader engagement, the research-boundary and pivot logic of an intermediate checkpoint is the operational companion to this scoping check, and our R&D consulting services frame the whole lifecycle from state-of-the-art check to bounded investigation to delivery. When is it safe to scope work as deliverable engineering? It is safe when the method exists, is documented, has been reproduced by parties other than its authors, and can be validated against an acceptance test you can write today. A concrete example: serving a known open-weight model on a maintained runtime is engineering — you can profile it, tune it, and hit a latency target, because the machinery already exists and the only open question is how well your configuration performs. The MLOps reading of a performance conference makes the same distinction from the operations side: settled serving is a tuning problem; novel serving behaviour is a research problem wearing a tuning problem’s clothes. It is not safe when the estimate secretly depends on a method existing that has not been shown to work at your scale. That dependency is invisible on a Gantt chart and obvious in the proceedings of a venue whose entire purpose is to track what the field has and has not figured out. FAQ What should you know about the ICPE conference in practice? ICPE is the ACM/SPEC International Conference on Performance Engineering, an annual peer-reviewed venue for research on measuring, modelling, and evaluating software and system performance. In practice it functions as a running record of which performance problems are considered settled enough to standardise and which are still open enough to publish about. That record is what makes it useful as a state-of-the-art check before you commit a budget. What kind of work does ICPE (the International Conference on Performance Engineering) publish, and who attends it? It publishes peer-reviewed work on performance measurement, modelling, benchmarking, and evaluation — the discipline of establishing how a system behaves under load. Attendees are a mix of academics, benchmark authors, and industry performance engineers, and the venue sits close to standards bodies like SPEC. Because acceptance rewards novelty, the presence of a topic in the proceedings signals that the topic is not yet routine. How can a peer-reviewed performance venue signal whether an AI performance problem is an engineering task or an open research question? Peer-reviewed venues publish what is not already known to work. If a required performance behaviour is the subject of current research at a venue like ICPE, that is a strong signal it is unsettled and cannot be scheduled as routine engineering. The reverse inference is weaker: absence from the venue has many causes, so treat presence as a stop signal but never treat absence alone as a green light. What does it mean if the behaviour I need from my system is still an active research topic at a venue like ICPE? It means your project contains an investigation you have not scoped as one. You cannot write an acceptance test for a method that does not reliably exist yet, so a milestone chart cannot tell you whether you are progressing or burning time. The fix is to split the project: settled parts get milestones, and the open question gets a bounded investigation with an explicit stop condition. How do I use the state of the art at a venue like ICPE as an input to an A5 Risk Assessment’s research-boundary output? An A5 Risk Assessment names research boundary identification as a core output — the line between deliverable engineering and work requiring investigation. Reading a venue like ICPE is one concrete way to test where that line falls, because the venue is adversarially reviewed for novelty and therefore a comparatively honest map of what the field considers unsettled. Combine it with internal reproduction attempts and the presence of a maintained runtime. When is it safe to scope performance work as deliverable engineering versus a bounded investigation with termination criteria? It is safe as engineering when the method exists, is documented, has been reproduced by parties other than its authors, and can be validated against an acceptance test you can write today. It should be a bounded investigation when the estimate secretly depends on a method existing that has not been shown to work at your scale. The rubric earlier in this article is designed to surface exactly that dependency before you commit. If you take one habit from this, make it a small one: before you sign a fixed timeline, run the required performance behaviour past the state of the art and ask whether you are scheduling a build or a discovery. That single classification — settled engineering or open research question — is the input an A5 Risk Assessment turns into a defensible research boundary, and getting it wrong is the failure class that quietly converts a fixed budget into an unmanaged burn.
