An autonomous coding agent takes a whole development task rather than a line of code. It sets up the environment, drafts a plan, edits files, runs the tests and opens a pull request, where a traditional assistant completes snippets as the developer types. The evidence on what happens to those pull requests tells a more sober story than the autonomous AI software engineering image suggests: agents get merged less often than human contributions, and they do best on narrow, well-defined work. More autonomy is a design choice with costs.
From autocomplete to agent: what changes when a model gets tools
A copilot, in the sense used by the 2025 generative AI roadmap for software engineering, is a passive component that supports engineers in their tasks, while a GenAI teammate proactively takes part in development, sharing goals and responsibilities instead of waiting to be invoked. Anthropic’s documentation for Claude Code describes three blended phases (gather context, take action, verify results), with the agent deciding each step from what the previous one returned and chaining dozens of actions together.
Tools are what make Claude Code agentic. Without them the model can only answer in text; with them it reads code, edits files, runs commands and feeds each result back into its next decision. Asked to “fix the failing tests”, such an agent might run the suite, read the errors, search for the relevant source, edit it and rerun the tests to check its work. Because it sees the whole project, it can make coordinated edits across several files, which an inline assistant limited to the current file cannot.
Autocomplete, chat assistant and coding agent compared
| Autocomplete | Chat assistant | Coding agent | |
|---|---|---|---|
| Triggered by | The developer’s keystrokes | A prompt from the user | A task, then its own sequence of steps |
| Sees | The current file | The prompt it is given | The repository, through search and file reads |
| Acts | Inserts a suggestion | Returns text only | Edits files, runs commands and tests, can open a PR |
Chat assistants versus agents: where the line actually sits
Is ChatGPT generative AI or agentic AI, and why does it matter for scoping?
Used as a chat window, an assistant plays a passive role: it responds to the user’s prompt and does not act on its own. The 2026 empirical study of agentic pull requests draws the same line, describing agentic coding tools as moving beyond LLM chatbots such as ChatGPT by autonomously handling more complex, integrated development tasks. An agent plans and executes the development actions itself, from code generation to preparing a pull request. Its output still needs human review for alignment with broader system requirements.
Is Claude (or other LLM chat assistants) considered agentic AI, or does agentic behaviour only emerge when the model is wired into a tool-using orchestration loop?
The model alone is not the agent: agency appears when the model is wired into a loop with tools whose outputs feed its next decision. Anthropic’s own usage data compares the chat product with the agent. In its analysis of 500,000 coding interactions, 79% of Claude Code conversations were classified as automation against 49% on Claude.ai, and “directive” conversations with minimal user interaction were 43.8% on Claude Code versus 27.5% on the chat product.
What is agentic AI, and how is it engineering-distinct from generative AI?
Anthropic separates workflows, where LLMs and tools follow predefined code paths, from agents, where the LLM directs its own process and tool use. The engineering consequence is that an agent’s number of steps cannot be predicted in advance, bringing higher costs and errors that can compound. The broader comparison, outside coding, is covered in our explainer on what agentic AI is and how it differs from generative AI.
Concrete examples of agentic versus generative AI in a development workflow
Asking a chat model to explain a stack trace is a generative use; asking Claude Code to fix the failing tests, where it runs the suite, edits the source and reruns it, is an agentic one. Agents go further than writing documentation text: they independently create pull requests, generate documentation and respond to review feedback.
How do agentic, generative and predictive AI fit into one architecture without overlapping?
We would treat them as layers with separate jobs: a predictive model produces a score or forecast, a generative model produces content, and an agent decides which of those to call and what to do with the result. They stop overlapping when each component has one contract: the agent orchestrates, and it should not be quietly doing the predictive model’s job in a prompt. The sources behind this page do not cover predictive or physical AI, so this is judgement rather than finding.
How does agentic AI relate to predictive and physical AI when scoping a project?
Physical AI adds actuation in the real world, which raises the stakes of any autonomous decision and pushes the design towards lower autonomy and harder guardrails.
What can AI coding agents actually do on their own today?
Agents already submit pull requests at scale. The AIDev dataset (Li, Zhang and Hassan, 2025) contains 456,535 pull requests opened by agents including OpenAI Codex, Devin, GitHub Copilot, Cursor and Claude Code across 61,453 repositories. More than 55% of those agent PRs are feature development or bug fixes, matching the human distribution. Speed is not the issue: GitHub Copilot’s agent delivers half of its PRs within 12.8 minutes, against human issue resolution that usually takes more than half a day.
Where their work gets merged is narrower. The 2026 study of 9,428 agentic PRs in Python repositories found that developers mostly use agents for repetitive, well-structured tasks such as documentation, dependency management and testing, and that narrowly scoped, well-defined tasks merge more often and more consistently than ones needing extensive context. Documentation is the clearest strength in AIDev, with Codex (88.6%) and Claude Code (85.7%) documentation PRs accepted more often than the human baseline of 76.5%.
METR’s time-horizon work puts a size on this. In METR’s 2025 measurement, models had almost 100% success on tasks taking human experts under four minutes and under 10% on tasks taking more than about four hours. That horizon has doubled roughly every seven months over six years, so the boundary is moving fast, yet METR still concluded that the best agents could not carry out substantive projects on their own. Anthropic’s internal data show that Claude Code went from about 10 autonomous actions before needing human input to around 20 over six months.
Benchmark scores overstate what agents deliver in real repositories
The AIDev authors identify a gap between benchmark performance and real-world effectiveness. In AIDev, OpenAI Codex achieved a 64% PR acceptance rate, Devin 49% and GitHub Copilot 35%, between 15 and 40 percentage points below human developers, while top systems report more than 70% on SWE-bench Verified. The authors say these gaps suggest systemic limitations rather than isolated implementation flaws.
Two other sources show benchmark scores overstating capability, in different ways. METR’s 2026 study had maintainers review 296 test-passing SWE-bench Verified PRs and found merge decisions about 24 percentage points below the automated grader. SWE-Bench Pro found Claude Opus 4.1 and GPT-5 at about 23% against over 70% on SWE-bench Verified.
The sources do not all agree on the size of the real-world gap. The 2026 agentic PR study estimates merge probabilities of 84.3% for Claude Code, 73.5% for Codex and 43% for Devin, in popular Python repositories. The two studies draw on different samples, so their figures are not directly comparable. METR also notes that its agents had no chance to iterate on feedback, so the gap may partly reflect missing human-in-the-loop, not a hard capability ceiling.
Levels of autonomy, and why more is not always better
Feng, McDonald and Zhang’s 2025 framework defines five levels by the role the user plays: operator, collaborator, consultant, approver and observer. Their central argument is that autonomy is a deliberate design decision, not an inevitable result of capability, and that a capable agent can still be built to check in with its user at each step. More autonomy means trade-offs in utility, efficiency, accountability and cost. At the top level, errors compound across steps with no one to intervene, which is why the authors judge that the risks of fully autonomous agents may outweigh the benefits in most cases.
Shipping tools such as Claude Code already expose this as a setting. Claude Code offers permission modes ranging from a plan mode that only proposes changes, through modes that ask before edits and shell commands, to an auto mode where a classifier blocks risky actions in the background. That is the levels framework in practical form: the team picks how much the agent may do unasked, per repository and per task.
Answer.AI’s trial of Devin shows autonomy turning into a liability. Of 20 tasks, 14 failed, 3 succeeded and 3 were inconclusive, and the team could not predict which would work. Devin spent more than a day trying to deploy several applications to a single Railway deployment, which Railway does not support, hallucinating features instead of reporting the blocker. The team concluded that workflows where the developer drives, as in Cursor, avoided most of those problems.
Autonomy does not guarantee productivity
The strongest counterweight to productivity claims is METR’s 2025 randomised controlled trial: 16 experienced open-source developers working on 246 real issues in their own repositories took 19% longer when AI tools were allowed. They had expected a 24% speed-up and still believed afterwards that AI had made them 20% faster. METR is explicit that this does not show AI is useless in software engineering, and suggests it may help less experienced developers or those in unfamiliar codebases.
Field data from Microsoft points the other way on output. In Microsoft’s early-2026 rollout of Claude Code and GitHub Copilot CLI, adopters merged about 24% more pull requests than a synthetic counterfactual, with a 95% interval of 14.5% to 33.7%. The authors note that merged PRs reward small, frequent changes and may miss quality costs such as added complexity. The two results measure different things (time on a fixed issue versus throughput across a team), and neither settles whether the extra output is better software.
When does a use case need an agent, and when is one generative call enough?
A predefined workflow suits well-defined tasks better than an agent. Anthropic recommends starting with the simplest solution, possibly no agentic system at all, because agents trade latency and cost for task performance, and adding agents only when simpler approaches fall short. Agents earn their cost on open-ended problems where the number of steps cannot be predicted in advance. Coding suits them when solutions are verifiable through automated tests that the agent can iterate against, though Anthropic adds that human review remains crucial for alignment with broader system requirements. Answer.AI’s verdict on a security review task makes the inverse point: Devin hallucinated issues, and a single call to an LLM would have done as well.
This is where TechnoLynx starts any generative or agentic AI engagement. TechnoLynx first establishes whether the outcome is plausible at current capability levels, and favours repeatable, well-scoped tasks with measurable return over goals that would need super-human capability.
Checklist: is this task ready to hand to a coding agent?
- The change is localised, ideally a few files, since resolve rates fall sharply as the number of files grows.
- Success can be checked by tests or a quick human read, the kind of task engineers say they “can relatively easily sniff-check”.
- The task is repetitive or well-specified, like dependency updates or CI configuration, which merge at high rates.
- The requirements are written down: SWE-Bench Pro scores dropped significantly when agents got only the problem statement without requirements and interface.
- The task is not governance- or cost-sensitive, such as license or token management, the least frequent agent tasks.
- At least one human is accountable for the work the agent is responsible for.
Where these do not hold, we would keep the developer in the driving seat and use the agent as a collaborator rather than an approver-level worker.
Running a coding agent: cost, rollback, latency and guardrails
Unlike a single generative call, an agent operates over many turns, bringing higher costs and the risk of compounding errors; our coverage of agentic AI news and developments tracks the wider shift. Claude Code snapshots files before editing so changes can be undone, but actions on remote systems such as databases, APIs and deployments cannot be checkpointed and have to be controlled through permissions. Anthropic recommends extensive testing in sandboxed environments with guardrails, because autonomy brings higher costs and compounding errors.
SWE-Bench Pro documents agents stuck in “endless file reading”, looping through exploration without ever implementing anything, and Claude Sonnet 4 had context overflow as its main failure mode at 35.6%. Latency tails matter too: Copilot’s agent has a 95th percentile above one hour, which the AIDev authors attribute possibly to cold starts, dependency resolution or API throttling, and which can delay CI/CD pipelines. Cost is a first-class metric, since at organisational scale token spend can run to millions of dollars a year. Our practical guide to building AI agents covers how these concerns change as a system grows from a single tool call to multi-step orchestration.
Where autonomous agents still fail, and where humans must review
Agents’ merge rates are lowest on tasks needing extensive contextual understanding. Function implementation and LLM integration tasks have low merge ratios, which the authors link to business logic, hidden assumptions and cross-module dependencies. On SWE-Bench Pro, wrong solutions accounted for 35.9% of Claude Opus 4.1’s failures, and some agents misread the task entirely, for example building an elaborate reproduction environment for a simple refactor. The best models scored under 20% on the commercial subset, which the authors attribute to the difficulty of enterprise codebases. He, Treude and Lo’s case study of ChatDev shows the complexity cliff in miniature: a working Snake game by the second attempt, but Tetris only on the tenth, still unable to clear completed rows.
Human review stays mandatory for three reasons. Test passes do not equal mergeability: METR’s maintainers rejected agent patches for code quality, for breaking other code and for not solving the issue at all. Accountability does not transfer: the RACI approach in the roadmap requires at least one human to be “Accountable” whenever an agent system is “Responsible” for a task, and accountability for fully autonomous systems remains unresolved. And review capacity becomes a bottleneck as agents generate PRs at scale, so the AIDev authors propose triage that routes low-risk PRs such as documentation to junior reviewers or bots and escalates system-level changes to experienced engineers.
Long-term maintainability remains an open question. AIDev found agent PRs alter fewer structural aspects of code, which raises questions about maintainability. The 2026 study found agentic PRs had comparable or lower defect proneness than human ones, yet cites other work showing agentic code is retained less often, churns more and carries more redundancy. Until those findings converge, we would treat maintenance cost as unknown, not as zero.
Will AI agents replace software engineers, or change what they do?
The evidence here describes a change of role, not replacement. In Anthropic’s survey of its own engineers, most said they could fully delegate only 0 to 20% of their work, and many described shifting towards managing agents, with one estimating their work had moved “70%+ to being a code reviewer/reviser”. Anthropic itself cautions that these early findings from a heavy-adopter organisation probably do not generalise. METR’s 2025 assessment that agents cannot yet substitute for human labour on substantive projects points the same way.
The work moves towards scoping tasks precisely, writing the requirements agents need, reviewing at volume and owning the merge, and the roadmap warns that over-reliance can erode developers’ own understanding of the code. The teams that gain will be the ones that get better at deciding what to delegate, which is a judgement no merge-rate table makes for them.
If METR’s time-horizon trend holds, the set of tasks that pass the checklist above will grow quickly. The open question for any team is how it will manage the review load, and the field still lacks agreed measures of quality.
