OpenAI Tops Gartner's Coding-Agent Quadrant. Now You Own a Production ML System.

On May 22, Gartner published its first Magic Quadrant for Enterprise AI Coding Agents ↗ and put OpenAI in the Leaders box, citing Codex for agentic software development, enterprise governance, sandboxing, and flexible deployment. OpenAI says Codex now sees more than 4 million weekly users ↗ and names Cisco, Datadog, Dell, and NVIDIA among enterprise adopters.

Good for them. Here is the part that lands on your desk: a coding agent rolled out across an engineering org is not a developer tool you procure and forget. It is a production ML system that writes code, runs tests, and opens PRs against your repos. It drifts. It regresses on model swaps. And right now most teams running these agents have zero monitoring on the output. The same silent-upgrade problem hits any inherited system, as our piece on OpenAI’s DeployCo and the forward-deployed observability gap argues. The Gartner write-up even tells you a version bump already happened mid-evaluation: OpenAI shipped GPT-5.5 into Codex “since Gartner’s evaluation earlier this year.” That sentence is a drift event. Your golden set just changed under you and nobody filed a ticket.

The signal

The MQ is the marketing surface. The operational signal is the deployment scale. When 4 million weekly users and a handful of Fortune 100s wire an agent into CI, the agent stops being an IDE autocomplete and becomes a service with a request distribution, a latency budget, and a quality SLO. Cisco reportedly built its AI Defense platform with Codex and cut timelines “from months to weeks.” Whatever the real multiplier, the volume of agent-authored diffs is now large enough that you cannot eyeball it.

The failure mode is not the agent producing garbage on day one. You would catch that. The failure mode is the agent silently getting 8% worse at your repo’s idioms after a provider-side model update, while your acceptance-rate dashboard stays flat because developers keep clicking accept out of habit.

Mechanics: what’s actually drifting

Three things move independently, and you want them on separate axes.

1. Input drift. The distribution of tasks you send the agent. New repos, new languages, a migration that floods it with one kind of refactor. This is plain old data drift, measurable with PSI or a KS test on your prompt features — the drift, data-quality, and decay metrics taxonomy covers the full signal set.
2. Model drift (concept drift, vendor edition). The provider swaps weights. GPT-5.5 lands. Same prompt, different output distribution. You do not control the deploy and you may not get a changelog that maps to your eval set.
3. Outcome drift. The thing you actually care about: are the agent’s diffs still getting merged, still passing review, still not getting reverted within a week.

Acceptance rate conflates all three and leaks badly. A developer accepting a suggestion is not the same as the suggestion being correct.

The metric that matters

Track net merge-survival rate: of diffs an agent authored or substantially co-authored in a window, the fraction still present in main after N days, minus the fraction reverted or hot-fixed.

merge_survival@7d = (agent_diffs_merged_and_surviving_7d
                     - agent_diffs_reverted_or_hotfixed_7d)
                    / agent_diffs_merged

Why this beats acceptance rate: acceptance is measured at the moment of weakest signal, before tests, review, or production have weighed in. Merge-survival is measured after the system has had a chance to reject bad work. It is the coding-agent analog of a delayed-label metric in any other production model, and like all delayed labels it costs you a reporting lag. Run it on a 7-day and a 30-day window so you see both fast reverts and slow rot.

Pair it with a fast leading indicator so you are not blind for a week: CI first-pass rate on agent-authored diffs (fraction green on first push, no human fix). That moves within hours and correlates with the slower merge-survival number.

Wiring it up

Treat every agent invocation as a span. The OpenTelemetry GenAI semantic conventions ↗ already define the attribute names, so use them instead of inventing your own and you get model/version on every record for free. Our walkthrough of instrumenting LLM apps with those conventions covers the span and metric shapes end to end.

from opentelemetry import trace
from opentelemetry.semconv._incubating.attributes import gen_ai_attributes as ga

tracer = trace.get_tracer("coding-agent-monitor")

def record_agent_run(task, response, outcome):
    with tracer.start_as_current_span("agent.code_task") as span:
        span.set_attribute(ga.GEN_AI_SYSTEM, "openai")
        span.set_attribute(ga.GEN_AI_REQUEST_MODEL, response.model)   # "gpt-5.5-codex"
        span.set_attribute(ga.GEN_AI_USAGE_INPUT_TOKENS, response.usage.input)
        span.set_attribute(ga.GEN_AI_USAGE_OUTPUT_TOKENS, response.usage.output)
        span.set_attribute("agent.repo", task.repo)
        span.set_attribute("agent.task_type", task.kind)             # refactor|bugfix|feature
        span.set_attribute("agent.ci_first_pass", outcome.ci_green_first)
        span.set_attribute("agent.diff_loc", outcome.lines_changed)

The gen_ai.request.model attribute is the one that earns its keep. When OpenAI rolls gpt-5.5-codex underneath you, your spans record the version change automatically, and your merge-survival series gets a vertical line you can correlate against.

Roll the spans into a daily Evidently report so model version becomes a comparison axis, not a footnote:

from evidently import Report
from evidently.metrics import ColumnDriftMetric, ColumnSummaryMetric

report = Report(metrics=[
    ColumnDriftMetric(column_name="task_type", stattest="psi"),     # input drift
    ColumnSummaryMetric(column_name="ci_first_pass"),               # leading indicator
    ColumnSummaryMetric(column_name="merge_survival_7d"),           # the metric that matters
])
report.run(
    reference_data=baseline_df,                # gpt-5-codex window
    current_data=current_df,                   # gpt-5.5-codex window
)
report.save_html("reports/agent_health_2026-06-03.html")

If you already run LLM tracing in Arize Phoenix, Weights & Biases Weave, or MLflow’s trace store, point the same OTel spans there and skip the parallel pipeline. The instrumentation is the asset; the dashboard vendor is interchangeable.

What you’ll see

Healthy looks boring. CI first-pass rate sits in a band, say 70–80% for refactors, and merge-survival@7d holds above your threshold across a model-version boundary. The PSI on task_type stays under 0.1.

Bad has a shape. The clean one is a step: a new gen_ai.request.model value appears, and within a day CI first-pass drops 10 points on one task_type while the others hold. That is a model regression localized to a task class, exactly the thing acceptance rate hides. The uglier one is a slow bleed where first-pass looks fine but merge-survival@30d sags because the agent has started producing diffs that pass tests and review but get reverted in production a week later. That is concept drift between your test suite and reality, and it is an argument for keeping the 30-day window even though the lag is annoying.

Caveats

• Attribution is the hard part. “Agent-authored” is fuzzy once a human edits the diff. Pick a rule (>50% of final lines from the agent span, or a commit trailer) and hold it constant. A moving definition manufactures drift that isn’t there.
• Delayed labels mean delayed alerts. Merge-survival@7d cannot fire on day one. Alert on the CI first-pass leading indicator for speed; use merge-survival to confirm, not to page.
• Cardinality. Do not put repo path or task ID into metric labels. Repo and task_type are bounded; free-form fields will blow up your Prometheus series count. Keep high-cardinality stuff in the trace store, not the metric labels.
• Survivorship bias in the denominator. If reviewers quietly drop bad agent diffs before they ever reach a PR, your merge-survival looks great while the agent is actually wasting human time upstream. Track rejected-before-PR volume separately or the metric flatters itself.
• Vendor changelogs are not your eval set. “Faster, better tool use” tells you nothing about your repo. The only ground truth is your own golden set replayed against the new model version. Keep one and rerun it on every version bump you detect.

The Gartner quadrant is a buying signal. The monitoring is yours to build, and nobody ships it with the agent.

Sources

• OpenAI named a Leader in enterprise coding agents by Gartner ↗ — OpenAI’s announcement of the May 22, 2026 Magic Quadrant placement, with the Codex capability claims, GPT-5.5 note, and enterprise adopter list.
• OpenAI Named Gartner Leader for AI Coding Agents ↗ — Third-party summary corroborating the 4M weekly users figure, named customers (Cisco, Datadog, Dell, NVIDIA), and the Cisco AI Defense timeline claim.
• OpenTelemetry Semantic Conventions for Generative AI ↗ — The official gen_ai.* span attribute spec used in the instrumentation example, including gen_ai.request.model for tracking provider-side version changes.