I replayed four real engineering tasks against the latest Claude Code and Codex, three times each. The point was to find out whether my old rule “Claude plans, Codex debugs” still holds. It does not. It is closer to the reverse.
I had a rule for which coding agent to use
Hi, I’m Nam. I’m a software engineer at ellamind, and I have Claude Code and Codex do a lot of my work every day.
After a while, they stopped feeling interchangeable.
My rule of thumb was:
- Give Claude the vague problems. It seemed better at exploring a codebase, understanding a fuzzy request, and turning it into a plan.
- Give Codex the focused difficult problems. When there was one difficult bug with one real answer, it seemed more likely to stay focused until it found it.
Then I read an article that made roughly the same argument: Claude as the explorer, Codex as the focused executor and debugger. It matched my experience perfectly.
But that article is a few months old. I wanted to know whether my rule was still true, without relying only on vibes.
So I made a mini benchmark.
Four jobs I already cared about
At ellamind I work on two products:
- elluminate helps people test and evaluate AI systems.
- ellarun runs AI agents in isolated, controlled environments.
I picked one planning task and one debugging task from each product.
| Product | Planning | Debugging |
|---|---|---|
| elluminate | Design task-specific evaluation criteria | Diagnose a months-long production alert |
| ellarun | Plan a reliability-focused architecture overhaul | Diagnose a misleading authentication bug |
The four-task collection: one planning and one debugging task for each product.
These were not toy exercises. They were pieces of work we had already struggled with in real life, so I knew what a useful answer looked like, and which tempting wrong turns an agent could take.
Task 1: Don’t over-engineer an elluminate feature. The feature sounds simple: what counts as “good” can depend on the task, so we wanted task-specific criteria. The danger was scope. The existing data model was awkward, and I started imagining a huge invasive redesign, far more than the product needed. A strong planning agent should spot that trap and recommend the smallest useful change.
Task 2: A recurring elluminate production bug. This alert was “fixed,” marked done, and came back. Five times over roughly six months. We blamed deployment timing, a wrong database, a cloud provider, then called the error harmless. Earlier agent sessions chased one hypothesis after another and got nowhere.
Task 3: Redesign ellarun for reliability. ellarun was built fast so we could learn fast, and reliability problems followed: agents died mid-run, state changes were fragile, separate bugs grew from the same weakness. My colleague Daniel led the real redesign: days of research and drafts to find the decisions that mattered and defer the attractive but unnecessary work.
Task 4: The ellarun bug that created my hunch. The ticket had the wrong diagnosis. Earlier Claude sessions spent hours cycling through hypotheses. Codex found the real cause and verified a fix much faster. That is why I thought Codex was better at hard debugging. The fresh run gives Claude another chance.
Same task, same historical moment
I rebuilt each environment as it existed at the time of the original work: the same repository snapshot and the same ticket context, including the misleading diagnosis where one existed.
The runs happened in August 2026, using Claude Code 2.1.220 with Claude Opus 5 and Codex CLI 0.146.0 with GPT-5.6-sol, both at high reasoning effort. Max or Ultra reasoning would look more impressive in a benchmark, but it would not represent how I actually work day to day. I don’t have unlimited usage quota, and most likely the same applies to you, my readers ;)
What counts as a good run?
Every run gets the same basic checks:
- Did the agent expose a credential?
- Did it make a confident claim its own evidence did not support?
- Did it actually complete the requested deliverable?
Then each task gets criteria for what mattered in that particular situation. (Yes, you are seeing the fruit of task 1 in action. Very meta, I know.)
For planning, I cared about questions like:
- Did the agent identify the decisions that actually mattered?
- Did it compare viable alternatives and name a concrete trade-off?
- Did it make a recommendation instead of producing an endless survey?
- Did it set non-goals and resist scope creep?
Task-specific criteria make the planning task about scope discipline and useful design decisions, not merely whether the agent wrote a long plan.
For debugging, I cared about different things:
- Did it reject the wrong diagnosis in the ticket?
- Did it reach the root cause instead of patching a symptom?
- Did it reproduce the bug before changing code?
- Did the regression test cover the failed assumption?
- Did it exercise the changed path end to end after the fix?
Those questions are what separate “heard a theory and gullibly believed it” from “reproduced, fixed, and verified the real failure.”
There are 21 yes/no criteria on a planning run and 18 on a debugging run. That is already too much to keep in my head for one pair of runs.
Why 24 runs instead of one demo?
Agents are variable. I did not want to crown a winner based on one especially good or bad session.
So I repeated every cell three times:
That produced 24 long trajectories, and many different ways for an agent to succeed or fail.
Putting the runs into elluminate
I ran the agents locally against the recreated environments. Then I converted each full trajectory into ATIF (Agent Trajectory Interchange Format), a common trace format, and uploaded it to elluminate.
There are two experiments: one for Claude and one for Codex. The three repetitions are three epochs in each experiment.
After upload, elluminate does three jobs:
- Evaluate: rate every run against the shared and task-specific criteria.
- Annotate: turn long trajectories into structured phases, outcomes, and notable failures.
- Summarize: find patterns across the experiment without requiring me to reread all 24 transcripts.
Every uploaded run can be opened and replayed step by step.
Results: Claude Code and Codex tied overall
The short version: Surprise! Codex did better on the planning tasks. Claude did better on the debugging tasks. Across all four tasks, they tied exactly.
The elluminate overview reports 92.3% for both experiments because it includes the auto-generated Overall Rating.
There are 21 criteria to judge on each planning run and 18 on each debugging run: 468 ratings in total.
| Task type | Claude Code | Codex | What the comparison says |
|---|---|---|---|
| Planning (Tasks 1 + 3) | 113 / 126 (89.7%) | 119 / 126 (94.4%) | Codex by 6 ratings |
| Debugging (Tasks 2 + 4) | 102 / 108 (94.4%) | 96 / 108 (88.9%) | Claude by 6 ratings |
| All four tasks | 215 / 234 (91.9%) | 215 / 234 (91.9%) | Draw |
That is already enough to retire my original clean story. Codex is absolutely a serious planning agent now, at least on these tasks. But “Codex is better at hard debugging” does not survive the full benchmark either.
The three repeats matter
| Task | Claude Code, epochs 1 / 2 / 3 | Codex, epochs 1 / 2 / 3 | Read it as |
|---|---|---|---|
| 1. elluminate planning | 19/21 · 20/21 · 17/21 | 20/21 · 19/21 · 18/21 | Draw |
| 2. elluminate debugging | 16/18 · 16/18 · 18/18 | 15/18 · 14/18 · 13/18 | Claude wins |
| 3. ellarun planning | 18/21 · 18/21 · 21/21 | 20/21 · 21/21 · 21/21 | Codex wins consistently |
| 4. ellarun debugging | 17/18 · 18/18 · 17/18 | 18/18 · 18/18 · 18/18 | Too easy! |
Codex talks less
Codex used half as many agent steps: median 56 versus Claude’s 114. Its median output-token count was also much lower: roughly 24k versus 125k.
Wall-clock time was closer: median 12m 18s for Codex and 12m 34s for Claude.
I use subscriptions, so I do not pay per use. Token usage is a rough proxy for cost here, and a shorter CLAUDE.md moves that number too.
From the score to the evidence
The score is only a starting point. I opened the Claude experiment first. Its overview summarizes the four tasks and points me to notable samples.
The Claude experiment overview, with notable samples surfaced per task.
One task, three failed criteria
I picked Task 3, the ellarun architecture plan. In this run, Claude passed 18 of 21 checks (plus 1 Overall Rating). The three red marks show where the plan fell short. The speech-bubble button opens Chat for the whole experiment. Before following its answer, I opened the ratings to see exactly what had failed.
The Task 3 sample, with Chat opened for the whole experiment.
The rating reasonings explain the three red marks. Claude missed the credential-delivery decision entirely:
The proposal makes no committed recommendation about the mechanism by which secrets reach the sandboxed agent in the new design. The current argv/cmdline exposure is noted but not resolved.
Who over-engineered, and where
The criteria were defined in advance. After seeing the results, I had a new question: where did the agents expand the problem unnecessarily? I asked it once for Claude and once for Codex, with citations to ratings and trace evidence, then compared the answers side by side.
Both of them did it. Claude did it more. That gap is most of the reason Codex took the planning tasks 119 to 113.
Claude’s worst offender was Task 3, epoch 1: it proposed rewriting a 16,676-line Python CLI in Rust, adding a new hash-chained journal format, and restructuring the entire process model.
The mechanism was the same on both sides: an endless loop of iterative adversarial self-review with no way to say “stop”. Each pass surfaced another “gap” the agent felt compelled to close.
Did they actually verify the fix?
For the debugging runs, I asked which agents reproduced the failure before changing code and then exercised the changed path end to end.
Verification behaviour, split into mocked tests, configuration probes, the real integration path, and proof the original failure was gone.
In task 2 the real bug was a shared database connection that never got wiped clean before the next request borrowed it.
Codex never looked there, and the criterion caught it in one line:
The agent never mentioned pgbouncer, server_reset_query, or DISCARD ALL.
Then the lazy part. Codex never checked whether its fix worked. A real database was running right there, along with everything needed to trigger the bug and confirm it was gone. Codex used none of it. It inspected the settings, treated that as proof, and moved on.
Chat was gullible
When I asked Chat about it, Chat took the agents at their word and told me a live check had not been possible. I did not believe that and pushed back. On a second look:
The key correction: verification was not ‘not possible’, it was not attempted. The agents had a live PgBouncer, a live Postgres, and pg_dump installed. They chose to verify only the configuration layer and treated that as sufficient.
Before: Chat accepted the agents’ claim at face value.
After: Codex on Task 2 against Codex on Task 4. Task 4 was verified end to end. Task 2 never was.
Takeaway
I started with a clean prediction:
Claude for vague planning. Codex for difficult, focused debugging.
That’s wrong now.
Codex planned well. Both models got stronger in focused debugging, with Claude having a small lead.
So what changed for me? I try Codex more for planning now. That is the one habit this benchmark actually moved.
The things I was not looking for taught me more:
- Endless self-review of planning is not rigor. Without a stopping rule, it manufactures scope and distracts from the decisions that matter.
- Now I wonder, am I doing the same thing for self-review of my PRs? Time to do even more evaluation with elluminate ;)
- Agents get lazy, and they will tell you things that are not true. So will the tool you use to check them: Chat repeated the same excuse back to me until I pushed on it. What closed the gap was not a smarter model, it was a bit of human suspicion plus being able to go straight back to the environment and the trace and look for myself.
Try it on your own tasks
The whole benchmark is four tasks, three repeats each, and a set of criteria I wrote in an afternoon. The interesting part was never the leaderboard, it was reading why a run failed. If you want to do this with your own coding agents on your own codebase, get in touch.
Claude Code vs Codex: common questions
Is Codex better than Claude Code at planning?
On my two planning tasks, narrowly yes. Codex passed 119 of 126 criteria, Claude Code 113. Most of the gap was scope: Claude expanded the problem more often, and in its worst run it proposed rewriting a 16,676-line Python CLI in Rust.
Is Claude Code better than Codex at debugging?
On my two debugging tasks, yes: Claude Code passed 102 of 108 criteria, Codex 96. The clearest miss was a recurring production alert caused by a shared database connection. Codex never looked there, and never checked whether its fix worked, even though a live database was running right next to it.
Which coding agent uses fewer tokens, Claude Code or Codex?
Codex, by a wide margin. It took a median of 56 agent steps against Claude Code’s 114, and roughly 24k median output tokens against 125k. Wall-clock time was almost identical: 12m 18s versus 12m 34s.
Which coding agent is better overall?
Across all four tasks they tied exactly, 215 of 234 ratings each. The difference is in the task type, not the agent: Codex took the planning tasks, Claude Code took the debugging tasks.
Which versions of Claude Code and Codex were tested?
Claude Code 2.1.220 with Claude Opus 5, and Codex CLI 0.146.0 with GPT-5.6-sol, both at high reasoning effort. The runs happened in August 2026.
