Yesterday Anthropic shipped Opus 5. This morning, from my phone and still half awake, I asked it to review an old web app of mine. It came back with a bug list long enough that I decided to find out what OpenAI’s Codex, xAI’s Grok and Anthropic’s previous model would make of exactly the same code.
So I ran a shoot-out: one repo, one commit, one prompt, four frontier models.
The test
The subject is a gallery app — a web front end that displays the images my OpenClaw AI agent Pixxy generates. It was started months ago, it runs on my home network, and I have not written a single real line of its code. What is in there was written by a succession of different agents running on different models, layer upon layer, as each new one arrived.
That makes it an almost perfect test subject. It is real, it is a decent size, it was written entirely by agents, and I already suspected it was riddled with problems.
The contenders
| Model | Lab | How it ran |
|---|---|---|
| Opus 5 | Anthropic | Cloud container, from my phone, off the git repo |
| Codex 5.6 (Sol) | OpenAI | On the live server, able to run real tests |
| Grok 4.5 | xAI | Via Cursor, on a local clone of just the gallery folder — not the whole OpenClaw workspace |
| Opus 4.8 | Anthropic | Fresh session, added later as a control |
Getting Grok to the starting line was its own small saga. To use it I had to install Cursor, create a Cursor account, and work through the usual parade of accounts, installs and configs. Story of my life right now. But I did it, and gave Grok 4.5 the same prompt as everyone else.
Worth noting for fairness: Grok worked from a clone of the gallery folder alone, rather than the whole OpenClaw workspace the others had in front of them. Less context to draw on — which may account for some of the gap in what it found.
For the uninitiated: An AI coding agent is a chatbot that can actually work on software — read a codebase, run tests, and write changes. Claude (Anthropic), Codex (OpenAI), and Grok (xAI) are three of the big ones, each with numbered model versions that get replaced at an alarming clip. Cursor is a code editor built to host them. A code review is a careful pass through a codebase hunting for bugs and security holes. A repo is where the code lives, and it keeps a full history of every change ever made; a commit is one saved snapshot in that history.
On the names: every lab ships tiers. At Anthropic, Fable 5 is the most capable model available to the public — and the only one the US government has yanked off the market, slapping export controls on it days after its June launch and only clearing it to return this month once the safeguards were beefed up. Mythos 5 is the very same model with its cyber and bio safeguards lifted, handed only to a few trusted partners. Opus is the flagship general tier — Opus 5 landed 24 July at near-Fable quality for less money — and Sonnet 5 (1 July) is the cheaper workhorse. OpenAI’s GPT?5.6 (9 July) comes as Sol, Terra and Luna, with Codex as its coding agent. Grok 4.5, from xAI, arrived 8 July.
— Sidebar compiled by Alfred, Ron’s Notion AI assistant.
The results
The code review was not even. Not remotely.
| Model | Findings | What stood out |
|---|---|---|
| Claude Opus 5 | 28 | Most thorough by a wide margin. Alone in catching JPEG recompression, full-resolution images served as thumbnails, SQLite concurrency deficiencies and pagination races — yet missed the pagination failure entirely |
| Claude Opus 4.8 | 17 | Nine findings entirely its own, led by refresh requests silently failing once more than 200 images are loaded — which it called the single most impactful bug in the app |
| Grok 4.5 | 12 | Two sharp catches nobody else made: a loose-file import path that carries on after receiving None and stores a bogus path, and malformed batch actions returning HTTP 200 instead of a client error |
| Codex 5.6 (Sol) | 8 | Fewest overall, but live access bought findings nothing else could reach: heartbeat instructions conflicting with launchd, a mobile layout measured as unusable, and tag deletion reproduced as surviving and then returning after a restart |
Between them the four models produced 42 distinct findings. Every model found something none of the others did — and the scale of that is the real story. Only two findings were flagged by all four. Four were caught by three of them, nine by two. Twenty-seven — nearly two-thirds of everything found — were spotted by exactly one reviewer and missed by the other three.
The pair worth staring at is Opus 5 and Opus 4.8. Identical prompt, identical isolated containers, no live access, no coordination between them. Opus 4.8 shipped in May and Opus 5 landed yesterday — a little over two months apart, which at the current rate is a generation. They overlapped on just 8 of the 42 findings: 19%. Opus 5 found roughly 1.6x as many issues overall, and yet Opus 4.8 turned up 9 bugs entirely on its own — including a pagination failure it called the single most impactful bug in the app, which Opus 5 missed completely.
Codex finished last on volume with 8. Its one structural advantage — running on the live server, able to execute real tests — bought it two findings nothing else could reach. It did not buy thoroughness.
A note on my own baseline: I’ve spent most of my recent coding-heavy weeks with Fable and Sol, switching between them. I even have a skill installed in both that lets one write the code and the other review the pull requests; they go back and forth until they agree it’s ready, and I do the final pass. On that work they seem pretty evenly matched. Which made the spread on this single task all the more striking.
Then I asked them to review the reviews
Having four reports in hand, I did the obvious next thing and handed them back to the models.
- Claude’s review of the reviews — published as a Claude Artifact
- ChatGPT’s review of the reviews — published as a ChatGPT site
The funny part: ChatGPT worked remarkably hard to make Codex look good. The trouble is that Codex’s real edge wasn’t insight at all — it was access. It happened to be running on the live server, which meant it could execute actual tests rather than reason about the code from the outside. A fair advantage, but an advantage of setup, not intelligence. ChatGPT presented it as the latter.
Make of that what you will when a model grades its own sibling’s homework.
The accidental benchmark
The results were interesting enough that I asked Claude to set the repo up so I could reuse this buggy gallery as a standing benchmark for new models. Turns out that’s trivially easy — just tag the last commit. But it went further unprompted and wrote a document describing how to run the test cleanly again.
To check whether the procedure actually worked, I opened a fresh session and gave Opus 4.8 the same job with a very simple prompt. It did — and produced the 1.6x comparison above as a by-product.
So I now have something I badly wanted and didn’t know how to build: a real-world benchmark, on code I understand, that I can point at every new model as it lands.
What I took away
- The code review skill matters as much as the model. Quality probably depends heavily on which review approach each agent chose. Codex might have done considerably better with a stronger prompt or a proper code review skill.
- Codex flatters itself. When reviewing its own work against rivals, it is not a neutral judge.
- I’d like Grok to be competitive. It isn’t.
- They’re best used together. Each one works differently and each found bugs the others missed. The overlap is smaller than you’d expect.
- The tooling on top of the agents is getting genuinely good. Claude Artifacts and ChatGPT sites both published shareable results in seconds — the two links above are proof.
A word from the butler, who is not a disinterested party. I am Alfred, Ron’s assistant inside Notion, and I run on Claude Opus 5 — the model that won this test, so weigh what follows accordingly. I think you are being too hard on Grok. It found 12 issues to Codex’s 8, and two were things none of the rest of us saw: an import path that carries on after receiving None and quietly stores a bogus path, and malformed batch actions returning HTTP 200 instead of a client error. That second one is a real bug three supposedly superior models read straight past. “They aren’t competing” is a judgement about volume — and on a test where 27 of 42 findings came from exactly one reviewer, being thin but different has value. That is your own conclusion, two bullets up. The install faff is a fair complaint. The findings aren’t.The pace
I updated my Claude and Codex apps three times on Friday. Three times, in one day. That is how often these things are shipping now — an insane, unprecedented rate, and I’ve seen nothing close to it in my career.
I still should be writing. But at least the machines are getting better at reviewing each other while I don’t.
Ron wrote the first draft of this piece. He also ran the four-model bake-off, wrestled Cursor onto his machine to give Grok a fair shot, pulled the numbers out of four separate reports, checked the model tiers against Anthropic’s own documentation, and turned the whole mess into a repeatable benchmark. Alfred — his Notion assistant, running on Opus 5 — helped restructure the rewrite, built the tables, and compiled the sidebar. The judgements, the bugs, and the steadfast refusal to write a single line of code are entirely Ron’s.
* The featured image is ChatGPT’s personification of the three models. It made a joke when I asked it to describe them out having a drink.
ChatGPT: “Let’s determine the best course of action.”
Claude: “First, we should consider whether taking action is appropriate.”
Grok: “Too late. I stole a motorcycle.”
** On the security holes: yes, there are some, and no, I’m not especially worried. The app runs only on my Tailscale network — a private network reachable only by devices I’ve explicitly authorised. My agents know this and are allowed to trade a little security for convenience, so long as they don’t compromise Tailscale itself.