A new benchmark from Alibaba-led researchers asks AI coding agents to do what real users ask of them: build or change working web apps. The best model fully completed just 26.4% of the tasks. Most of the field finished fewer than one in five.
The test is called Co-Design Bench. The paper was posted to arXiv on Oct. 26, and coverage this week from The Decoder has put it in front of a wider audience.
What the Test Covers
The researchers built 273 tasks from real coding-assistant sessions. They started with 26,314 user sessions and 37,463 deduplicated requests. Each task comes with hidden acceptance criteria written by experts, 2,331 in total.
There are two tracks. In the smaller one (100 tasks), an agent builds something new from a request. In the larger one (173 tasks), it makes changes to an existing multi-file front-end project. Ten models were run through all of it, using the same Claude Code harness for the main comparison, with a cap of three hours and 120 turns per task.
The Scores
The paper reports two numbers. The “strict” score is the average share of criteria a model passed on each task. The “full pass” rate is the share of tasks where every criterion passed. The gap between them is the story.
- GPT-6 Astra (OpenAI): 69.9% strict, 26.4% full pass.
- Claude Opus 5 (Anthropic): 66.1% strict, 18.3% full pass.
- Qwen3.8-Max (Alibaba): 65.3% strict, 16.5% full pass.
- Grok 4.6 (xAI), last place: 51.1% strict, 6.6% full pass.
So the top model got roughly seven in ten individual requirements right, yet delivered a fully correct result on about one task in four. A button that works but a form that doesn’t still counts as a miss for the whole task.
Other Findings Worth Noting
Rankings shifted by track. Alibaba’s smaller Qwen3.8-27B model placed fifth on building new apps but edged past Opus 5 on changing existing projects (64.5% vs. 64.2% strict). Cost differed too: on the smaller build tasks, Z.ai’s GLM-5.2 used about 7.3 million tokens against 1.2 million for GPT-6 Astra, nearly six times as many.
The project page also notes that an older Claude model reached 99.4% on an earlier, simpler editing benchmark. That is the contrast the authors are drawing: scores on tidy benchmarks can look near-perfect while messy, realistic requests still trip agents up.
The Catch
- Grading is done by AI. A fixed AI evaluator builds each app and checks it in a browser, and a second model tries to refute its passing verdicts. Two front-end experts spot-checked 200 outputs, but the authors acknowledge that judging errors can’t be ruled out.
- Each task is a single request. The test doesn’t simulate follow-up questions or corrections, which is how many people actually work with these tools.
- The benchmark is front-end focused and led by one company, so it says little about back-end or data work. The paper is also a preprint, not yet peer reviewed.
What It Means for You
If you use AI to build or edit a website or app, treat the first result as a draft. Test every button, form and page yourself, and ask for fixes one at a time. Running a coding agent without a human check is a gamble on the roughly three-in-four tasks where something was still off.
It also suggests being skeptical when a vendor quotes a single headline score. Ask what was measured, how it was graded, and whether it covers the kind of work you actually do. For a related look at AI coding economics, see our story on Revolut’s self-hosted coding model.
More on Contoh
- Revolut Cut Its AI Coding Costs About 90% by Hosting Its Own Model
- Claude Opus 5.5 Is Cutting Off Reverse-Engineering Work. Here’s the Likely Reason