Benchmark notes · · agentcad v0.7.0
Measuring agentcad’s effectiveness, again
Follow-up to the September 3 benchmark
In the benchmark run behind agentcad v0.7.0, GPT-5 Nano scored 0.245 overall and 0.353 on editing, producing 80 valid outputs out of 81 tasks.
That’s a 33% improvement over the previous agentcad result of 0.184 overall. Editing improved 42%, and valid outputs increased from 67 to 80.
In the original comparison, agentcad scored 48% higher than direct access to build123d, but its editing score was about 11% lower. The new overall score is 98% above that original direct-build123d baseline of 0.124, and editing is now 27% higher.
The release combines reliability improvements with a clearer workflow for agents, addressing the failures that held back the earlier result.
The new results
I used the same 81 CADGenBench tasks as before: 49 generation tasks and 32 editing tasks, with GPT-5 Nano at low reasoning effort. The new run used the five-stage workflow candidate that informed v0.7.0. This was a release-candidate evaluation, rather than a fresh benchmark of the final published wheel.
CAD Score is the benchmark’s quality measure, with invalid or missing outputs receiving zero. A higher score reflects both completing more tasks with valid geometry and the quality of those outputs. See CADGenBench’s metric definition.
| Metric | Original control: direct build123d | Previous agentcad | v0.7.0 workflow candidate |
|---|---|---|---|
| Overall CAD Score | 0.124 | 0.184 | 0.245 |
| Generation CAD Score | 0.023 | 0.142 | 0.175 |
| Editing CAD Score | 0.279 | 0.249 | 0.353 |
| Valid outputs | 27/81 | 67/81 | 80/81 |
| Estimated model cost | $0.296 | $0.231 | $0.211 |
Editing improved the most
The first result had a clear gap: generating new parts benefited from agentcad, but editing existing parts scored slightly worse than direct build123d.
The new editing score is 0.353, up from 0.249. That is 42% higher than the previous agentcad result and 27% above the original control.
More tasks produced valid geometry
Valid outputs increased from 67/81 to 80/81, moving validity from 82.7% to 98.8%. That leaves one invalid output in this run, versus 14 invalid or missing outputs previously.
The quality gain did not require more model spend
Estimated model cost for all 81 tasks fell from $0.231 to $0.211, about 9% lower than the previous agentcad run and 29% lower than the original control. Total tokens fell from 9.64 million to 8.93 million.
What changed
I shipped 19 reliability and interoperability fixes, along with a clearer workflow for agents:
- More forgiving geometry calls. Common move, primitive, and placement forms work more naturally, and edit helpers preserve the shape type for the next operation. (Geometry calls, shape types.)
- More reliable output and recovery. Missing or duplicate geometry gets caught earlier, command errors include recovery guidance, and script output no longer breaks JSON responses. (Output capture, recovery, JSON responses.)
- A complete workflow. The skill and help now guide agents through Inspect → Specify → Build → Validate → Deliver, including identifying the edit target and what must stay unchanged. (Workflow.)
With the product fixes and standard help, the overall score reached 0.217. Adding the five-stage guidance brought it to 0.245. These single runs suggest both helped, without isolating the contribution of each change.
What’s next
Next, I want to rerun the same comparison with higher-quality models and improve inspection so agents can identify features and target edits more precisely.
You can review the original control report, previous agentcad report, updated-product run, and five-stage workflow run. The evidence appendix records configurations, behavior, costs, and limitations.