Benchmark notes · September 3, 2026
Measuring agentcad's effectiveness
Using CADGenBench's eval set to isolate agentcad's impact on 3D design quality and efficiency.
To isolate the benefit of agentcad for CAD design, I had two GPT-5 Nano-backed agents attempt the same set of CAD evals. One agent used GPT-5 Nano and Pi, a lightweight coding-agent harness. The other used the same setup with agentcad installed and its skill included in the prompt.
CAD Score
Higher is better
Valid outputs
Higher is better
Model cost
Lower is better
CAD Score is CADGenBench's aggregate quality measure.
Review the methodology and per-run evidence.
The agent using agentcad scored 48% higher on CADGenBench, produced 2.5 times as many valid outputs, used 11.6% fewer total tokens, and cost 21.8% less to run than the agent without agentcad.
What I tested
Control
GPT-5 Nano + Pi
The agent worked directly with build123d. View results →
Experiment
GPT-5 Nano + Pi + agentcad
The agent worked with agentcad 0.5.1. View results →
Both agents attempted the same 81 tasks: 49 generation tasks and 32 editing tasks. They used the same model and reasoning effort, benchmark and dataset revisions, time limits, and concurrency. Each task started in a fresh session.
I chose GPT-5 Nano because it is inexpensive to run and because a smaller model is more sensitive to the quality of its tools. If agentcad was confusing, Nano was unlikely to work around it.
The public evidence bundle includes the pinned revisions, exact harness sources, per-fixture outcomes, token and cost records, and timeout disclosures.
Results
Compared with the control, agentcad improved both quality and efficiency.
Quality
Compared with the control, agentcad performed better on both overall CAD Score and the number of valid outputs.
CADGenBench defines CAD Score as a 0–1 composite of shape similarity, interface match, and topology match, after a hard validity gate. Put simply, it checks whether a part has the right overall geometry, the right pieces and holes, and mating features in the right place. This matters because a part can look plausible but still be unusable if a hole is missing or an interface does not line up. Invalid or missing models score zero.
Overall CAD Score
Higher is better
Valid outputs
33% → 83%
Overall CAD Score rose from 0.124 to 0.184. Valid outputs rose from 27 to 67, moving the validity rate from 33% to 83%. Getting to valid geometry is an important part of the result because invalid outputs cannot earn quality credit.
It wasn't all positive. Most of the gains came from generation. On editing tasks, agentcad performed slightly worse.
Generation
CAD Score
Editing
CAD Score
Efficiency
Across all 81 tasks, total token usage fell from 10.91 million to 9.64 million, an 11.6% reduction. Cached input made up most of both totals and also fell in the agentcad run.
Total tokens
Input, output, and cache
Cached input
Tokens read from cache
Input + output
Non-cached tokens
Cached input fell from 8.21 million tokens to 7.52 million, an 8.3% reduction. Non-cached input and output fell 21.6%, from 2.71 million to 2.12 million. Estimated model cost fell from $0.296 to $0.231, a 21.8% reduction. These figures come from Pi's session-level usage records and cover model usage only.
Review the source data
The reports contain all 81 fixture results. The evidence bundle covers methodology and execution records.
What comes next
My next step is to repeat this experiment with stronger models to see if the result generalizes. I also have a series of features focused on improving agentcad's utility in editing workflows.
Follow the next benchmark
Get an email when I publish new agentcad results and releases. I'll send a confirmation link before adding you.