3D and Graphics
Produce Three.js scenes, SVG illustrations and CSS motion from a brief. Scored on rendered output.
No published scores in this category yet.
Suite v1.0 · published Sep 21, 2026
Which model for which job. With reasons.
XOBENCH gives every model one clear request at its highest reasoning effort and takes its first answer, three times over. Thirteen categories of real engineering and business work, scored by deterministic checks and a blind three-judge panel from three vendors, with every raw response, rationale and dollar published.
- Categories
- 13
- 7 engineering, 6 business
- Models
- 1
- in the published run
- Attempts
- 3
- 3 per task per model
- Judge scores
- 39
- deterministic + panel + human
Capability matrix
Top three per category from the latest published run. Score is the mean of task medians; consistency is the average spread across the three repeats. There is deliberately no overall ranking.
Engineering track
7 categories5 tasks
Coding
Implement functions and modules against a precise spec. Scored by hidden tests.
No published scores in this category yet.
Data and SQL
Write correct queries and transformations against a described schema. Scored by result comparison.
No published scores in this category yet.
Debugging
Given code and a symptom, find and fix the real bug without breaking anything else.
No published scores in this category yet.
IT Scripting
PowerShell and shell automation for Microsoft 365, Windows and network administration. Scored on safety and correctness.
No published scores in this category yet.
Security
Find real vulnerabilities in real-looking code, and implement sensitive features safely.
No published scores in this category yet.
UI Design
Build complete, polished interfaces from a written brief. Scored on rendered output.
- Consistency1.3
Business track
6 categoriesClient Communication
Write the emails and announcements professionals actually have to send, in a specified voice, under constraints.
No published scores in this category yet.
Document Extraction
Pull structured data out of messy real-world documents into exact JSON. Scored field by field.
No published scores in this category yet.
Hallucination Resistance
Questions built to tempt fabrication. Scored on whether the model flags what it cannot know.
No published scores in this category yet.
Legal Drafting
Draft and revise letters, clauses and policies a small firm would actually send. Judged by panel with required-element checks.
No published scores in this category yet.
Reasoning and Analysis
Multi-step business reasoning with checkable answers: math, scheduling, policy interpretation, logic.
No published scores in this category yet.
Summarization
Condense long documents into briefs for a named audience without inventing anything.
No published scores in this category yet.
Single shot, highest effort
One prompt, one answer, no tools, no retries. Every model at the top reasoning setting it offers, recorded per run.
Blind cross-vendor panel
Three judges from three vendors, never the vendor under test, scoring against anchored rubrics with written rationales. Deterministic checks wherever a check can be automated.
Median of three, spread shown
Every task runs three times. We publish the median and the spread, plus cost per task and cost per point, because price is part of the answer.