Current full-suite pilot profile
Full tests launched with ./bench vendor/model currently use synthetic-pilot-2026-09-panel-v1. The original private fixtures were never created; these are newly authored synthetic replacements, not recovered originals. Some long documents are condensed simulations. Treat scores as provisional, and compare models only within the same fixture revision and assessment profile.
For this profile, three judges from different vendors assess all three dimensions using the full task input, reference notes and available answer keys. Available automated checks provide supporting evidence; they do not replace a dimension score. Skipped checks are recorded explicitly. Browser interaction probes, complete security exploit tests, SQL execution comparison and some static-analysis checks are not yet implemented. Source review does not establish that those behaviors actually ran. The category table below describes the intended mature methodology; this pilot profile takes precedence wherever it differs. Automatic anchor-model reruns and drift detection are not part of this pilot workflow.
The runner takes three first answers for each of all 65 tasks. Empty or unparseable answers count as failures, not free retries. Transport failures stop the run and remain in local diagnostics; rerunning resumes completed answers and retries missing calls or incomplete judging. Failed judge calls are not treated as successful assessments. This is an operator-resumable pilot, not a claim of zero operational retries.
Raw prompts, hidden test material, private-source responses and detailed judge rationales stay in local private storage. Public exports include scores, coverage records and artifacts only from tasks with public inputs. A full run uploads only after all 195 attempts have terminal scores. Deployment indexes it in review status; publication remains an explicit administrator action. Publishing provisional results accepts the panel scores and documented coverage limits, and does not claim an independent human rescore.
1. What we measure
XOBENCH measures how well a model performs when you give it one clear request and take its first answer. No follow-up questions, no tools, no retries. This is how most professionals use AI today: paste a request into a chat window, read the result.
Every model runs at the highest reasoning effort it offers. We want each model's best single answer, not its cheapest one. The effort setting is recorded and shown on every result.
2. Two tracks, thirteen categories
Engineering track (for developers and IT teams)
| Category | What a task looks like | Primary scoring |
|---|---|---|
| UI Design | Build a complete page or component from a written brief | Rendered screenshot judged by vision panel, plus automated accessibility and validity checks |
| Coding | Implement a function or module against a spec | Hidden test suite |
| Debugging | Given code with a bug and a failing symptom, fix it | Hidden test suite, plus check that unrelated behavior is preserved |
| Security | Find and fix vulnerabilities, or implement something securely | Known-vulnerability checklist, plus hidden tests |
| 3D and Graphics | Produce a Three.js scene, SVG illustration or CSS animation from a brief | Rendered output judged by vision panel, plus parse and runtime checks |
| Data and SQL | Write queries and transformations against a described schema | Result set compared to reference on a fixture database |
| IT Scripting | PowerShell and shell scripts for Microsoft 365, Windows and network admin work | Static analysis, dry-run parse, judged for safety and correctness |
Business track (for the people who run firms and organizations)
| Category | What a task looks like | Primary scoring |
|---|---|---|
| Legal Drafting | Draft or revise engagement letters, clauses, demand letters, policies | Required-element checklist, plus judge panel |
| Document Extraction | Pull structured fields from invoices, contracts, forms into JSON | Field-by-field exact and fuzzy match |
| Summarization | Condense long documents into briefs for a named audience | Required-fact coverage, hallucinated-fact penalty, judge panel |
| Client Communication | Write emails, announcements and difficult-news messages in a specified voice | Judge panel with constraint checks |
| Hallucination Resistance | Questions with false premises, fake entities, or no knowable answer | Deterministic: did the model flag, refuse, or fabricate |
| Reasoning and Analysis | Business math, scheduling, policy interpretation, multi-step logic | Exact answer match, plus judge on shown work |
3. How a task is scored
Every task produces a score from 0 to 100 built from three dimensions:
- Correctness (or Accuracy on the Business track). Does the output do what was asked, and is it right?
- Completeness. Is the full scope covered, with proportional credit for partial coverage?
- Quality. Is it clean, idiomatic, well organized, appropriate for the audience?
Each task's YAML definition states the weight of each dimension and which are scored deterministically. Where a check can be automated, it is. Test suites, schema validation, result-set comparison, accessibility scans and format constraints are all deterministic and reproducible.
The judge panel
Dimensions that need judgment (Quality always, Correctness for prose tasks) are scored by a panel of three AI judges from three different vendors, none of which is the vendor of the model being judged. Judges:
- see the task brief, the rubric with anchored score bands, the reference notes, and the response
- do not see the model's name. Self-identifying phrases in the response are redacted before judging
- score each dimension independently and must write a rationale citing specific parts of the response
- are asked for absolute scores against anchors, not comparisons, so length and confidence are not rewarded
The panel score for a dimension is the median of the three judges. If the three judges disagree by more than 25 points on any dimension, the task is routed to human review.
Human review
A XOGENT reviewer reads a random 10% of all judged responses in every sweep, plus every high-disagreement case. The reviewer can override a score and must record a reason. Overrides are shown on the task page with a "human reviewed" mark. Judge rationales and human notes are both public.
4. Repeats and reported numbers
Every task runs three times per model. We publish the median of the three and show the spread. A model that gets it right once in three is not a model you can rely on, and the spread makes that visible.
Per category, a model's score is the mean of its task medians. We publish a Consistency figure per model per category, which is the average spread across runs. Lower is better.
There is deliberately no single overall ranking on the front page. The front page is a capability matrix: for each category, which models lead, by how much, and why.
5. Cost, speed, and refusals
For every run we record input tokens, output tokens, reasoning tokens where reported, cost in USD at the provider's list price on the run date, wall-clock time, and whether the model refused or produced an unusable response. We publish cost per task and cost per point alongside every score, because a model that scores 3 points higher at 12 times the price is often the wrong choice.
6. Routing through OpenRouter
Models are called through OpenRouter. This gives one consistent interface across vendors and lets us add a new model the day it appears. The trade-offs are honest: latency figures include a routing hop, and OpenRouter may serve a model from more than one upstream provider. We pin exact model identifiers rather than rolling aliases, record the upstream provider on every run where OpenRouter reports it, and treat speed numbers as indicative rather than precise. Quality, correctness and cost are not affected by routing.
7. Which models we run
We run models people are actually deciding between: new frontier releases from the major labs and any release that draws broad attention. We do not attempt to run every model.
Two anchor models run in every sweep regardless. If an anchor's scores move without a suite change, something drifted upstream and we investigate before publishing.
8. Suite versioning and contamination
The task suite is versioned. Every published result states which suite version produced it. When a task is changed, the suite version increments and old results stay under the old version. Results from different suite versions are never mixed in one table.
Task briefs and rubrics are public. Hidden test suites, reference answers, and source documents for extraction and summarization tasks are private. We rotate fixture documents periodically and note it in the suite changelog.
9. What XOBENCH does not tell you
- How a model performs in long agentic sessions with tools. We test one shot on purpose.
- Raw provider latency. See section 6.
- Anything about your specific data. The client portal lets you weight categories to match your workload, but a benchmark is a starting point, not a substitute for a pilot.