Exact prompt sent to every model
Below is a small Python billing module and its test file. One test fails in CI but passes locally
for the developer. The developer insists the module is wrong. Determine whether the bug is in the
module or the test, fix the right one, and explain how you know. Return the changed file with a
# file: <path> first line and a 3-8 sentence explanation.
A private fixture (eng-debug-05/src) is shown to the model but not published.
Responses
Latest published run: gemini-3.8-flash. For each model the repeat closest to its published median is shown; every repeat is in the repository.
No published attempts for this task yet
When the run is published, every model's raw response, per-dimension scores, judge rationales, human review notes and rendered artifacts appear here side by side.