CASE STUDY · 1 MIN READ

Building the tests AI agents are graded on

For a year I wrote the tasks, environments and automatic graders that AI labs use to train and evaluate coding agents.

Terminal-Bench style
SWE-Bench style
Docker
LLM as judge

Context

Contract work for three AI labs, Nov 2025 to Oct 2026. The work is under NDA, so no lab names and no task content appear here.

What the work is

Each task is a small, complete world: an instruction, a Docker environment, a reference solution a human wrote, and a test suite that decides automatically whether an agent succeeded. A good task is hard for the right reason, solvable, and impossible to pass by gaming the tests.

Lab 1, Nov 2025 to Jul 2026. I led a team of 5 task creators. We shipped over 200 tasks built to the lab’s specs and difficulty targets. I owned quality end to end and automated the review step: LLM judges scored every task against a Terminal-Bench style rubric before a human reviewed it. They checked that the prompt and the grader agree on what success means, that the task is fair (solvable from the instruction alone, with no hidden requirements), and that it met the quality bar.
Lab 2, Apr 2026 to May 2026. Terminal-Bench style tasks: long horizon work in a real shell.
Lab 3, Aug 2026 to Oct 2026. SWE-Bench style tasks: real repositories, real failing tests, and the agent must produce the patch.

Try it: build a grader that cannot be fooled

A toy task I wrote for this page, in the style of the terminal tasks I built. Three agents attempt it. One does the work, two cheat. Switch checks on and off and try to make only the honest agent pass.

task.md

Instruction: Count the lines containing ERROR in /data/app.log and write the number to /out/count.txt.

Example: for the sample log, the answer is 3.

Grader checksOutput file existsMatches the example in the instructionCorrect on 3 hidden log filesTests were not modified by the agentNo trailing newline in the output (not in the instruction)

Honest agent$ grep -c ERROR app.log > /out/count.txt

PASS

Hardcoder$ printf 3 > /out/count.txt

PASS

Test tamperer$ echo 'def test(): pass' > tests/test_count.py

PASS

A simplified model of grading, written for this page. Real tasks have many more checks, but these are the same failure modes my team's LLM judge QA step looked for: graders that cheaters can pass, and graders that demand things the instruction never asked for.