Cover every requirement
A good rubric checks everything the user asked for, in detail, and nothing they did not ask for.
The user is the source of truth
Requirements come from the user's messages, above all the final request, and from what the user supplied: their code, logs, error messages and screenshots. Earlier AI replies are context only. They may be wrong. Never turn something an AI reply said into a requirement unless the user's final request depends on it.
Make a requirement map before you write
List Every Goal
List every goal in the final request. Each one usually becomes a critical row.
List Every Limit
List every limit the user stated in any turn that still applies: versions, libraries allowed or not allowed, "don't change X", file or output format, length ("keep it short" needs a checkable bar such as "leads with the code, at most two sentences of explanation").
List Every Requested Step
List every step the user asked for, in order. If the user asked to check something before doing something else ("first check which values are allowed, then retry"), the check is its own row. An answer that skips the requested step must not pass just because the final result is right.
List What Each Image Shows
List what each image actually shows and which rows need it.
List the Tempting Wrong Answers
List the one or two most tempting wrong answers: the user's own failed attempts, a common misdiagnosis, a fix that breaks a stated limit. Each becomes a penalty, written as the mistake.
Write every small requirement, but invent nothing
Small details matter when the user stated them or when a correct answer cannot work without them. A version number, a forbidden library, a file the user said not to touch, an exact output format: each gets its own row. Details the user never mentioned are not requirements, however sensible they seem.
Technical checks must be complete
When a row checks a technical judgment, include everything an expert would require, not just the most obvious part.
"The response estimates memory from the size of the data once loaded into memory, not only from the file size on disk."
"The response checks the file size against the available RAM."
Why it works: an expert knows the loaded data can be several times larger than the file.
Why it fails: an answer could pass this row and still give the wrong conclusion.
Weight by what the user asked for
- Part of the main outcome the user asked for: critical.
- Something the user explicitly asked for, even if small: critical.
- A helpful extra inside the scope of the request: bonus.
- Something the user did not ask for: not in the rubric at all, not even as a bonus.
User asked to make the item list load. Bonus: "The response explains that the request works from curl but not from the page because the browser enforces the same-origin rule."
User asked to make the item list load. Bonus: "The response adds login and authentication to the API."
Why it works: the bonus deepens the answer to the user's actual question.
Why it fails: the bonus rewards work far outside what the user asked for.
Bonus rules
- A bonus must be something a strong answer has a real reason to include.
- Never write a bonus that only pays for avoiding a mistake ("correctly does not ..."). Write the mistake as a penalty instead.
- A bonus must not mirror a penalty.