Common errors
These are the mistakes reviewers see most often. The examples are made up.
1. Trusting the draft's description of an image
Opening every image and fixing a draft criterion that describes it wrongly.
Keeping "The screenshot shows a green success message" without looking, when the image actually shows a red error.
Always check the image yourself.
2. Locking in one solution
"The response gives a fix that would stop the error."
"The response must use a try/except block."
Let every valid fix pass, unless the user asked for a specific approach.
3. Confusing penalty wording
"The response claims the error is a network timeout."
"The response does not fail to avoid the timeout claim."
A penalty is "true" when the mistake happens. Write the mistake itself.
4. Wrong image numbers
Selecting only Image 2 when the criterion is about the error shown in Image 2.
Selecting every image, or renumbering images from the final message.
Use the numbers on the GitHub page, and only the images a person needs.
5. Marking a nice extra as critical
Critical only when every correct answer must do it.
Critical for a tip that a correct answer could leave out.
Use the "leave it out" test for every critical criterion.
6. Accepting a draft without really checking it
Choosing Accepted after checking every criterion, weight and image number, and explaining why the draft held up.
Choosing Accepted after a quick skim.
Reviewers check that drafts were genuinely reviewed.
7. Bundling several requirements into one row
Separate rows: "The response identifies that the config file path is wrong." and "The response gives the corrected path."
"The response identifies the cause, fixes it and explains why."
One requirement per row, so each part can be marked on its own.
8. Stacking extra sentences onto a row while fixing feedback
Fixing only what the comment asked, and checking whether another row already owns the new check.
Adding "It must also explain each step and keep the variable names." to a row that already worked.
Each stacked sentence is a new requirement and often overlaps another row.
9. Overlapping rows and inverse pairs
Critical: "The response identifies that the config file path is wrong." Penalty: "The response says the file is corrupted."
Critical: "The response identifies that the config file path is wrong." Penalty: "The response does not identify the cause."
A penalty must catch a different mistake, not mirror a critical.
10. Judgment words instead of concrete content
"The response sets the header to http://localhost:3000 or *."
"The response adds the header properly."
Write what a correct answer contains, so every grader marks it the same way.
11. A bonus that rewards something the user did not ask for
The user asked to fix a failing import. Bonus: "The response explains why the import path is resolved from the project root."
The user asked to fix a failing import. Bonus: "The response adds a logging system."
A bonus must stay inside the user's request.
12. Loosening a row so that a wrong answer now passes
"The response gives a fix that stops the error without changing the routes."
"The response makes any suitable change."
List every answer the new wording allows. If a wrong one passes, the row is too loose.
13. Deleting a draft row without explaining it in the rationale
Rationale: "Removed c4 because it required tests the user never asked for."
Removing c4 and not mentioning it anywhere.
Your overall edit rationale must name every deleted row and why.
14. A change log that does not match the edit
"Changed weight from Critical to Bonus because a correct answer can skip the tip."
"Reworded."
Describe the real change, including weight changes and removed requirements.
15. Turning something an earlier AI reply said into a requirement
Rows built only from the user's messages and the material the user supplied.
"The response adds caching", when only an earlier AI reply suggested caching.
Earlier AI replies are context only and can be wrong.