AI Prompts Library

Build me a model evaluation set from real work instead of a public benchmark.

Given the closed issues and their merged fixes below, produce 20 eval tasks. For each:

1. TASK: the issue restated as a self-contained instruction, with no hints about the actual fix
2. CONTEXT NEEDED: which files a model would have to read to solve it
3. PASS CRITERIA: an objective check, ideally a test command, that distinguishes a real fix from a plausible one
4. DIFFICULTY: trivial / moderate / hard
5. TRAP: what a model is most likely to get subtly wrong here

Exclude issues that are pure dependency bumps, typo fixes, or anything where the fix is stated in the issue title.

Issues and fixes:
[PASTE]

I am being sold on a model based on the benchmark claims below. Stress-test them.

For each claim:
1. Is the benchmark contamination-prone? Public repos, well-known problem sets and anything predating the model's training cutoff all are.
2. Was the number produced under conditions I can reproduce? Note any scaffolding, retries, best-of-N, or custom harness.
3. What does the benchmark actually measure, and is that the thing I care about?
4. What is the sibling benchmark this result should predict, and does it?
5. What is conspicuously absent from the claim set?

Then tell me: what would I have to test myself to know whether this model is better for MY workload, and what is the cheapest version of that test?

Claims:
[PASTE]
My workload:
[DESCRIBE]