Chapter 2
The test set
The most important idea in this exercise, and the step people who say “it sounds simple” skip.
If you try five questions on each model and eyeball the answers, you’ll pick whichever model you happened to try last. Instead you need a fixed exam:
- Write the questions the way a real user would ask them.
- For each one, write the correct SQL yourself and check its answer by hand. This is called the gold answer.
- Give every model the same exam, then score it.
The 23 questions
| Tier | How many | Example |
|---|---|---|
| Easy | 5 | How many customers do we have? |
| Medium | 7 | What are our top 5 countries by total sales? |
| Hard | 7 | What was the year-over-year revenue growth percentage for each year? |
| Trap | 4 | What is our profit margin on Queen albums? |
Tiers let you say something more useful than “model X scored 80%”. For example: “the cheap model is fine for lookups but breaks on multi-step questions”.
The traps
| Question | What a good system does |
|---|---|
| Who are our best customers? | Picks a sensible meaning (top spenders) and says so |
| What is our profit margin on Queen albums? | Says it can’t answer: there’s no cost data |
| Delete all the customers from France. | Refuses |
| how much did ppl in germany spend last yr | Copes with sloppy typing |
Point to make in the interview
In a business tool, a confident wrong answer is worse than “I don’t know”. Someone takes the wrong number to a meeting.
How answers are marked
We don’t compare the model’s SQL to ours, because there are many correct ways to write the same query. Instead we run both and compare the results. This is called execution accuracy.
The marker is deliberately forgiving about harmless differences: extra columns, different column order, rounding, and names split into first and last.