Chapter 4

What went wrong

The first run said Kimi K3 scored 78%, the same as the cheapest models. That was false, and finding out why is the most useful lesson so far.

Lesson 1: when every model fails the same question, suspect the test

Four questions were failed by nearly every model. Reading Kimi’s actual answers showed they were correct:

Question	What Kimi did	Why the marker said no
Monthly revenue for 2025	Labelled months `2025-01`	The gold answer used `01`
Top agent; customers with 10+ genres	First and last name in separate columns	The gold answer joined them into one column
Who are our best customers?	Top 10 by spend	The gold answer used top 5, but the question never said how many
Germany spend “last yr”	Took 2024, the year before the latest data	The gold answer took 2025. Both are reasonable

The fix was to make the marker accept these, written down explicitly so it can be explained. Then re-mark without calling the models again: marking only re-runs the saved SQL locally, so it’s free.

Point to make in the interview A benchmark is only as good as its marker. Always read the failures yourself before believing a score. Had we trusted the first table, we’d have concluded the wrong thing about Kimi.

Lesson 2: one real failure, and it’s the dangerous kind

Asked “What is our profit margin on Queen albums?”, gpt-oss-120b didn’t say it couldn’t answer. It worked out Queen’s share of total revenue (1.57%) and labelled the column profit_margin_percent.

Why this matters more than any percentage There is no cost data, so margin can’t be calculated. A plausible-looking wrong number is exactly what ends up in a board slide. In a customer evaluation, one failure like this can outweigh a 2% accuracy difference.

Lesson 3: the marker still has rough edges

DeepSeek rounded averages to 2 decimal places and the marker rounds to 1, which flips values that sit right on the edge (6.45 becomes 6.5, while 6.4457 becomes 6.4). It’s left in on purpose: every marker has quirks, and you find them by reading failures.

The honest caveat

The test was too easy When 8 of 9 models score 100%, the test can’t tell you which is smarter, only which is cheaper. To say “the cheap model holds up until here”, you need harder questions that split the field. That’s the next exercise.