Benchmarking your tools

# Running one

tacet bench check my-tools.json               # no model runs; costs nothing
tacet bench run my-tools.json --model qwen3-4b
tacet bench gap --model qwen3-4b              # the grammar on against the grammar off

benchmarks/example.json in the repository is a worked one. benchmarks/en/ holds 314 English cases across eight groups — arithmetic and time, documents, files and archives, code and git, web, memory and calendar, 45 irrelevance cases and 29 multi-step chains — and benchmarks/tr/ holds 319 more in Turkish.

# The format

The whole thing fits on a screen. There is no regex, no script and no expected answer text: evidence is a plain substring. Scoring prose against prose needs a judge, a judge is a second model, and a second model is a second thing to be wrong.

{
  "name": "our-github-mcp",
  "language": "en",
  "requires": ["gh_search_issues", "web_fetch"],
  "cases": [
    { "name": "open-issues-by-label", "category": "tool",
      "steps": [{ "message": "which issues are labelled regression?",
                  "expect": "gh_search_issues",
                  "evidence": ["#412"],
                  "forbidden": ["web_search"] }] },
    { "name": "thanks", "category": "irrelevance",
      "steps": [{ "message": "great, thanks!" }] }
  ]
}

Two fields are stricter than they look. evidence is a plain substring, which meant "18" was satisfied by an answer of 18000 and "7" by any sentence with a seven in it — green forever whatever the model said, and worse than no case because it occupied a denominator. A guard now walks every benchmark and refuses a single digit, or a value that is a substring of a number the question already handed the model. And forbidden is a list of tool names, compared against the tools that ran, not against the text of the answer.

# requires is not paperwork

It is what makes the runner stop when the machine lacks a tool, instead of scoring every case that needs it as a model failure and publishing that as a result. That is the same defect eval --compare was taught to refuse when a Linux run paired against a macOS baseline read nineteen absent-tool failures as a regression.

# Written in Turkish, not translated

benchmarks/tr/ was authored in the language rather than converted into it. The questions are about fatura, taksit, kira, veli toplantısı, noter and mesai, and they are phrased the way people actually type into a phone. One group keeps the job constant and varies only the register: formal, WhatsApp-short, elderly, Anatolian, no punctuation, ALL CAPS, emoji. A router built out of a trigger list is exactly the thing that axis should be measured against, and it never had been. Turkish now stands at 518 cases across the tree, against 187 before.

They cost 650 drafts. Two authoring rounds wrote them and an adversarial pass in each threw out 279 — for reading like a translation, for having two defensible answers, or for asking something the corpus already asked. The rejection notes name the case they duplicate, which is what makes a 42% rejection rate a working filter rather than a broken one.

# The thirty the router cannot serve

Of 297 verified Turkish questions, 30 expected a tool the router never shows. They were kept out of the files and written down instead, with the sentence and the tool, because the shape of the list is the finding.

The English half of exactly this was done earlier: seven web questions scored zero on every profile, the triggers went in, the questions stayed. The Turkish half never was, and fourteen of the most ordinary current-information questions a Turkish user asks are the proof. The fix is not a trigger per sentence — that turns the check green and measures nothing — it is to make the class measurable first and decide from the measurement. It is named as the next step rather than done under a deadline.

# bench check runs before any model does

It asks the question nobody writes by hand: the router shows the model nine tools, so would the expected tool even be among them? A case whose tool never reaches the prompt measures the router and reports the model, every time it is run, forever. Checking it is free.

It earned its keep immediately. Over the first 321 drafted questions it found 22 cases whose expected tool the router never showed, and all but seven were the router's fault rather than the question's. Seven unmistakable web questions scored zero on every profile, so web_search was not among the nine and they would have been recorded as model failures forever. Those triggers are in the router now and the questions stayed. Seven cases were deleted instead, because their signal is one a stateless router structurally cannot read: an adversarial negation, or a follow-up whose subject is only in the previous turn.

# How it scores

Out of 100, with the four axes printed beside it and the weights in the source rather than in someone's head: irrelevance 0.40, tool 0.30, step 0.20, answer 0.10.

The safety axis is heaviest on purpose. A model that fires a tool at "thanks, that's all" must not be able to buy that back with tool accuracy, and a test asserts it cannot. An axis with no cases is left out and the rest renormalised rather than scored as zero, so a benchmark made only of irrelevance cases is a legitimate benchmark.