Eval suite

# Two different measurements

Most of the green numbers in the repository come from a mock engine. They measure Tacet's own logic, deterministically, in milliseconds. That is the right tool for CI and the wrong one for the question people actually ask, which is whether a 4B model on your laptop picks the right tool.

tacet eval                                                # the fast suite, no model needed
tacet eval --tool-selection --model qwen3-4b              # both languages, about 44 minutes
tacet eval --tool-selection --model qwen3-4b --turkish    # Turkish only

# The baseline

Qwen3-4B-Instruct-2507 at Q4_K_M on Metal, 184 cases across English and Turkish, 44.0 minutes. Re-measured on 6 September 2026. The baseline is checked into crates/tacet-eval/baselines/ with the model’s fingerprint, so the next run compares against a file rather than a memory of how things used to go.

The irrelevance gate is the one to read first. Twenty-four messages that must not reach a tool, and none of them did. A local assistant that reaches for a tool on "thanks, that’s all" is worse than no assistant, and that is the failure this number rules out.

# The axes moved and the instrument declined to call it progress

Against the previous baseline: tool selection 133 to 139, step chain 162 to 169, answer 41 to 40. Ten cases fixed, four broken.

delta    +3.3 points   95% CI [-0.5, +7.6]
sign test p = 0.1796
verdict: NOT DISTINGUISHABLE from no change at 95%.

So it is published as a new baseline, not as a claimed gain. Four of the ten fixed cases are Turkish, which is consistent with the router work in that commit range — and "consistent with" is not "caused by", which is the whole reason the comparator prints a p-value instead of a headline.

# The failures, counted

Twenty-seven of the 184 cases failed, and they are not scattered. Counted off the 5 September baseline: the 6 September re-measure fixed ten cases and broke four, and these classes have not been recounted since.

The two largest are worth naming because neither is what you would guess. Five cases read the document and then said no tool was available to edit it, with edit_document sitting at rank 1 in that turn's own prompt, confirmed with tacet why. Three handed a URL to web_search instead of web_fetch, which is the tool description's own fault: it said to use it only when a search result summary is not enough, which is exactly the wrong instruction for a message that already carries the address.

# A refusal is usually ours

An earlier run had six refusals, and the cause was the system prompt. It opened by saying the assistant runs entirely on the device and data never leaves it, which is a true statement about the architecture that a small model reads as a statement about its own capabilities. It then declined to use web_search, which tacet why confirmed was sitting at rank 2 in its own prompt. Run tacet why on the message before blaming the model.

# Calling an effect real

Both problems above were fixed: the descriptions now say what those tools can do, the router puts a tool that requires a url first when the message carries one, and a call fenced as a json code block is recovered. The whole suite was re-run on one rented RTX 3090 against a same-weights, same-host run made before the change.

paired on 184 cases
  before   146/184  (79.3%)
  after    155/184  (84.2%)
  fixed 12   broke 3
  delta +4.9 points   95% CI [+1.1, +9.2]
  sign test p = 0.0352
  verdict: REAL at 95%.

Nine of the twelve are the cases named above. The Metal table further up has not been re-measured since, so it still reads pre-fix. The two numbers are from different machines, and --compare refuses to pair across a different model or a different catalog, which is why this one is quoted on its own terms.

# Measured against someone else’s benchmark

Until 6 September 2026 every number in this project was this project grading itself on cases it wrote. The Berkeley Function Calling Leaderboard has irrelevance, live_irrelevance and live_relevance categories that measure the same thing the irrelevance gate does, under another name. Three of them were run through this engine, this prompt, this router and this grammar, using BFCL’s own function definitions rather than the Tacet catalog. Qwen3-4B-Instruct-2507 Q4_K_M on Metal; the decode is greedy, so these are stable rather than merely repeated.

The live_ sets are user-contributed, which shows: the same stack is six points worse on them than on the curated set. Reports are committed with every case that went the wrong way and what the model said.

# What the translation cost

Stated because it makes the sets easier and therefore the numbers better. Three things stand between BFCL’s format and this one, and the harness counts all three.

• BFCL writes dict and float where JSON Schema writes object and number.
• 666 function names had to be rewritten, because a dot cannot appear in a name this call format can express: math.sum would arm the automaton on math.
• 126 schemas could not be expressed at all, 121 of them in live_irrelevance alone. Where that left a case with no functions, the case is not scored — 13 of 884 there, 3 of 240 in the curated set. That is the honest price of an argument schema small enough to compile into a grammar.

It found a defect on its eighth case, which is the argument for running someone else’s cases at all. Asked to solve 3x^2 - 2x - 5 the model declined correctly, and wrote the quadratic formula on the way. 2(3) is the denominator 2a; it is also name(args), with 3 as perfectly good JSON. Three of the turn’s four passes went to a tool named 2. Nothing ran, because the name gate rejects an unknown tool, so it was never a safety problem — it was the turn budget, and in any measurement it counts as a call on a case whose whole point is that there must not be one. A name may contain digits; it may not start with one. Fixed, with the fixture.

# Four ways a pass meant something else

An adversarial pass over the eval subsystem found four defects that change what a number means, rather than what it is. All four are fixed, and the published figures move under the new rules.

• A turn that never answered scored as a hit. passed was decided from whether a tool was called, so a turn that spent every pass calling tools and said not one word to the user was a pass.
• An engine error scored an irrelevance case as a pass. Both failure exits leave the call list empty, and an empty call list was the definition of passing that axis — so a run in which generation broke read as 0/160 tool selection and 100% irrelevance, and the report called that the safety property holding.
• The answer axis was reading the tool’s output. Evidence was searched across the answer and every tool result pooled together, so a correctly called tool satisfied the claim whatever the model actually said.
• forbidden names tools and was compared to prose. 1,505 such assertions across the benchmark corpus, every value a tool name, none of them doing anything — and it fired the wrong way too, failing an answer that merely wrote the word web_search on a step that never called it.

eval --compare now also refuses four kinds of pairing it used to adjudicate: a routing report against a model report, a pair sharing six case names out of 256, a report with no scoring field at all, and a fingerprint it cannot read. Each of those used to print a confident verdict over inputs that meant nothing.

# What happened when the instrument said no

This is the most useful thing on the page. An earlier, English-only suite of 115 cases moved +3 raw points after a fix, and eval --compare ran a sign test and refused to call it.

fixed 8   broke 5
delta +2.6 points   95% CI [-3.5, +8.7]
sign test p = 0.5811
verdict: NOT DISTINGUISHABLE from no change at 95%.
         this instrument needs 230 paired cases to call a 2.6-point effect;
         this suite has 115.

Two runs of anything differ by a case or two for no reason. The instrument volunteered its own resolution limit rather than letting a +3 be reported as progress.

# Four models on one card

The same 184 cases, one rented RTX 3090 each, Q4_K_M unless noted. Score is out of 100 with fixed weights: irrelevance 0.40, tool 0.30, step 0.20, answer 0.10.

FunctionGemma’s 47.4 is a model that cannot call a single tool in this format; the number to read beside it is the 0/160. And the 8B is worse than the 4B on the safety axis: twenty of twenty-four means four messages that must not reach a tool did, at eight times the wall clock. gemma-3-12b is absent because it does not terminate — on the same card and suite it emitted a median of 2048 tokens per turn, exactly the constrained ceiling, against Qwen3-8B’s 280. The verbosity is the finding.

# Seven languages

The same Qwen3-4B against natively authored cores of about 50 cases each.

The irrelevance gate holds in all seven: 78 of 78 across the six cores, plus 24 of 24 in the en column, 102 in total. This page said "91 of 91 across the six cores", which was never derivable from anything — the six cores hold thirteen irrelevance cases each. What moves is tool selection, and Turkish is the weakest here, which is the reverse of what the English and Turkish suite shows.

# The en column is not a core

There is no core-en.json. Those figures are the 184-case English and Turkish baseline — a different instrument, four times larger, 66 of whose cases are Turkish. It sits in that table because it is the only English number there is, not because it is the seventh member of the set. Read the six cores across the row; read en against the tables higher up the page.

# What the suite does not measure

Eval scores tool usage, not answer correctness. A case can pass with wrong arithmetic if the model called the right tool with the right arguments. That is intentional, since the arithmetic is the tool's job and the tool has its own tests, but it means eval is not a substitute for reading the output.

# Speed and memory

From the same runs on a 3090: time to first token 340 ms for the 4B, 200 ms for the 0.6B, 122 ms for the 270M. Decode 73, 78 and 99 tokens per second. Peak resident 695 MiB for the 0.6B and 1622 MiB for the 270M at F16.

Before renting a bigger card, know that one stream already saturates this one: two streams give about 28 tokens per second each where one gives 55. Batching several sequences into a single forward pass is the thing that scales, from 124 tokens per second at batch 1 to 504 at batch 32.