A trillion tokens later, it still can't count

The tiny test

Ask a model for a thoughtful explanation of a difficult idea, then ask it to count something in the prompt. The second question can be less reliable even though a tiny program could answer it exactly. That contrast tells us something about the system we chose to build.

For an exact task, I want an exact procedure. I do not want a longer paragraph defending a plausible integer. The model can still decide which procedure applies, but the arithmetic or string operation should be inspectable.

That distinction matters when we evaluate the whole product. A model with access to tools is a different system from a model answering directly. The useful question is how often the complete system returns the right answer within the time and cost budget.

Specify what counts

Before writing a benchmark, define the operation. Counting bytes, Unicode code points, visible characters, whitespace-delimited words, and model tokens are different tasks. A string containing a combining mark can expose that difference without any difficult language understanding.

For this walkthrough, use 1,000 ASCII-only strings and ask for occurrences of an explicitly named character. Case sensitivity is fixed in the prompt. The reference answer comes from a direct scan of the original string. That narrow definition avoids confusing an instruction ambiguity with a counting error.

Include punctuation, repeated characters, empty strings, and strings of different lengths. Keep the same inputs for every system variant. Do not quietly remove cases where the tool receives malformed arguments. That failure belongs in the end-to-end score.

Compare the whole system

The chart is a worked benchmark with example outcome counts. It compares direct answers, longer reasoning, optional tool use, and a tool path that validates arguments against the source input. It is not a ranking of named commercial models.

Exact-match accuracy is correct answers divided by all 1,000 requests. The final configuration checks that the tool receives the unchanged input, that the requested character is valid, and that the returned result is the value shown to the user.

Where the remaining errors go

The optional tool variant leaves 32 failures. One possible accounting is 17 failures to call the tool, nine altered input strings, and six mistakes when returning the result. Those categories sum to the observed error count and point to different changes in the interface.

The validated path leaves three failures. Relative to 236 failures in the direct-answer variant, that is a 98.7% error reduction after rounding. The corresponding accuracy gain is 23.3 percentage points. Write down which quantity you mean. An improvement in error rate and an improvement in accuracy are not interchangeable percentages.

A correct counting function does not make the whole application correct. A transport failure, a bad operation choice, or an incorrect rendering step can still spoil the answer. Inspect the original input, selected operation, arguments, result, and final response as separate stages.

Accuracy has a time budget

An accuracy table alone leaves out waiting. Measure median and p95 end-to-end latency, including tool selection and retries. Keep cold and warm runs separate. If the tool is local and deterministic, a long delay is probably in orchestration rather than the counting operation.

I would also report calls per request, timeout rate, and the fraction of requests that require clarification. Do not claim a latency improvement from these example accuracy counts. It requires a timed run on a specified deployment.

The simplest path for a well-defined counting task may skip the model entirely. Use the model when it helps interpret a request. Once the operation and arguments are known, repeating them through another generation step can add cost and another place to make a mistake.

A useful division of labor

Let uncertain components handle uncertain interpretation. Let exact procedures handle exact operations. Make the boundary between them visible enough to test.

The benchmark I want to keep is small enough to understand and broad enough to catch interface failures. If an improvement only helps the explanation while the integer remains wrong, it has not improved the task.