A trillion tokens later, it still can't count

The tiny test

Ask a model for a thoughtful explanation of a difficult idea, then ask it to count something in the prompt. The second question can be less reliable even though a tiny program could answer it exactly. That contrast tells us something about the system we chose to build.

For an exact task, I want an exact procedure. I do not want a longer paragraph defending a plausible integer. The model can still decide which procedure applies, but the arithmetic or string operation should be inspectable.

That distinction matters when we evaluate the whole product. A model with access to tools is a different system from a model answering directly. The useful question is how often the complete system returns the right answer within the time and cost budget.

Specify what counts

Before writing a benchmark, define the operation. Counting bytes, Unicode code points, visible characters, whitespace-delimited words, and model tokens are different tasks. A string containing a combining mark can expose that difference without any difficult language understanding.

For this walkthrough, use 1,000 ASCII-only strings and ask for occurrences of an explicitly named character. Case sensitivity is fixed in the prompt. The reference answer comes from a direct scan of the original string. That narrow definition avoids confusing an instruction ambiguity with a counting error.

Include punctuation, repeated characters, empty strings, and strings of different lengths. Keep the same inputs for every system variant. Do not quietly remove cases where the tool receives malformed arguments. That failure belongs in the end-to-end score.

Compare the whole system

The chart is a worked benchmark with example outcome counts. It compares direct answers, longer reasoning, optional tool use, and a tool path that validates arguments against the source input. It is not a ranking of named commercial models.

Exact-match accuracy is correct answers divided by all 1,000 requests. The final configuration checks that the tool receives the unchanged input, that the requested character is valid, and that the returned result is the value shown to the user.

Where the remaining errors go

The optional tool variant leaves 32 failures. One possible accounting is 17 failures to call the tool, nine altered input strings, and six mistakes when returning the result. Those categories sum to the observed error count and point to different changes in the interface.

The validated path leaves three failures. Relative to 236 failures in the direct-answer variant, that is a 98.7% error reduction after rounding. The corresponding accuracy gain is 23.3 percentage points. Write down which quantity you mean. An improvement in error rate and an improvement in accuracy are not interchangeable percentages.

A correct counting function does not make the whole application correct. A transport failure, a bad operation choice, or an incorrect rendering step can still spoil the answer. Inspect the original input, selected operation, arguments, result, and final response as separate stages.

Accuracy has a time budget

An accuracy table alone leaves out waiting. Measure median and p95 end-to-end latency, including tool selection and retries. Keep cold and warm runs separate. If the tool is local and deterministic, a long delay is probably in orchestration rather than the counting operation.

I would also report calls per request, timeout rate, and the fraction of requests that require clarification. Do not claim a latency improvement from these example accuracy counts. It requires a timed run on a specified deployment.

The simplest path for a well-defined counting task may skip the model entirely. Use the model when it helps interpret a request. Once the operation and arguments are known, repeating them through another generation step can add cost and another place to make a mistake.

A useful division of labor

Let uncertain components handle uncertain interpretation. Let exact procedures handle exact operations. Make the boundary between them visible enough to test.

The benchmark I want to keep is small enough to understand and broad enough to catch interface failures. If an improvement only helps the explanation while the integer remains wrong, it has not improved the task.

Dataset and grading

The larger evaluation contains 12,000 requests split across character counts, substring counts, word counts, byte length, Unicode code points, and grapheme clusters. Each request names the unit explicitly. A deterministic reference implementation produces the answer and records the normalized input. Ambiguous prompts live in a separate set because clarification quality is different from exact counting.

Half the strings come from templates designed to stress repeated characters and punctuation. The rest are sampled from permissively licensed text and transformed with recorded operations. Length ranges from zero to 8,192 code points. Unicode cases include combining marks, emoji sequences, right-to-left text, and normalization pairs. Accuracy is reported by task family and length bucket, never only as one average.

The system variants share the same model checkpoint and decoding settings. Direct answering disables tools. Optional-tool mode lets the model choose. Required-tool mode routes well-defined count operations to a deterministic function. Validated mode also checks that the exact source string and requested unit reach the tool. Each run saves the model decision, tool arguments, tool result, and final answer.

Exact match is intentionally strict. A response containing the correct integer plus a conflicting explanation fails. The product has to return one unambiguous result.

export function count(input: string, unit: CountUnit): number {
  switch (unit) {
    case 'bytes': return new TextEncoder().encode(input).length;
    case 'code_points': return Array.from(input).length;
    case 'graphemes':
      return Array.from(
        new Intl.Segmenter('en', { granularity: 'grapheme' }).segment(input)
      ).length;
    default: throw new Error('unsupported count unit');
  }
}

Accuracy by length and task family

The first worked figure makes the direct-answer failure visible as strings grow. Accuracy starts at 96.2% below 32 characters and falls to 41.8% beyond 2,048. The validated tool path stays above 99% in every bucket. Length is not the only source of difficulty, but it is an easy variable to control and inspect.

The second figure separates units. Byte and code-point counts are nearly perfect through the validated path. Grapheme counting is lower because the requested locale and segmentation rules still matter. The tool can be deterministic and the specification can still be incomplete. That residual error belongs to interface design, not arithmetic.

All values have exact counts in the table. Percentages without denominators are hard to audit, particularly when bucket sizes differ. Each length bucket here contains 2,000 requests. Each task-family row contains 2,400.

Cost and latency

Tool use adds an orchestration step, but longer model reasoning also costs time. Measure wall-clock latency from request receipt through rendered answer. In a worked deployment, direct answers have a 410-millisecond median and 1.2-second p95. Longer reasoning reaches 1.9 seconds at the median. A local validated tool path finishes in 520 milliseconds at the median and 1.4 seconds at p95.

Those timings depend on queue load, model serving, region, and retries. They should not be carried into another system. The useful observation is methodological: exactness, time, and cost need separate measurements on the same requests. A routing change may improve all three, or trade one for another.

Count model tokens, tool calls, and retries per successful answer. Reporting cost per request hides expensive failures if failed requests are removed. I prefer cost per correct answer, with timeouts and clarifications left in the denominator.

Error taxonomy and release gate

Every failed request receives one primary label: wrong operation, altered input, invalid argument, tool failure, misread result, or ambiguous specification. Two reviewers label a 200-case sample and reconcile disagreements before one reviewer completes the rest. The categories are only useful if another person can apply them consistently.

A release gate might require at least 99.5% exact match on unambiguous cases, no length bucket below 99%, and a p95 under two seconds. Ambiguous requests have a different target: ask a relevant clarification rather than guess. Combining those sets would reward a system for confidently choosing one interpretation.

The reproducibility bundle contains input IDs, reference outputs, unit definitions, model settings, tool revision, raw traces, and the script that builds figures. It excludes licensed source text when redistribution is not allowed, but retains transformations and hashes so authorized reviewers can reconstruct it.

The point is not that models are bad at counting. The point is that a capable system should know when generation is the wrong computation. A small exact function, surrounded by careful routing and evaluation, is often the stronger design.