Small model, real maths

Chrome's built-in AI model runs entirely on your own machine. On everyday maths and dates it is wrong about half the time. Letting it write a little Python — which the page checks and runs in a locked sandbox — took it to 82–92% on questions it had never seen.

September 2026 · part of chrome-chat, an open-source chat app for Chrome's on-device model (Gemini Nano) · try the lab yourself

The problem: confident, plausible, wrong

A model small enough to run in a browser is good with language and bad with arithmetic, letters and calendars. It doesn't see letters, only chunks of words, and it doesn't calculate, it predicts. Asked plainly, three times each, Nano said:

None of these is hard. All of them are the kind of thing a real assistant gets asked, and every wrong answer sounded exactly as sure as the right ones.

The fix: let it write code, then check the code

The page gives the model one job it's good at — reading the question and writing a few lines of Python — and does everything else in plain code:

1. Decide whether to offer Python at all.code
   Only questions with a number, a quoted word to count, or "sort/order" in them. Everything else is answered normally. This single rule took wrongly-triggered code from 11 in 18 plain questions to zero.
2. Ask for code, not an answer.prompt
   About 90 words: you can't see letters and you make arithmetic and date mistakes, so write a Python function answer() that computes the result.
3. Check the code uses the question's own values.code
   If the question says 918 × 64, the code must contain 918 and 64. This turned out to be the most important check of all — see below.
4. Run it in a locked sandbox.code
   Python (Pyodide) runs in a background worker with its network functions removed, a 10-second limit, and no access to the page or the chat.
5. Check the answer's type, and format it.code
   "How many" must be a whole number (not True); "what date" must be a date; "to 5 decimal places" gets rounded to 5. A fixed wrapper does this — never the model's code.
6. On a failure, say why and retry (twice at most).prompt, sent by code
   "Your code does not use 918, 64 from the question — compute the answer from them; do not write the answer yourself."

The check that mattered most

Asked to write code, Nano's first move was usually to hard-code its guess — code that just returns the wrong number it would have said anyway:

def answer():
    return 14200761      # its guess for 4817 × 2953

Three runs, three different wrong guesses (14,200,761 · 14,226,001 · 14,201,061). Because none of them contains 4817 and 2953, each was rejected with the reason, and the retry came back as:

def answer():
    return 4817 * 2953   # 14,224,601 — right, every time

The same happened for powers, percentages, averages and letter counts. Rejecting code that doesn't use the question's values is what turns a model that guesses into one that computes.

Results on questions it had never seen

The pipeline was tuned on 29 questions, which flatters it. So two further sets were written and committed before the changes they test, then run once each on the real model, with nothing changed afterwards:

The gains are biggest exactly where the plain model is confidently wrong: dates (from almost never right to almost always), long multiplication, percentages and operator precedence. On things it already gets right — capitals, translations, a polite email — it's left alone.

What didn't work

• Letting the model decide when to use Python. It mostly didn't: on a test model it wrote code for 16 of 54 questions that needed it — and answered "how many r's in strawberry" in its head, wrongly, every time.
• Telling it to use code for "any question involving numbers". It then wrote code for 11 of 18 plain questions too, including one that returned the boiling point of water as 32.
• Trusting the model's own label for the answer type. It labelled a sorted list as a number, the check demanded a number, and it obediently returned just the first item. Only types plain code can infer from the question are now enforced.
• More thinking time. A hidden "reasoning" mode on the test model didn't fix date mistakes and doubled the time. Changing the tool did: Python's date library 24/25 vs JavaScript's 15/25.

What's still wrong

• Code that runs but is wrong. "4 hours 50 after 11:20" came back as 15:10 — the code forgot to carry the hour. No check catches right-looking code with a logic slip. Showing the code next to the answer at least makes it checkable.
• No clock. Without today's date it believes it is 2024. The fix is to tell it the date — but that answers "what day is it", not "what has happened since it was trained".
• Silent assumptions. "3/4/2027" was read as 4 March (the US way) every time, never flagged.
• Exact fractions and rounding asked for in words ("as a fraction", "to the nearest whole number") still slip.

How it was tested

• Every question and its right answer were written and committed before the code that answers them. The two unseen sets were run once each on the real model and never tuned on.
• Every answer was scored by plain code against the known answer — the model never marked its own work. Failures and their code are kept, not just totals.
• A smaller open model on a laptop stood in for fast iteration; every headline number here is from the real on-device model in Chrome 154.
• Small sets: 28 unseen questions, each run 3 times (84 answers). Treat the percentages as a clear direction, not a precise rate.

Why it matters

On-device AI keeps data on the machine, costs nothing per question and works offline — but only if it can be trusted. The lesson here generalises: don't ask a small model to be right; give it a tool, make it use the tool, check the result in plain code, and show the working. The same pattern — source or tool, plain-code checks, visible evidence — is what makes it usable for real documents and real decisions.