Why We Didn't Put a Language Model in Our Poker Engine

"AI-powered GTO analysis" shows up in a lot of poker app descriptions this year. Under the hood, that phrase usually means one thing: a language model reads your hand and writes an explanation of what you should have done.

We considered doing the same thing. Before we did, we tested it: rigorously, against a held-out benchmark, the way we'd want a claim like that tested before we trusted it ourselves. This post is what we found, and why HandRecorder's GTO engine computes your numbers instead of asking a model to guess them.

This isn't a claim about any other app's implementation. We haven't audited anyone else's code. It's a report on the method itself: what happens when you point a language model at a poker decision and ask it to be right.

What we tested

We benchmarked three on-device language models (two trained specifically on poker, plus Apple's own on-device Foundation Models) against 200 hands from a held-out academic benchmark (the PokerBench test split, arXiv 2501.08328). Each model got its best result across four prompt formats, scored on identical spots.

For comparison: always guessing the most common action ("check") scores 27.5% on the same 200 spots.

The results, plainly

The best-performing model picked the right action 39.5% of the time: genuinely better than guessing, and the only one of the three with real signal. The 4B model, despite being larger, was not statistically distinguishable from the guessing baseline (p=0.0501), and scored on the answers it actually committed to (rather than prose we had to extract an answer from), it fell to 17.5%.

Apple's model scored 23.5%, below the 27.5% baseline. We ran a prior-matched permutation test (shuffling its own predictions to destroy any per-hand correspondence while keeping its overall answer distribution) and it came back with p=0.7457. In plain terms: the model had no detectable per-hand insight at all. Its score is explained entirely by the fact that it answered "call" on 185 of the 200 hands, not by reading the spot in front of it.

Fluent isn't the same as correct

Decision accuracy was one problem. Explanation accuracy was the more surprising one.

When we asked the poker-trained 4B model to explain its reasoning, it stated wrong facts with total confidence. Across the reasoning traces that mentioned a call amount, it stated the wrong figure in 32 of 56, then calculated pot odds from that wrong number, producing a fluent, well-formatted answer to the wrong question. In one case it computed required equity as 90/227 = 39.6%, when the actual call was 64, not 90: the correct figure was 31.8%.

The 1.7B model had a different failure mode. Handed the correct numbers directly, it repeated them faithfully. But its own explanations were internally self-contradictory in 15 of 40 cases. One reasoning trace read: "Raise 37 chips, go all in, and fold."

The one thing that did work

There's a genuinely positive result in here, and it's worth stating clearly: if you hand a model the correct numbers rather than asking it to derive them, it will repeat them back accurately.

We tested this directly with Apple's model, supplying pot size, call amount, and required equity as pre-computed context. Across 104 verifiable numeric claims in its output (pot size quoted back 66 times, call amount 10 times, required equity 28 times), it made zero errors. The closed-enum output format we used also produced zero illegal actions, by construction.

That's a real, useful capability. It's just not analysis. It's transcription. A model that faithfully repeats a number you already calculated is a writing tool. It doesn't tell you whether the number is right.

Why this settles it, for now

The role a language model could plausibly fill here was never deciding: our solver already does that exactly, using closed-form pot odds and minimum defence frequency, exact equity enumeration where the board allows it, and a counterfactual-regret solver postflop. The only role left for a model was narrating what the solver already knows.

Both poker-trained models failed at exactly that job (one with confidently wrong facts, the other with self-contradictory sentences), and grounding fixed the fact-accuracy problem without fixing the judgment underneath it. A narrator with no per-item signal, dressed in complete sentences, is a fluent coin flip.

So we compute first. The solver decides, the maths is exact, and if we ever add narration, it will only ever describe numbers we calculated ourselves, never numbers a model guessed.

What this means for you, today

Every number in HandRecorder's hand replay (pot odds, MDF, equity, per-decision EV loss in big blinds) comes from a real solver running on your iPhone, not a round-trip to a server. There's no server in that loop at all: the GTO engine makes zero network calls, and your hand histories never leave your device. Where we can't be exact (multiway pots, wide ranges), we say so and label the number as an estimate rather than presenting a guess as a fact.

Takeaway

• We benchmarked three on-device language models against 200 held-out poker hands before deciding whether to add one to HandRecorder.
• The best scored 39.5% action accuracy against a 27.5% guessing baseline; Apple's model scored 23.5%, below baseline, with no detectable per-hand signal.
• Grounded models repeat supplied numbers faithfully (104/104, zero errors) but that's transcription, not judgment. The poker-trained models still misstated facts in their own reasoning.
• So HandRecorder computes your numbers with a solver, and only ever describes numbers it calculated itself.

Exact poker maths, computed on your phone. HandRecorder's GTO engine runs pot odds, MDF, equity, and postflop strategy locally: no server, no upload, no guesswork. Free to start on iPhone & iPad; the GTO Explorer is an optional paid upgrade. See it on the App Store.