Poker Solver Convergence: Reproduce Exploitability in Kuhn Poker

日本語で読む

After 1,000 iterations, our average strategy's self-play payoff is only 0.000001664 ante per hand from Kuhn Poker's exact game value. It is still exploitable by 0.007269 ante per hand. The payoff looks finished long before an exact best response agrees.

This article completes a deterministic, 100,000-iteration experiment so every part of that apparent contradiction can be inspected. We run full-tree vanilla counterfactual regret minimization (CFR), compare the current policy with its reach-weighted average, enumerate an exact legal best response for each player, and publish the data and both implementations.

Disclosure: this is a synthetic three-card experiment produced by GTO Gecko. It is not output from the GTO Gecko product, a Hold'em solve, a benchmark of another solver, or evidence that one iteration target fits every game.

A payoff can look solved before the strategy is

In this game the exact expected value for the first player, P0, is −1/18 = −0.055555556 ante per hand. At iteration 1,000 our average profile returns −0.055557220 when played against itself. The absolute difference is 0.000001664.

That comparison asks only how two members of the same evolving profile perform against each other. It does not ask whether either player can change strategy and do better. An exact information-set-consistent best response still finds a combined unilateral improvement of 0.014538213; under the convention used here, half of that is 0.007269106 ante per hand of exploitability.

The complete game fits in one small tree

Harold Kuhn's original simplified-poker paper reduces poker enough to analyze optimal strategies while retaining hidden information and bluffing. Our implemented two-player specialization uses three ranks, J < Q < K.

• Both players ante one unit and receive one private card. The third card stays unseen.
• P0 checks or bets one. After a check, P1 checks or bets one.
• After either bet, the opponent folds or calls. There is no raise.
• A checked-down showdown pays the winner a net +1 and the loser −1. A called-bet showdown pays +2/−2.
• The six ordered deals—JQ, JK, QJ, QK, KJ and KQ—each have probability 1/6.

Narrow screen? Scroll horizontally to inspect the complete betting tree.

This tree is small enough to remove two common excuses. There is no random sampling: every iteration traverses all six deals. There is also no opaque best-response approximation: each side has exactly six two-action information sets, so we can test all 2⁶ = 64 pure response plans.

Exploitability needs a non-cheating best response

A best response fixes the opponent's policy and selects the responding player's most valuable legal strategy. “Legal” is the important word. At an information set, the responder knows its own card and the public action history; it cannot choose one action when the hidden opposing card is J and another when it is K.

Our JavaScript generator and independent Python verifier both key a response only by player | own card | public history. They enumerate 64 complete plans for P0 and 64 for P1. A response table keyed by both dealt cards would be clairvoyant, not a poker best response, and would exaggerate the gap.

Let Δ₀ and Δ₁ be the value each player could gain by deviating alone. We retain both quantities:

The second line follows the two-player constant-sum convention implemented in OpenSpiel's pinned exploitability module. OpenSpiel describes it as performance against a worst-case non-cheating opponent, averaged across both seats. Other software may show NashConv directly or normalize by a pot. The label alone is not enough.

At iteration 100,000, the average profile gives P0 an improvement of 0.000784906 and P1 an improvement of 0.000567533 ante per hand. Their sum is NashConv 0.001352439; dividing by two gives the reported 0.000676220 ante per hand. These are expected utility units, not percentages.

Why the average and current strategies separate

The original CFR paper defines counterfactual regret at each information set and connects diminishing average regret to an approximate equilibrium in two-player zero-sum games with perfect recall. That average matters.

Our frozen implementation does three things each iteration:

1. Use positive cumulative counterfactual regrets to form the current policy through regret matching.
2. Traverse the complete game under that policy, recording the value of each available action and accumulating regret.
3. Add the policy to a behavioral average, weighted by the probability that its owner reaches each information set.

We apply all regret changes simultaneously after the six ordered deals. “Current strategy” below means regret matching after that completed update. “Average strategy” means the reach-weighted average of policies used from iteration one onward. No sampling or seed is involved.

All table values are OpenSpiel-style exploitability in net antes per hand. They are not percentages. On narrow screens, scroll horizontally.

The current policy at 20,000 iterations is more exploitable than at 10,000 in these selected checkpoints, even while the average improves. At 100,000, current exploitability is about 395.6 times the average value. This is a result for this deterministic vanilla-CFR schedule, not a statement that every algorithm's last iterate behaves the same way.

The final frequencies are a cross-check, not the score

The final average policy also resembles the known equilibrium family published in OpenSpiel's pinned Kuhn implementation. For example, P0 opens J as a bluff 20.5332% and K for value 62.4364%; the second is close to three times the first. P1 bets J after a check 33.5669% and calls an opening bet with Q 33.3412%.

Frequency matching is useful here because the complete game has an analytic solution family. It is still not the convergence score. Multiple equilibrium policies can share one game value, and a policy can put plausible numbers in visible cells while leaving a profitable response elsewhere. The exact two-sided best response evaluates the whole profile.

Six labels to keep beside any stopping number

1. Game: Which ranges, actions, chance events, payoffs, rake or ICM rules exist?
2. Profile: Is the value for an average policy, current policy, checkpoint snapshot or a post-processed strategy?
3. Response: Is the opponent's best response exact, bounded, sampled or otherwise approximated? Does it respect information sets?
4. Aggregation: Is the display one player's improvement, their sum (NashConv), or their average?
5. Unit: Chips, antes per hand, big blinds, milli-big-blinds, percentage of which pot, or something else?
6. Scope: Whole game, one subtree, one street, one node, one hand or one action?

Do not convert a whole-game number into a guaranteed tolerance for one frequency. Our solver suit-frequency experiment shows why action EV and input isomorphism still need their own checks. For how a product team can verify inputs and artifacts around a solution, see how GTO Gecko's solutions are made. The broader equilibrium idea starts with what GTO poker means.

What the Kuhn experiment cannot establish

Kuhn Poker has one private card per player, one bet size, no board, no future street, no rake and no tournament payouts. Full-tree traversal is trivial here. A Hold'em tree can add ranges, card removal, many public runouts, multiple bet sizes and approximation choices; a tournament model can add non-linear stack value.

For that reason, 100,000 iterations is an input to this experiment, not a recommended threshold. The final 0.000676 value does not certify any commercial solver, and it says nothing direct about a particular hand's frequency. A different CFR variant, averaging delay, update schedule, payoff scale or exploitability convention can produce different traces and printed values.

Where GTO Gecko fits

The current Japanese GTO Gecko App Store listing describes plan-dependent libraries of precomputed poker scenarios that expose action frequencies, EV and range composition, plus guided simulated practice. This public Kuhn program is separate from those product outputs and does not validate them.

The useful transfer is the reading discipline. When studying a mixed frequency or EV, preserve its game definition and unit, and do not turn a strategy-level accuracy label into a hand-level promise. GTO Gecko is off-table educational study software; it does not provide wagering, prizes or real-money play.

Method, downloads and independent check

The release bundle contains no private solve or customer data. It is deterministic and has a fixed file allow-list:

• Method and reproduction README
• Complete experiment JSON
• 1,006-row convergence CSV
• Final current and average strategies
• JavaScript generator and exact best response
• Independent Python verifier
• SHA-256 manifest

The Python implementation repeats all 100,000 full-tree updates, recalculates all 16 retained checkpoints, checks the 24 final strategy rows, re-enumerates both 64-plan best responses and verifies every released digest. Running it in both normal and optimized (-O) modes is part of the verification contract.

FAQ

Does exploitability zero mean every visible action frequency is unique?

No. A game can have multiple equilibrium strategies. Zero under a declared exact convention means neither player can improve by deviating alone; it does not require one unique table of frequencies.

Why not test only the self-play payoff?

Two imperfect policies can offset each other and produce the correct aggregate payoff. At iteration 1,000 in this run, payoff error is about 0.000001664 while exact best responses still find 0.007269106 ante per hand of exploitability.

Why does the current strategy keep oscillating?

Regret matching can move probability sharply as cumulative positive regrets change. The classic CFR guarantee used here concerns the average strategy. This trace does not claim that every solver or CFR variant has the same last-iterate behavior.

Can I compare 0.000676 directly with another solver's number?

Only after matching the game, profile, best-response method, aggregation and unit. A system showing NashConv would display about 0.001352 for this same final average profile; a pot-normalized display would differ again.

Sources

• H. W. Kuhn, “A Simplified Two-Person Poker” — original game and optimal-strategy analysis.
• Zinkevich et al., “Regret Minimization in Games with Incomplete Information” — primary CFR paper and average-regret result.
• OpenSpiel Kuhn Poker source at commit 4840189 — independent game and equilibrium-family cross-check.
• OpenSpiel exploitability source at commit 4840189 — NashConv and two-player convention.