A watermark is a budget, and better models have less to spend

Where a watermark actually lives

Both schemes I tested work the same way underneath. Neither touches the words after the fact; both lean on the moment of choosing the next one.

The older one, the green list from Kirchenbauer and colleagues in 2023, hashes the previous token with a secret key and uses the result to split the vocabulary into a "green" quarter and a red remainder, then adds a fixed boost (called δ) to the green scores before the model picks. Text written that way carries slightly more green words than chance, and a verifier that knows the key counts them and reports a z score, which is just "how many standard deviations above chance this is". The newer one, SynthID-Text from Google DeepMind, published in Nature in 2024, is the family Anthropic says Claude will use. It leaves the model's probabilities untouched and changes only the source of randomness used to sample from them, running a small keyed tournament between candidate tokens, seeded from the previous four tokens rather than one. That difference in context length turns out to matter more than anything else about the two schemes, and it gets its own section below.

Both need the same raw material: a moment where more than one word would have been acceptable. The measure of that is the model's predictive entropy, its own uncertainty about the next token, in nats. Zero nats means one live option and no choice to make. After "Newton's most famous work was called Principia", which is Anthropic's own example, there is nothing to choose, so there is nowhere to put a mark. That entropy is the limiting resource has been known since the first scheme was published. What I measured is the two questions that follow from it and that the published work leaves open: what happens to the budget as models get better, and who actually pays it in a language like Polish.

The mechanism itself held everywhere I pointed it, which is worth one sentence before the surprises. Per token, the correlation between the model's uncertainty and the strength of the mark is +0.427 on the smallest Bielik and +0.291 on the flagship (at δ=5), against −0.053 and −0.009 in the unwatermarked control. The mark lands where the model hesitated, exactly as advertised. Everything that follows is about how little hesitation is left, and where it goes.

Finding 1: capacity falls as the model gets better

The honest unit of watermark capacity is not the token but nats of entropy per word, measured on the unwatermarked run, because that is the raw material both schemes spend. I could build five scale rungs where a smaller and a larger model share a bit-identical tokenizer, across three unrelated families, so segmentation is pinned and whatever moves is the model. All five rungs fall. gemma-4 E4B to 12b drops capacity to 0.33 of the smaller model. Bielik 1.5B to 4.5B drops it to 0.44, Bielik Minitron-7B to 11B to 0.62, Qwen3 1.7B to 4B to 0.73, Qwen3 4B to 14B to 0.80. The mean ratio per rung is 0.58. Better models are more certain about what to write, and certainty is precisely what a watermark cannot be written into.

Detection follows the capacity down. With the green list at δ=5, temperature 0.7 and 400 tokens, the share of texts detected across the four Bieliks runs 100%, 100%, 69.6%, 21.6%. SynthID, which needs more uncertainty per position because a tournament with a single live candidate has nothing to decide, falls faster: at the same 400 tokens it detects 86.4% on the 1.5B, 30.0% on the 4.5B and 0% on both larger models at temperature 0.7, and 100%, 94.9%, 6.0% and 0% at temperature 1.0. The flagship writing at full sampling temperature is invisible to the method's own detector.

Temperature deserves its own sentence, because it is the dial a provider actually turns. Sampling temperature decides how much of the model's residual uncertainty is in play at all: on the 11B, green-list detection at 400 tokens goes from 21.6% at temperature 0.7 to 83.7% at 1.0. A provider serving at low temperature, the normal choice for factual work, is spending down an already small budget.

For anyone planning a deployment, the direction matters more than any single number. Watermarking gets weaker along exactly the axis models are improving on. This is a trend to engineer around, and no amount of waiting for better models will fix it, because better models are the problem.

Finding 2: the Polish penalty is the tokenizer, not the language

I went into this expecting Polish itself to be expensive. The hypothesis was respectable: Polish is highly inflected, an inflected form has to agree with its sentence, agreement narrows the set of valid substitutions, and fewer valid substitutions should mean less entropy for the mark to live in. It is a testable claim, and I tested it, and it is not supported.

The clean comparison is the same model writing about the same entities in both languages. Per word, capacity is nearly language-invariant: 3.703 nats per Polish word against 3.680 per English word on Bielik-1.5B, 0.416 against 0.502 on Bielik-11B, ratios of 0.99 and 1.21. Per token, the flagship's capacity differs 2.28-fold (0.140 against 0.320 nats), because its tokenizer grinds Polish into deterministic fragments: 2.956 tokens per Polish word against 1.580 per English word. And detection thresholds are quoted in tokens. At the same 400 tokens on the same flagship, the green-list watermark is 3.72 times stronger in English (the mean z score goes from 6.10 to 22.69, detection from 83.7% to 100%), and SynthID goes from 0% detected in Polish to 23.5% in English.

The inflection hypothesis got a direct test, separate from the tokenizer. Splitting entropy by position type separates choosing which word comes next (word-initial positions) from spelling out the rest of it (word-internal positions). On the 1.5B, whose vocabulary keeps whole Polish words as single tokens (▁Zdzisław is one token there, where the flagship's vocabulary spells it ▁Z, dz, is, ław), word-initial entropy in Polish is 1.039 times the English value, and 1.076 per word. If anything, Polish offers slightly more choice. The estimate is conservative, because the 1.5B's weaker English inflates the English side. The flagship appears to support the old hypothesis at 0.705, and is disqualified from testifying: at three tokens per Polish word its "word-initial" position is a fragment of a word, so the choice of word smears across positions the metric cannot see.

The cross-linguistic literature says the same thing, and said it first. Poelman, Bauwens and de Lhoneux (EMNLP 2025) measure Accessor Variety, a corpus count of how many distinct continuations follow a given word beginning, across 21 languages: English 2.12, Polish 4.74, Finnish 7.14. Richer morphology gives a writer more live continuations, not fewer. Meral and colleagues argued back in 2009 that agglutinative Turkish is fertile ground for watermarking for exactly this reason, and my smallest model agrees with them across a 17-year gap.

The fragmenting vocabulary belongs to Bielik-11B, the flagship of the SpeakLeash open-source effort I collaborate with, and it is there through lineage rather than anyone's choice: the model was scaled up from a base whose vocabulary was not built around Polish, and the tokenizer travels with the weights. The same vocabulary turns out to be the best calibrated of the Polish tokenizers in the false-alarm test at the end of this article, at 25 times the declared rate where the whole-word Polish vocabulary sits at 278, so the inheritance cuts both ways.

I want to be precise about why the corrected version of this finding is better news than the one I expected to publish. A language cannot be fixed, but a tokenizer can. If the penalty were morphological, every Polish deployment would carry it forever; since it is lexical coverage, a vocabulary built for the language recovers the capacity, and my own measurements show the recovered version already exists in the smaller Bieliks.

Finding 3: the choice that makes SynthID distortion-free is the one that makes it fragile

The green list keys its hash on one previous token. SynthID seeds its tournament from the previous four. The wide window is a large part of what earns SynthID its "distortion-free" label: the keyed randomness almost never repeats within a text, so no fixed phrase-level bias accumulates for a reader to notice. The same wide window is what an editor tears. Delete one word and every position whose four-token context contained it now hashes to different randomness, so a single deletion corrupts four times as many scoring positions as it does for the green list.

That is not a story, it is a measurement, and the arithmetic checks out against the observed damage. Deleting 10% of words leaves the green list 86.9% of its detection signal and SynthID 67.9%. At 20% deleted: 77.3% against 47.6%. At half the words deleted: 51.8% against 14.8%. A paraphrase by Claude leaves the green list 78.7% of its signal, with detection still at 100% on this source text, while SynthID keeps 21.8% and detection falls to 41.7%. The measured share of positions with an intact keyed context predicts the surviving signal almost exactly: 0.901 predicted against 0.869 observed, 0.802 against 0.773, 0.519 against 0.518 for the one-token window, and 0.662 against 0.679, 0.426 against 0.476 for the four-token one. I have not found this trade-off named anywhere in the published literature, and it is a property of the method family Anthropic is deploying: the design decision that buys the quality guarantee is the same one that sells the edit resistance.

And the quality guarantee itself is narrower than its name. Scored as average log probability under the model's own unbiased distribution, paired by seed, over exactly the first 200 tokens on the 11B, the green list costs −0.0246 at δ=2 and −0.3002 at δ=5, and SynthID costs −0.0419, confidence interval [−0.0546, −0.0293], which does not contain zero. (Scored over whole generations instead of the matched window, the δ=5 figure is −0.4183; whenever one of these is quoted, it matters which.) A real cost from a provably distortion-free method is not a contradiction, it is a misquoted guarantee: "distortion-free" means unbiased averaged over keys, and any deployment that wants to verify its own text has to fix one key. With the key fixed, sampling tilts toward tokens that happen to score well in the tournament, which is random with respect to how likely the token was, so slightly toward less likely words, and the tilt compounds across the tournament's thirty layers. The method is behaving as proven. The proof is about an average over hypothetical keys, and your deployment is not an average.

Finding 4: surviving an attack is about margin, not method

"Is this watermark paraphrase-proof" turns out to be a question with no answer, because the same attack on the same scheme gives opposite verdicts depending on how much signal there was to start with. The 1.5B's Polish text starts at a median z of 30; after Claude paraphrases it, detection is still 100%. The flagship starts at 5.52, and the same paraphrase drops detection from 75% to 20%. The literature's benchmark number for this attack (Krishna and colleagues, NeurIPS 2023, with the DIPPER paraphraser) is 100% falling to 57.2%, and my two models land on both sides of it. The question worth asking a vendor is not "does it survive paraphrase" but "how many standard deviations of margin does my model, my language and my temperature leave", because margin is what an attack subtracts from.

One result here surprised me enough to double-check it: a round trip through English (Polish to English to Polish, both directions by Claude) is gentler than a straight paraphrase. SynthID keeps 0.321 of its signal after the round trip against 0.218 after paraphrase; on the flagship's green list it is 0.591 against 0.385. Translation tries to reconstruct the text faithfully, and faithful reconstruction preserves phrasing. "Rewrite this in other words" is an instruction to maximise exactly the change that kills the mark.

Then there is the question I actually wanted answered, because it is the realistic evasion scenario and it needs no API: take text marked by model A, ask a free local model B to "paraphrase the passage below, keeping the style", and score the result with A's detector. This is not perturbation of the signal but replacement, since the output is drawn from B's distribution, and A's green list only has purchase where B copied phrasing. Laundering the 1.5B's text (that comfortable z of 30) through a local Bielik-11B leaves 0.191 of the signal, where Claude's paraphrase left 0.787. A free local model beats the frontier API as a laundering tool. It still does not remove the mark reliably: 63.8% of the laundered texts are detected anyway, because a paraphrase copies word sequences, and copied sequences carry the mark with them. Against the marginal flagship source, the same laundering drops detection to 21.7%, so margin decides here too.

The third reading of the laundering table is the one I would not have guessed: laundering strength tracks the launderer's fluency in the target language, not its size. Qwen3-4B is the worst launderer at 0.783 signal left, despite being nearly the size of Bielik-4.5B, which manages 0.524. A model that writes Polish poorly copies more of the original instead of rewriting it, and copying is what preserves the watermark.

One limit has to be stated plainly. None of this could be run on real Claude output, because nobody outside Anthropic holds the key. I tested the mechanism with Bielik as the marked source. The conclusion should transfer, since the launderer neither knows nor cares who marked the text, but it is an analogy, not a measurement on Claude.

The mark still does not know the model is lying

My original reason for caring about watermarks was a hallucination hypothesis: if the mark rides on the model's freedom, and making things up is a kind of freedom, confabulated text should carry a stronger mark than text the model knows. That would give you a free hallucination signal on text already being watermarked for other reasons. I pre-registered it, and it failed, though not as cleanly as a textbook null.

Comparing well-known, obscure and invented Polish entities at a fixed 200-token length, 5 of 28 informative cells show a significant difference against 1.4 expected by chance, and the significant cells sort by capacity: fabricated text carries more mark on the highest-capacity model and less on the two lowest. An intriguing pattern, and it does not replicate. Across all nine models the cell-level correlation between capacity and the contrast is +0.410 (p=0.006), but clustered by model, which is the honest unit since cells nest inside models, the confidence interval is [−0.021, +0.593], and Qwen3 and gemma-4 produce only small positive values with none of the sign-flipping the Bieliks show. I have recorded it as exploratory and not established, which is also how anyone reading this should treat it. The null is not an artefact of my corpus: the invented names are mutations of real ones, and the masking this could cause shows no dose-response in any cell (p from 0.32 to 0.82).

The practical version is unchanged from my original conclusion: a watermark answers "did a machine write this", with limited confidence, and has no opinion on "is this true". Build a hallucination detector on watermark strength and you have built a random number generator. Hallucination needs its own signal, which is what the rest of my research is about.

The verifier is fragile, and its false-alarm rate is not what it declares

Everything so far assumed the verifier works. This last part is for anybody about to buy or build one, because it has nothing to do with the model and everything to do with plumbing. The green-list statistic is a keyed hash, and every input to that hash has to match the generator exactly.

Take the same text, the same key and the same configuration, and verify it on a different machine: at δ=2 the z score goes from 6.75 to −0.29, at δ=5 from 11.50 to 0.04, at δ=10 from 22.47 to 0.37. The verifier does not report an error; it reports "no watermark". I isolated the cause without loading a model: torch.randperm with the same seed produces a different permutation on different devices. Declared vocabulary size does the same thing more quietly. Llama-3.1 and PLLuM tokenize this corpus into identical token ids and differ only in the declared vocabulary length, 128,256 against 128,258, two special tokens that never appear in text. Sweeping that one integer by ±2 moves the mean z on human text from −0.385 to +0.426, and the +2 offset reproduces PLLuM's value exactly. Even round-tripping the text through the tokenizer is not free: decoding and re-encoding turns 400 stored token ids into 398, which costs 4.9% of the green-list signal and 2.9% of SynthID's, before any attacker does anything. None of this is a hole in either scheme. It is a property of the reference implementation, and a checklist for anyone building a verifier: pin the key, pin the backend, pin the declared vocabulary, and verify on stored ids where you can.

Then the number that should worry anyone pointing a detector at a person. I ran 250 human-written texts from Polish Wikipedia, drawn from 86 biographies, text no model wrote, through the green-list statistic at the standard threshold (z of at least 4), whose declared false-positive rate is 0.003%. Averaged over five hashing keys, the realized rate is 278 times the declared one on the whole-word Bielik vocabulary, 101 times on PLLuM's, and nominally 25 times on the Bielik-7B/11B, Llama-3.1 and Qwen3 vocabularies, though each of those is a single alarm; only EuroLLM produced no false alarms at all. The mechanism is effective sample size. Whether a (context, token) pair is green is fixed once the key is chosen, and prose repeats its pairs, so roughly 100,000 scored tokens contain only 971 to 6,197 independent hash draws, depending on the tokenizer. The declared rate is not wrong as mathematics; the sample is smaller than it looks, so the tails are fatter than the token count suggests.

One result from my earlier analysis of this data is withdrawn, and I would rather withdraw it in public than quietly. I had ranked the tokenizers by their mean drift on human text, with some biased toward false accusations and others toward missed marks. Under five keys, every tokenizer's range overlaps every other tokenizer's range: the ranking was a property of a single key draw, not of the tokenizers. What survives the key sweep is the key-marginal false-alarm rate above, and the lesson: a declared p-value must not be quoted without calibration on the target language and tokenizer, because the two Polish whole-word vocabularies sit two orders of magnitude above it, and the 25-times rows rest on a single alarm each.

What was already known, and what this adds

Part of the framing I started with turned out to be already published, and the honest map matters if you are going to rely on any of this. That entropy is the resource a watermark spends is established: Kirchenbauer's original paper, the Christ, Gunn and Zamir undetectability results, and the SynthID paper in Nature all build on it. The per-language version is also taken: Park, Park, An and Han (LUNA, ACL 2026) use next-part-of-speech entropy as an explicit capacity proxy across a typology-aware six-language grid, and find that richer morphology does not reduce capacity, which my word-initial measurement independently confirms on a language they did not test. Mohamed and Gubri, across 133 languages, attribute multilingual watermark failure to tokenizer vocabulary sparsity, explicitly not morphology; my Finding 2 is the runtime, per-position version of their corpus-level result, measured inside the model at generation time.

What I have not found published anywhere: that capacity falls with model quality, shown on five scale rungs in three families with the tokenizer pinned (LUNA does not test scale at all); the keyed-context-length trade-off of Finding 3, with its mechanism measured rather than argued; effective sample size as the explanation for detector miscalibration on real text; and the decomposition of a language's capacity into choosing the word versus spelling the form, which is also, as far as I can tell, the first watermark capacity measurement on a fusional Slavic language.

Why this matters if you ship models

If you are the one obliged to watermark, your detectability is a property of your model, your language and your decoding settings, not of the scheme you adopted, and it has a direction: every model upgrade spends entropy you were counting on. Measure it before you promise it, and re-measure it when you swap models, because a provider can be fully compliant and still emit text nobody can identify.

If your users write a language the tokenizer fragments, the penalty is real and it is fixable, in the vocabulary rather than the language. Before accepting "this language is hard to watermark" from anyone, ask for capacity per word, not per token. The per-token number confounds the language with the tokenizer; the per-word number is the one that transfers.

If you are on the receiving end, a school or a hiring team, the declared false-positive rate is a statement about independent draws that real prose does not supply. On the two Polish whole-word vocabularies the realized rate was 278 and 101 times the declared one. Four further vocabularies produced exactly one alarm each, which is borderline against an expectation of 0.04 rather than evidence of a 25-fold gap, and EuroLLM produced none at all. A watermark verdict is evidence at the level of "this warrants a conversation", never at the level of an accusation, and without a calibration run on your own language you do not have a number at all.

If you are threat-modelling evasion: assume the adversary has a free local model, because that is the best laundering tool I measured, better than the frontier API, and assume it will still fail against high-margin text. And keep provenance and truth separate in your architecture. The watermark has no opinion on whether the text is right.

How I tested it, and what I got wrong on the way

Nine instruct models in three families: Bielik 1.5B, 4.5B, Minitron-7B and 11B, Qwen3 1.7B, 4B and 14B, and gemma-4 E4B and 12b. Two schemes: the Kirchenbauer green list with a green fraction of 0.25 at δ=2 and δ=5 (δ=10 in the backend probe), and SynthID-Text. Two languages on the Bieliks, three temperatures, roughly 8,400 generations, 238 statistical tests. Every prompt runs once per arm, so the unwatermarked arm supplies an empirical baseline for every statistic. Per-token uncertainty comes from a separate teacher-forced pass with the watermark off, so the explanation is never contaminated by the treatment. The attack leg re-computes 1,980 scores over deletion, truncation, paraphrase, round-trip translation and cross-model laundering, with a quality gate that excluded 8 of 360 laundering runs (symbol garbage, refusals, texts too short), each exclusion logged with its reason. The human control is 250 Polish Wikipedia biographies. Confidence intervals are bootstrapped and clustered on the unit that is actually independent, which brings me to the part worth reading even if you skip every number.

This project inflated its own significance four times by pooling at the wrong unit, and I am recording all four because each one survived until someone went looking. Pooling tokens instead of generations made a pilot show the mechanism result backwards. Treating every (context, token) pair in the hash as an independent draw turned what was actually noise into a "+2.6 point bias" of the null, because 100,000 tokens contain about a thousand independent draws, not 100,000. Pooling test cells inside a model made the capacity-ordered hallucination trend look like p=0.003, when the model-clustered interval includes zero. And pooling temperatures inside an arm produced a "detection rate of 53.7%" that was really the average of 21.6% and 83.7%, two numbers no deployment would ever mix. Three of the four were caught by adversarial review of my own analysis, not by me. If you take one methods lesson from this article, take that one: the unit of independence is a decision, and it is the easiest place in the whole pipeline to lie to yourself.

Two more corrections are part of the record. An early version of the analysis scored the SynthID arm with the green-list detector, which measures nothing except that detector's false-alarm rate on out-of-scheme text; every SynthID number in this article comes from its own tournament statistic, after that fix. And one pre-registered hypothesis was withdrawn rather than tested to failure: I had predicted that a tokenizer which splits words into more pieces reaches the detection threshold in fewer words, since it spends tokens faster. The arithmetic was right and the premise was false. The extra pieces are the deterministic tails of fragmented words, they carry no entropy, and a token budget divided by tokens-per-word assumes every token carries equal capacity, which Finding 2 shows is exactly what is not true. Tokens per word itself varies by 65% across these tokenizers, so the temptation to convert will come up; resist it.

The limits I know about: one corpus (biographies) and one register, so code, support replies and legal prose are open questions with their own entropy profiles. The highest entropy band is nearly empty on the large models, so the top of the mechanism curve rests on the 3-to-4-nat band. The English control exists on two Bieliks at temperature 1.0 only. Qwen3 is one model generation behind its newest siblings, chosen because the newer releases do not offer a size ladder; those numbers describe Qwen3-era models. And two pre-registered legs never ran: the blinded judge of Polish inflection quality on watermarked versus unwatermarked text, and the teacher-forcing of two models on identical text that would split the model's confidence from its choice of content in Finding 1.

What I'm looking at next

The two unrun legs first, because they are pre-registered and the design is sitting there: the blinded inflection judge, and the paired teacher-forcing that would tell how much of the capacity fall is the better model being more certain about the same text versus choosing more predictable text to write. Then other registers, because biographies are one entropy profile and contracts, code and support replies will each have their own. And a per-tokenizer calibration pack for the false-alarm problem, since a declared p-value that is off by a factor of 25 to 278 is worse than no number at all. Stay tuned.