← Talks

Talk · AI Ninjas Dojo · August 2026

A watermark detector recognises its own key, not AI

An hour on how the invisible mark gets into generated text, and on the gap between what a detector measures and what people read into it.

Grzegorz Brzezinka· 26 August 2026· AI Ninjas Dojo· 60 minutes, members only

The short version

I took text watermarking apart on local models, live, for the AI Ninjas community. One idea reframes every headline about AI detection: a detector only ever recognises the key it was given. Hand it text marked by a different scheme and it reports human writing, which is also what it reports when a human really wrote the text. Everything below follows from that, including why a clean negative result proves so much less than it looks.

Materials

The workshop, the deck and the notebook are in Polish; this write-up is in English. The notebook carries its corpus inside it and runs on CPU, so you can rebuild the whole mechanism and measure your own numbers on a laptop. The figures on this page come from a separate study on nine models, and that repository is not public.

Why a provider marks its own text

Marking generated text serves three different interests, and it is worth separating them before judging whether the mechanism is any good, because each one wants something else from it.

The first is training hygiene. Models learn from text scraped off the web, and a growing share of that text now comes out of models. A provider that can recognise its own output can filter it back out before the next training run. Here the mechanism only has to work on average over millions of documents, and a mistake on any single one costs nothing.

The second is provenance for a person who is about to read something: a term paper, a job application, a product review. This is the interest most people have in mind, and it is the one the mechanism serves worst, because the decision is about one document and it lands on one named human.

The third is measurement. Whoever holds the key is the only party who can say how much of the text on the internet came out of their own model. That is a description rather than an accusation, but it does mean the same technology delivers a market metric alongside the transparency.

The visible push is regulatory: the transparency duties in the EU AI Act for generated content, and the code of practice that sits under them, are what moved marking from research into shipping roadmaps this year. The Act asks for solutions that are effective, interoperable, robust and reliable as far as this is technically feasible (Article 50(2)). The whole engineering argument lives in that last clause.

How the mark gets in

Nothing is added to the text. No invisible characters, no odd spacing, no hidden metadata. If you copy a marked paragraph into a plain text editor and compare it byte by byte against an unmarked one, there is nothing extra to find.

What changes is who breaks the tie. A model writes one token at a time, and at many of those steps several continuations are equally acceptable. "The company reported" or "The company posted" are both fine, and nothing about the meaning depends on which one lands. A watermarking scheme takes those genuinely free choices and hands them to a keyed pseudo-random process instead of to plain chance. The text stays natural because only meaningless choices are touched. The detector, holding the same key, later asks how often the key's own preference won.

a watermark adds nothing to the textthe mark lives in the statistics of the choices, not in the lettersthe model hesitatesseveral words fit equally wellthe key picks oneinstead of plain chancethe detector countshow often that key wona watermark adds nothing to the textthe mark lives in the statistics of the choices, not in the letters
Where the mark actually lives. At the points where the model is genuinely undecided, the key rather than chance picks the winner, and the detector counts how often the key's preference came out on top. The text itself gains nothing that a byte comparison could find.

Interactive · one token at a time

Step through a sentence: at each point where two words fit equally well, the one on the key’s list wins. In the green-list scheme a quarter of the vocabulary sits on that list, so unmarked text lands on it about a quarter of the time, and a run of choices going the key’s way is what the detector is looking for. The two-way ties here are a simplification; the real count runs over the whole vocabulary, and one sentence proves nothing. The statistic only separates from chance over hundreds of tokens.

A detector recognises its own key

A detector is not a classifier that has learned what machine writing looks like. It recomputes a keyed pseudo-random process over the text it is given and asks whether the observed choices line up with that key more often than chance allows. That single design fact carries most of the practical consequences.

It means a detector is blind to every scheme but its own. In my runs, text carrying one scheme's mark is statistically indistinguishable from human writing when the other scheme's detector reads it, and that holds in both directions. It also means there is no such thing as an "AI detector" in the watermarking sense: there are only per-vendor, per-key verifiers, and none of them can speak about another vendor's model.

It goes further than the scheme, too. The split of the vocabulary is drawn by a device-local random number generator, so the same key and the same seed give a different split on a GPU, on Apple's MPS backend and on CPU, which is why the notebook tells you to score on CPU.

So the honest reading of a clean negative result is narrow: this text does not carry this key, which is not the same as no model having written it.

What a detector can honestly say about one document

Detection is a statistical test, so it comes with the two error rates every test has. A threshold is chosen, texts above it are called marked, and two things can go wrong: a human text lands above the line, or a marked text lands below it. The threshold I use throughout, and the one Kirchenbauer and colleagues use, is a z score of 4.0. On the assumptions the test is built on, that corresponds to a false alarm about once in every thirty-one thousand documents.

Real prose does not satisfy those assumptions, and the control set showed it. My control was 250 human-written passages drawn from 86 Polish Wikipedia biographies. The loudest false alarm from the older of the two schemes came from a 399-token passage of the Beata Kozidrak entry, scored with one key on the Bielik-1.5B tokenizer at z = 6.156 against the threshold of 4.0. An encyclopaedia passage, written by people, comfortably over a line that is supposed to be crossed about once in thirty-one thousand documents. On one set of keys the newer scheme raised no alarm at all on the same corpus, with a highest score of 3.24. That contrast is worth stating plainly, because it says the failure belongs to a scheme rather than to watermarking as an idea.

Move the slider below to see how much room there is between a verdict and its opposite.

Interactive · what an edit does to the verdict

below the threshold: this key is not confirmed

paraphrased · median z = 3.53 · detected in 41.7% of runs

The same marked text, edited four ways, scored against the same key and the same threshold. Untouched it sits at a median z of 17.46 and is caught every time. A paraphrase brings it to 3.53, under the line, and detection drops to 41.7% of runs. The four stops are separate measurements rather than a continuous curve. SynthID-Text on Bielik-1.5B, temperature 1.0, 60 generations per attack.

The strongest sentence a detector supports is narrower than the one people want to write. It is this: the text matches the watermark pattern of this model and this key. That claim is weaker than “a machine wrote this”, and on its own it is not grounds for any decision about a person.

What survives an edit

How much survives an edit turned out to belong to the scheme rather than to watermarking, and the two schemes I measured sit at opposite ends. The older green-list design keys on a single preceding token, which makes it crude but, on a small model, hard to shake off: on Bielik-1.5B it survived every edit I threw at it. On the 11B model the same scheme is far weaker, and even the untouched control only scores 75%. The newer design keys on a longer window, which is what lets it leave the text's quality untouched, and also what makes it tear when that window is disturbed.

same edit, two schemes, opposite verdicts% detected · Bielik-1.5B · T=1.0 · up to 60 generations per attackgreen list (2023)SynthID (2024)untouched100%100%paraphrased100%41.7%half the words deleted100%18.3%same edit, two schemes, opposite verdicts% detected · Bielik-1.5B · T=1.0 · up to 60 generations per attack
What is left after the same edit, applied to each scheme's own marked generations from Bielik-1.5B. Paraphrasing leaves the green list detecting every run and the newer scheme detecting 41.7% of them. Deleting half the words leaves the green list at 100% and the newer scheme at 18.3%. Same edit, two schemes, opposite verdicts.

Two things I will not do with those numbers. I will not read them as a ranking of schemes, because the older one buys its robustness with a measurable cost to the text it marks. And I will not turn them into a recipe: the work measures how much signal an edit leaves behind and stops there. There is no removal pipeline in the repository and there will not be one.

If you are writing an AI policy

The practical payoff of the whole session fits into three rules, and they are the part I would keep if everything else here were forgotten.

  1. Detection is circumstantial evidence, never the sole basis for a decision about a person. A term paper, a job application or a disciplinary case needs more than a verifier's output, and the verifier's output needs to be quoted with its threshold attached.
  2. Calibrate the threshold on your own corpus and your own tokenizer. A declared false-alarm rate is computed under assumptions your documents do not meet. Measure your own rate on your own texts; the notebook has a cell for exactly that.
  3. Absence of signal proves nothing. No mark means the text does not carry this key. It does not mean a human wrote it, and it does not mean the text was never near a model.

If your organisation is drafting a policy that references AI detection, those three lines are worth pasting in ahead of any tooling decision.

What I am looking at next

Two directions. One is calibration as a shippable artefact rather than a warning: a per-tokenizer pack that tells you where your own threshold actually sits, because a declared p-value that is wrong by orders of magnitude is worse than no number at all. The other is register: everything here was measured on encyclopaedic prose, and contracts, support replies and code behave differently enough that I would not extrapolate. Stay tuned.

A write-up of „Czy da się dowieść, że ten tekst napisała AI?”, a workshop given for the AI Ninjas community on 26 August 2026, with thanks to the Dojo for the invitation. The measurements behind it are my own runs on local models; the notebook and the deck are in the repository above.

Workshop materials → Kirchenbauer et al., green-list watermark → SynthID-Text, Nature 2024 → Shumailov et al., model collapse →

Part of the same line of work as my papers on what Polish models know about their own knowledge, A confidence dial you can read, and turn and Does Bielik know what it doesn't know?, and of the Eskadra Bielika effort to build sovereign Polish language models.