Research · LLM reliability · July 2026
A confidence dial you can read, and turn
A Polish open model's sense of what it knows is graded rather than all-or-nothing, it holds up across languages, and in one model it is a setting you can turn up or down.
The short version
My first paper showed that a model can tell a real name from a made-up one before it answers. This one asks how fine-grained that sense is, whether it survives across languages, and whether you can act on it. The familiarity signal turns out to be a dial rather than a switch: it rises smoothly with how well-known an entity actually is. The dial is sharper in models adapted to Polish than in larger general-purpose ones, so it comes from language training more than from size. It reads the same whether you ask in Polish or English. And in the one model that naturally declines to answer, a single internal direction controls that decision, so refusal is a setting you can turn. The models with the strongest internal sense, though, never use it.
From a switch to a dial
In the first paper the question was binary: does the model recognise this entity or not. It does, and you can read that cheaply from the model's internal activations before it writes anything. The more useful question stays open. When a model recognises something, does it have a sense of how well it knows it, or is recognition all-or-nothing? To find out I built a larger, harder test: 1,440 Polish entities across four subject areas, graded by real-world popularity, run through twelve models from four families, including Polish-adapted models and general-purpose ones of similar or larger size.
Finding 1: a dial, not a switch
The familiarity signal is graded. I sorted 1,200 real Polish entities into ten bands by how often their Polish Wikipedia page is viewed, from the obscure long tail up to household names, and added invented names as a floor. In Bielik-11B the internal familiarity score climbs steadily across that whole range, tracking real-world popularity closely (a correlation of 0.57). The model does more than separate real from fabricated: it carries a smooth sense of how well-known something is, available in a single forward pass before it answers a word.
For anything you build on a model, that turns a yes-or-no check into a confidence level. You can treat a household name and a barely-documented local figure differently, and send the low-confidence, high-stakes questions to a document lookup or a person instead of letting the model guess.
Finding 2: the dial comes from Polish training, not size
The graded sense is not spread evenly. It is strong in the two Polish-adapted families, Bielik and PLLuM, and close to flat in the general-purpose Gemma-4 and Qwen3, even though those match or exceed them in size. Qwen3 barely grades whether it has 1.7 billion parameters or 14 billion, so raw size is not the cause.
The clearest evidence is a before-and-after. Take a base model and continue training it on Polish text, and the dial sharpens. Llama-3.1-8B grades at 0.19; after Polish adaptation, the PLLuM-8B built on it grades at 0.36, and the Mistral-based PLLuM-12B shows the same jump over its own base. The language training is what installs the finer sense, on the same architecture and at the same size.
This matters for anyone deploying AI in a language other than English. A model adapted to your language carries a more reliable internal sense of what it does and does not know about local entities, which is exactly where a general model is weakest. For Polish, that is a concrete argument for the Bielik and PLLuM line of work.
Finding 3: the read is about the entity, not the language
If this signal were a quirk of Polish grammar or tokenisation, it should fall apart when the question changes language. It does not. I kept every entity name unchanged and swapped only the surrounding question, from "Kim jest…?" to "Who is…?", then trained the probe on one language and tested it on the other. It kept 96 to 101 percent of its ability to tell real names from invented ones. The model's sense of familiarity attaches to the entity itself, not to the words around it.
In practice one guardrail then covers users who mix languages, which is common in Polish business settings where people move between Polish and English mid-conversation.
Finding 4: refusal is a knob you can turn
Reading the signal is one thing. The bigger surprise is that you can act on it directly. Gemma-4-12B was the only model in the study that ever declines to answer on its own. Inside it, I found a single direction, at a single layer, that controls that behaviour. Add it and the model becomes more cautious; subtract it and it becomes more willing.
The effect is clean and works in both directions. Pushed toward "unfamiliar", the model starts refusing even well-known people, with refusal on famous entities rising from 24 percent to 100 percent. Pushed toward "familiar", it stops refusing the ones it should, confidently answering about people who do not exist, with refusal on invented entities falling from 73 percent to zero. A random direction of the same strength never does this; it just breaks the output. So this particular direction is the handle for abstention.
The practical version: you do not have to retrain a model to change how cautious it is. If a model already has some refusal behaviour, you can tune it with a small, cheap intervention and set the balance between answering too much and refusing too much where your use case needs it.
The gap that matters
Put the pieces together and a gap appears. The models with the best internal sense of what they know, the Polish-adapted ones, never actually say "I don't know". In 1,600 answers each, Bielik and PLLuM refused zero times. The model that does refuse, Gemma-4, has a weaker sense, and it refuses by topic rather than by knowledge: it is cautious about biographies of real people (declining 52 to 64 percent of the time) and almost never cautious about cities (3 percent). Its caution is a blanket policy about a subject, not a judgment about whether it knows the specific answer.
Having the knowledge signal and using it are two separate things. The signal is cheap to read and, as the knob shows, possible to wire into behaviour. That gap is the opening for a practical safety layer.
Why this matters if you ship models
The through-line is a signal you get almost for free. It reads before the model generates anything, in a single forward pass, so it costs about one extra read rather than the several full generations that sampling-based checks need. In my tests a calibrated familiarity probe held its own among other pre-generation methods, though methods that inspect the finished answer still predict outright errors better on average, so the honest picture is that this is one useful layer, not a complete solution.
Where it earns its place is routing and caution. A Polish assistant in finance, law, or support can read the familiarity score for a name before answering, and when it is low, pull in supporting documents or route to a person. Because the score is graded, you can make that call by risk instead of a single fixed threshold. It survives a language switch, so one mechanism covers mixed-language users. And the underlying direction is steerable, so a model that already refuses can have its caution tuned to the setting rather than left at the vendor's default. On the long-tail benchmark in this study the base error rates were severe, above 80 percent for most models, so knowing when not to answer is worth a great deal.
How I tested it
The dataset is 1,440 Polish entities across four subject areas: athletes, cities, writers, and musicians. Twelve hundred are real, drawn from Wikidata and sorted into ten popularity bands by Polish Wikipedia pageviews, and 240 are invented Polish names, screened so their length and spelling cannot give them away. I ran all twelve models from the Bielik, PLLuM, Gemma-4, and Qwen3 families, measuring at the final token of a one-sentence question, in a single forward pass before any answer. Familiarity is read with a small trained probe on the model's hidden state or with an unsupervised spread measure, and every number uses fixed seeds and held-out selection. Full detail is in the paper.
What I'm looking at next
One question is left wide open. Why does a model with a strong internal familiarity sense still never use it, while another with a weaker sense does? Something must connect the signal to the model's behaviour, or fail to. Finding that mechanism is what I am working on now, along with whether the caution you can steer from outside can also be trained in so it stays. More on that when it is ready. Stay tuned.
Grzegorz Brzezinka. Graded Entity-Familiarity Readouts in Language Models: Polish Adaptation, Cross-Language Robustness, and Refusal Steering. arXiv:2607.13568, 2026.
@article{brzezinka2026graded,
title = {Graded Entity-Familiarity Readouts in Language Models: Polish
Adaptation, Cross-Language Robustness, and Refusal Steering},
author = {Brzezinka, Grzegorz},
journal= {arXiv preprint arXiv:2607.13568},
year = {2026}
}
Reads on from my first paper, Does Bielik know what it doesn't know? Part of the Eskadra Bielika effort to build sovereign Polish language models.