Frontier models read a short document almost perfectly, so a 128K-token window looks dependable end to end. But ask the same questions in your own words, and it becomes clear the reliable part of that window is far shorter than the box advertises.
Published ·Updated
01 / 04
Ask a model 100 questions about a short document, using the document's own words, and it answers 99 of them right. This is the kind of demo you see in launch posts, and it makes the model look like it has truly read every page. But the catch is that each question repeats the document's exact wording, so the model can locate the answer the way Ctrl+F does in a document, i.e. it acts, to some extent, like direct text matching rather than understanding it.
02 / 04
Now ask those same 100 questions the way a person actually would i.e., in your own words, and grow the document to about fifty pages. The same model now gets only 47 right. Basically, more than half of what it got a moment ago has flipped to wrong. Nothing about the questions got harder, it is just that there is simply more text to hold at once. What we can understand from this is a model spreads a fixed amount of attention across all of it, so the few sentences that carry each answer get a thinner and thinner slice.
03 / 04
This is not just one weak model. If you take the twelve models that advertised a 128K-token window in 2025 and give each the same fifty-page test then ten of the twelve keep less than half of their short-document score, and only two managed to stay above. What about the newer models? Well they move the wall without removing it. Measurements from March 2026 find most current models reliably use about 50 to 65 percent of the window they advertise. So the fade is basically built into how these systems read long inputs.
04 / 04
So, what is the solution? No spec sheet prints this number, so you measure it yourself. First decide how accurate your task needs the model to be, and draw that as a line. The green line here is set at 90 right out of 100. Then ask your questions again as the document grows, at 2 pages, then 10, 25, 50, and 100. Your usable window is the length where the score first falls below your line, which is between 10 and 25 pages in this example. Past that point you pay full price for every token and get less back, so keep each prompt inside the window you measured and pull in only the passages the task needs.
Every number in the grid is a reported rate shown across 100 questions. The 99 is the exact-words demo score, and the 47 renders the typical model below-half result once questions are reworded and the document grows long. In the model chart, which of the twelve keep above or below half of their short-document score is the reported 2025 finding, while the exact height of each column is drawn to read, and March 2026 RULER measurements still find most current models using about 50 to 65 percent of their advertised window. The final staircase is a worked example of the measurement, its heights and its 90 line are illustrative, and the crossing for your task and your model has to come from your own test. Lengths use roughly 650 tokens a page, so 32K is about 50 pages and 128K about 200.