A long conversation with an assistant eventually betrays you. In the opening message you asked for a formal register and no emojis; forty messages later the tone has loosened and the first emoji has surfaced. Nothing has malfunctioned. The drift follows directly from how large language models read text: at any moment they can only see a bounded slice of it, counted in tokens and held in what the industry calls the context window.
- Tokens are not words: providers estimate roughly four characters, or about three-quarters of an English word, per token.
- The window is large but finite: frontier APIs in October 2026 advertise around one million tokens, and that budget covers input, invisible reasoning and output together.
- Capacity is not attention: models retrieve information at the start and end of a long input far better than material buried in the middle.
Anatomy of a token
Before a model reads anything, it chops the text into fragments. A short, common word such as “the” counts as a single token in OpenAI’s tokenizer, while “tokenization” splits into “token” and “ization”. OpenAI puts one token at about four characters, or 0.75 words; Google reckons 100 tokens at between 60 and 80 English words in its Gemini models. Both companies scope those equivalences to English and publish no figure for other languages, so treat them as orientation rather than law.
One cost rarely captured in estimates: reasoning models burn internal tokens before they answer. OpenAI’s documentation states that these reasoning tokens stay invisible, yet they occupy window space and are billed as output. A terse reply can consume far more capacity than its length suggests.
Working memory, not a filing cabinet
Anthropic describes the context window as working memory, deliberately distinct from the data a model was trained on, and Google borrows the same analogy of human short-term recall. In practice everything piles up inside it: your prompts, the replies, uploaded documents, system instructions. Each new turn drags the previous ones along.
The published numbers for October 2026 are generous. OpenAI’s GPT-6 Astra, GPT-6.1 Sol and GPT-6 Luna list 1.05 million tokens through the API. Anthropic’s Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5 reach one million, with other models at 200,000. Gemini 4 Argon sits at 1,048,576 tokens shared across input, reasoning and output. Paid claude.ai plans publish one million for the current models, 500,000 for some earlier ones and 200,000 elsewhere, which the company frames as roughly 500 pages of text or more. Google offers its own yardstick: a million tokens is about 50,000 lines of code or eight average-length English novels. Those figures describe the developer API; the consumer app you open every day may enforce different limits by plan.
Why a million tokens still forgets
Two separate mechanisms are at work. The first is capacity. As a thread approaches the ceiling, something has to leave: interfaces may run the window first-in, first-out, and paid plans with code execution can summarise older messages to free space. A summary keeps the gist but can discard the exact wording of an instruction you gave at the start.
The second is attention, and it bites even when plenty of room remains. Anthropic’s technical documentation calls the effect context rot, noting that precision degrades as token counts climb and that more context “isn’t automatically better”. Google concedes its models excel at finding a single fact in a long text but that when several facts must be combined, “the model does not perform with the same accuracy”. There is a compute tax too: because attention compares every fragment against every other, doubling input length can push processing cost toward four times, which is why windows grew gradually rather than overnight.
“Performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models.”
That finding comes from “Lost in the Middle: How Language Models Use Long Contexts”, published on arXiv in July 2023 by Nelson F. Liu and six co-authors and later accepted by the Transactions of the Association for Computational Linguistics. The curve is U-shaped, and your opening instruction slides steadily toward the sagging middle as messages accumulate.
Six habits that keep instructions alive
- Put the question last. Google’s own guidance for long prompts is to place the query at the end, after any pasted document.
- Restate critical rules. Tone, format and language constraints should be repeated every so often, which pulls them out of the middle.
- Start a new chat when the task changes. Mixed threads fill the window with noise; a fresh session seeded with a short brief works better than endless accumulation.
- Ask for a summary before closing. Save it and paste it into the next session, so you control what survives instead of leaving it to automatic truncation.
- Do not paste more than you need. If the model does not require those tokens, Google advises, “it is best to avoid passing them”.
- Work from scoped sources. Tools that ground answers in your own documents reduce the odds of key details dissolving in a marathon conversation.
The race toward ever-larger windows is marketed as infinite memory, and it is not. A million tokens is a vast warehouse, but attention still distributes unevenly across it, so the practical skill is deciding what enters the window and where it sits. That is also why curated, persistent memory layers for coding assistants are gaining traction: a small, ranked briefing loaded at the start of each session beats a transcript replayed in full. Bigger context has made assistants more capable; it has not made careful prompting optional.