Why loss is not zero

Conditional entropy falls as context grows. That is not controversial.

If the prompt is only “After tomorrow is”, the calendar is unpinned. Supply “Today is Sunday” and Tuesday is the unique calendar answer. Ambiguity was unfinished context. Semantic collapse is real.

What does not follow is: therefore the next token becomes unique. All of these can be true on Sunday:

• After tomorrow is Tuesday.
• After tomorrow’s Tuesday.
• After tomorrow it will be Tuesday.
• After tomorrow is the 16th.
• After tomorrow? Tuesday.

Same fact. Different tokens. The training objective is not “recover the unique true proposition.” It is “assign probability to the exact next token that happened to be in this document.”

Semantic collapse. Given enough context, the intended meaning often becomes unique. “After tomorrow,” plus “today is Sunday,” plus “we are talking about weekdays,” leaves one date.

Realizational leftover. How that meaning gets worded, whether the speaker continues the sentence at all, what they mention next, in what register — that usually stays open.

A Born-rule picture is a fair analogy for sense: many readings exist while the sentence is incomplete; context makes one reading definite. It is a weaker analogy for the next token, because the “measurement” here is “whatever this particular writer typed,” and many typings are compatible with the same collapsed meaning.

A huge fraction of remaining loss is missing context, not linguistic randomness. The data-generating process is many speakers, not one oracle with one lawful continuation. The model is not given the full world: it gets a window of text, not “it is Sunday and the speaker intends to name the weekday.”

Shannon already measured this with humans (1951). People who had seen the preceding text still could not guess the next character with certainty. Human conditional entropy of English is low, not zero. An LLM is trying to match that same process.