Home / What a language model is, without the jargon

Meaning without a dictionary, and why order matters

Claude opens no dictionary, it associates each word with words of similar meaning and weighs each one against all the others at the moment of answering, which means the place of an instruction in the message changes its real weight in the answer.

Claude consults no dictionary to understand a word. It associates each word with the words of similar meaning it encountered during training, then, at the moment of answering, it weighs each word in the message against all the others to decide what matters most. This weighing is not uniform across the whole length of the message, the position of a piece of information changes the real weight it receives.

The beginning and the end are better retained than the middle

Anthropic has published a study on this exact point, carried out on very long messages. The central result fits in one sentence, placing an instruction right at the end of the message increases the chances that Claude remembers it at the moment of answering, because the end of the message carries more weight in the final calculation. The same study specifies that this finding is not a universal law, on another model tested, the observed relationship traced more of a dip shaped curve instead, with a beginning and an end equally well retained, and certain writing techniques, such as providing examples or asking for written reasoning before the answer, can actually work against a piece of information placed right at the end of a very long document.

Moving the instruction that really matters

The practical rule that follows from this stays simple despite the nuance, when a message goes beyond a few sentences, the most important instruction is best placed right at the beginning or right at the end, never buried in the middle of a long paragraph of context. Here is the same message, before and after this move.

Before: here is the minutes of the meeting, ten paragraphs of scattered notes,
summarise these minutes in three points for absent members, then the appendices.

After: summarise these minutes in three points for absent members.
Here are the minutes, ten paragraphs of scattered notes, then the appendices.

In the second version, the instruction opens the message, it does not wait to be found after ten paragraphs of context.

The principle seen in the previous lesson, a deeper model does not fix a poor structure, also applies here, the order of words acts on a lever distinct from the choice of model, within a single message.