There's a strange moment that happens when people first grasp what a language model is actually doing: they realize it's not retrieving answers the way a search engine does, and it's not reasoning the way a human does — it's doing something weirder and more interesting than either. It's playing an extremely sophisticated game of "what word comes next," one token at a time, forever, until it decides to stop. That reframing changes everything about how you interact with these systems. What's Actually Happening at Inference Time When a language model generates text, it doesn't see your prompt and produce a response in one shot. It generates one token at a time, and each token it produces gets appended to the context before it generates the next one. The mechanism is this: the model takes everything in the context window — your prompt, any prior conversation, its own previous output — and produces a probability distribution over its entire vocabulary. Every possible next token gets a score. "The" might get 12%, "A" might get 8%, "However" might get 4%, and so on across tens of thousands of candidates. Then one token is sampled from that distribution, appended to the context, and the whole process repeats. This is sequential probability sampling, and it has a profound implication: the model isn't holding a complete thought in mind that it's translating into words. It's constructing meaning token by token, where each choice constrains and shapes all subsequent choices. The model that wrote "The capital of France is" has now dramatically narrowed the probability mass toward "Paris." The model that wrote "The capital of France is, surprisingly," has shifted that distribution toward something unexpected. Same underlying model, radically different next-token probabilities — because the context changed. The Distribution Is Everything The technical term for those raw scores before they become probabilities is logits, and the process of converting them to a proper probability distribution is a softmax function. But the more important concept is temperature — a parameter that controls how peaked or flat that distribution is. At low temperature (say, 0.1), the model almost always picks the highest-probability token, producing confident, predictable, often repetitive text. At high temperature (say, 1.4), probability mass spreads out, lower-probability tokens get sampled more often, and the output becomes more surprising, creative, and occasionally incoherent. Temperature doesn't change what the model thinks is likely — it changes how aggressively it commits to that belief. There's also top-p sampling (sometimes called nucleus sampling), which works differently: instead of flattening or sharpening the whole distribution, it cuts off the long tail of very-unlikely tokens and samples only from the tokens that together account for, say, 90% of the probability mass. This prevents the model from ever generating truly bizarre low-probability tokens while still preserving variety among plausible ones. Why This Changes How You Write Prompts Here's the insight that rewires prompt engineering: your prompt doesn't give the model instructions in the way a function call gives a computer instructions. It shifts the probability distribution over all possible continuations. When you write "Answer concisely," you're not flipping a switch — you're adding tokens to the context that make concise-style continuations more probable, because the training data is full of examples where that phrase preceded short answers. When you add "Think step by step," you're making chain-of-thought tokens more probable, because that phrase is statistically associated with methodical reasoning in the training data. This means prompt engineering is essentially distribution sculpting. Every word you add is a nudge toward or away from certain regions of output space. Vague prompts leave the distribution flat and diffuse — you get generic, averaged responses. Specific, contextually rich prompts concentrate probability mass around exactly the kind of output you want. A few-shot example isn't teaching the model anything new; it's demonstrating a pattern that makes similar patterns more probable in the continuation. --- The thing about inference is that the model isn't deciding anything in the intentional sense — it's sampling from a distribution you helped shape. Which means the quality of your prompt isn't about being polite or clever; it's about how precisely you've sculpted the probability landscape before the first token ever gets generated. A concrete way this shows up in practice: if you're getting rambling, unfocused outputs, the instinct is to add more instructions. But often the real fix is to show the model the shape of a good response — paste in an example, mimic the format you want, write your prompt in the register of the answer you're hoping for. You're not instructing a mind. You're conditioning a distribution.