Inside LLMs: Tokens, Context Windows, and How They Generate Text

Why PMs Need to Understand LLM Mechanics

As an AI PM you don’t need to train models, but you do need a working mental model of how LLMs operate. The three concepts that drive most product decisions—cost, latency, quality, and reliability—are tokens, context windows, and next-token generation. Understanding these lets you scope features realistically, estimate costs, and translate engineering reality to business stakeholders.

Tokens: The Unit of Everything

LLMs don’t read words or characters—they read tokens. A token is a chunk of text, roughly 4 characters or about 0.75 words in English. “Tokenization” is the process of splitting input text into these chunks using a learned vocabulary.

• Common words are often one token (apple). Rare words split into several (tokenization → token + ization).
• Whitespace and punctuation count. Code, JSON, and non-English text often use more tokens per word.
• Both input and output are billed in tokens. A 500-word prompt plus a 500-word answer is roughly 1,300 tokens.

Why it matters for product: pricing is per-token, so prompt design directly affects unit economics. Latency also scales with output tokens—long responses feel slow because the model generates one token at a time.

Context Windows: The Model’s Working Memory

The context window is the maximum number of tokens the model can consider at once—input prompt plus generated output combined. Models range from a few thousand tokens to hundreds of thousands or more.

• Anything outside the window is invisible to the model. The model has no memory between separate API calls unless you re-send the relevant text.
• “Memory” in chat apps is an illusion: the app re-sends prior conversation each turn until it no longer fits, then truncates or summarizes.
• Bigger windows cost more and can be slower; relevant context beats maximum context.

Why it matters for product: features like “summarize this 100-page doc” or “remember my preferences” require deliberate engineering—chunking, retrieval (RAG), or summarization—not just a bigger prompt.

How LLMs Generate Text: Next-Token Prediction

At its core, an LLM is a very sophisticated autocomplete. Given a sequence of tokens, it predicts a probability distribution over the next token, picks one, appends it, and repeats. That’s it.

• There is no database lookup and no guaranteed truth—just statistical prediction learned from training data. This is why models hallucinate plausible-sounding but false content.
• Temperature controls randomness: low (≈0) is more deterministic and focused; high (≈1+) is more creative and varied.
• Top-p / top-k limit which candidate tokens are considered, another lever on creativity vs. consistency.
• Output is generated sequentially (streaming), which is why you see text appear word by word.

Why it matters for product: because output is probabilistic, the same prompt can give different answers. Plan for variability with evaluations, guardrails, and human review where stakes are high.

Connecting Mechanics to Product Decisions

Your Deliverable: The One-Page Explainer

Create a single page for non-technical teammates that covers: (1) what a token is and why it affects cost, (2) what the context window is and why the model ‘forgets,’ (3) that the model predicts text and can be confidently wrong, and (4) three implications for our product. Use plain language and one concrete analogy per concept. Keep it skimmable—headers, short bullets, no jargon without a definition.