What happens after you hit send

When you hit send, the model does two very different jobs. First it reads everything you sent, all of it at once. Then it writes the reply one small piece at a time, running the whole model again for every piece. The first job is quick. The second one is slow and expensive, and most of what makes an AI product feel fast or slow, cheap or pricey, comes down to which of the two you’re asking for more of. Which makes it the most useful thing to understand if you’re the one making it work for actual users.

If you’ve seen the context pile, this is the zoomed-in version of a single turn. There, an agent sent its whole pile every turn and the model re-read all of it before deciding one small next move. This is what that one read, and the reply after it, actually involves.

Quick vocab before the picture. The small pieces are tokens, chunks of text a bit smaller than a word. OpenAI’s token explainer has the rough rule for English: “100 tokens are approximately 75 words.”

Reading is fast, writing is slow

Reading is called prefill. The model gets the whole prompt at once and works through it in parallel, which is exactly what GPUs are good at. Writing is called decode, and it can’t go in parallel the same way, because word 50 depends on word 49. Hugging Face’s explainer puts it plainly: “Autoregressive model generation can be slow because it makes a prediction one token at a time.” A few paragraphs of answer is hundreds of trips through the model, back to back.

Each trip doesn’t redo the reading, thankfully. The model keeps notes on every token it has processed, called the KV cache, “reused for subsequent tokens, avoiding the need to recompute.” But every new token still looks back across all of them, doing a little math while pulling a lot out of memory. vLLM, the big open-source serving engine, calls the two jobs “compute-bound (prefill) and memory-bound (decode)” in its tuning docs.

You feel this as two waits: the pause before anything appears (time to first token, mostly reading) and how fast the words stream after that (all writing). OpenAI’s latency guide is blunt about which matters more: “cutting 50% of your output tokens may cut ~50% of your latency,” while “cutting 50% of your prompt may only result in a 1–5% latency improvement,” unless you’re dealing with “truly massive context sizes,” which is kinda what the agent pile turns into by turn 60.

So when someone says their AI feature is “slow,” the first question is slow to start or slow to finish. A long wait before the first word points at reading: a giant prompt, a cache that keeps missing, a crowded server. Words that dribble out point at writing: answers that are too long, too much thinking, a slow tier. Mix them up and you can spend weeks trimming a prompt that was never the problem.

Which is why the answer costs more

All four APIs I checked price this gap into the menu. Anthropic’s Claude Opus 5.5 is $4 per million input tokens and $20 per million output. OpenAI’s gpt-6.1-sol is $2 and $10, Moonshot’s Kimi K3 $3 and $15, Z.ai’s GLM-5.3 $1.4 and $4.4. Writing costs 5 times reading on three of them, a bit over 3 on GLM. None of the pricing pages say why. My read is the memory thing above: writing leaves a lot of the chip idle, which is why vLLM mixes reading and writing into the same batch for “better GPU utilization.”

For anyone scoping a project, this flips the intuition. A tool that reads a 50-page contract and writes one paragraph is cheap. A tool that turns a one-line brief into five pages is the pricey one, even though it “uses less data.” Model the output, and remember the cheapest optimization in AI is still asking for shorter answers.

Busy servers, and speed as a menu

Nobody gets a GPU to themselves. Servers batch lots of people’s requests and reshuffle after every token, so finished replies leave and new ones hop in (vLLM lists “continuous batching of incoming requests” at the top of its features). Sharing means trade-offs: vLLM “batches all pending decode requests before scheduling any prefill operations,” so people mid-answer keep streaming while new arrivals wait longer for their first word. Big open-model deployments now go further and put reading and writing on separate GPUs, which vLLM’s docs say is for steadier waits and refreshingly admit “DOES NOT improve throughput.” The vLLM and SGLang recipes for Kimi K3 ship that split ready-made.

Hosted providers sell the same trade-off as a menu. Anthropic’s fast mode gives “up to 2.5x higher output tokens per second,” explicitly “not time to first token,” at $8 and $40 on Opus 5.5 instead of $4 and $20. OpenAI’s pricing page lists five tiers for the same model, from Batch and Flex at half price “in exchange for slower response times” up to Ultrafast at six times Standard for gpt-6.1-sol. Paying six times more for the same model to talk faster is a very 2026 sentence.

This is build-vs-buy in disguise. Hosted, you rent speed per request: overnight jobs on Batch, the customer-facing chat on a faster tier. Self-host an open model and you pay for GPUs whether they’re busy or not, and every trade-off here becomes a config line someone on your team owns.

The speed tricks, and who controls them

Prompt caching, the frosty blocks from the context pile: if the start of your prompt matches something read recently, the server reuses that reading. It trims the first-word wait and the input bill (cache hits on Opus 5.5 cost 5% of normal input), and does nothing for streaming speed. On open models the cache can now spill past the GPU; SGLang’s K3 recipe tiers it into regular memory and storage “for long multi-turn workloads.”

Quantization stores the model’s numbers with fewer bits, which vLLM’s docs describe as trading “model precision for smaller memory footprint,” so less to haul around per token. In 2026 it’s moved into training: the Kimi K3 card says it “applies quantization-aware training from the SFT stage onward, using MXFP4 weights,” a 4-bit format.

Draft models are the fun one. A small model guesses the next few tokens, the big one checks them all in a single reading-style pass and keeps the ones it agrees with, so a good guesser gets you several tokens per big trip. vLLM’s docs call it “theoretically lossless up to the precision limits of hardware numerics,” and K3’s SGLang recipe ships a 4.5 GB draft model, Kimi-K3-DSpark, guessing for the 2.8-trillion-parameter one. It works best at “medium-to-low QPS,” though: one of K3’s vLLM benchmark setups guesses 7 tokens ahead with one request in flight and only 3 with 12 to 14. Busy GPU, fewer guesses.

When you’re the one advising, the useful split is who owns what. On a hosted API your levers are prompt structure (so the cache hits), answer length, effort, and speed tier. Quantization, draft models and split serving are the provider’s business. On open weights it’s all yours, which is either the whole point or a full-time job, depending on the team.

Thinking is just more writing

Open models make this part easy to see. When a model “thinks,” it’s writing, same loop, one token at a time, and on open models it’s literally text between tags: SGLang’s parser docs list Qwen3 and DeepSeek wrapping it in <think> … </think>, which the server splits out before you see the answer.

So it’s billed like writing. Anthropic’s docs say thinking is “billed as output tokens,” and “you are billed for the full thinking process, not the thinking content visible in the response.” OpenAI’s reasoning guide says the same. And the budget detail nobody reads: the GLM-5.3 card says reasoning_effort “defaults to max if not passed,” and Kimi K3 “always has thinking enabled,” also defaulting to max. Leave it alone and every simple request gets the deluxe treatment. Very generous of them.

Steering the thinking

Steering is this year’s big add, and the open-source docs show what it actually is. Effort is a note written into the prompt: SGLang’s thinking budget docs say reasoning_effort is “forwarded into the chat template,” and call these “hints to the model, not token guarantees.” Anthropic says its effort is “rendered into the prompt, so changing it between requests invalidates cache breakpoints.” Change effort mid-conversation and you’ve edited the front of the pile, so the frosty blocks melt. A hard cap is blunter: once thinking hits the budget, “vLLM forces the model to produce reasoning_end_str,” per its reasoning docs. Basically it writes “ok, done thinking” on the model’s behalf.

Steering mid-thought on an open server is stop, add your correction, continue. vLLM’s continue_final_message means “the model will continue this message rather than starting a new one,” and with prefix caching on, the untouched part is still cached. Just steer forward, since Z.ai’s docs warn that editing past thinking means “cache hit rates may be affected.”

The chat apps wrap this in buttons. ChatGPT’s GPT-5.4 Thinking shows “an upfront plan of its thinking, so you can adjust course mid-response,” GPT-5.6 Sol added “a new slider to choose how much thought ChatGPT puts into a response,” and GPT-6, released October 7, “can also begin answering while it continues to think” (release notes). Claude’s app has an effort menu by the send button, where higher effort means “more thorough responses, but they take longer and use more tokens” (help page), and Anthropic’s API got a September beta that changes effort mid-conversation “while preserving the prompt cache” (release notes).

Tokens are the ROI, especially for agents

A chatbot answers once. An agent makes dozens of requests per task, each re-reading the whole pile (input) and writing a tool call, often after some thinking (output). For agents, tokens are the business case.

Anthropic’s pricing page has a worked example: a one-hour coding session on Claude Opus 5 with 50,000 input tokens and 15,000 output. Output is under a quarter of the tokens but $0.375 of the $0.625 token cost. Cache 40,000 of those input tokens and the token cost drops to $0.445, with output now about 84% of it. The same mix on Kimi K3’s prices also lands at 84%, on GLM-5.3 about 73% (my arithmetic on their example, so illustrative). Once the pile is cached, the writing is the bill.

Thinking can be paid for twice. Anthropic bills earlier turns’ thinking that stays in context “as input tokens” on newer models, and Kimi K3 requires you to send its thinking back each turn, warning that “historical thinking content keeps occupying the context window and is billed accordingly.” Written at output prices, re-read every turn after. The good news, per Anthropic: in a tool-use loop “the first request after new user input typically carries most of the reasoning,” and steps that just process tool results can skip it.

So the number to track on an agent project is cost per finished task, not cost per token. The order of operations is pretty boring once you see it: cache the pile, keep answers and tool calls short, set effort per step instead of leaving it on max, and only then argue about which model is cheaper. Self-hosting changes the currency (GPU time, not a per-token bill), not the logic, and a hard thinking cap in SGLang or vLLM is the most direct lever you’ve got.

So yeah, the model reads like a speed-reader and writes like someone paid by the word. Which, technically, it is.

Further reading

← All posts