Chain-of-Thought Prompting: Teaching AI to Show Its Work

The fastest way to get a smarter answer out of an LLM isn't always a smarter question. Sometimes you just have to ask it to slow down.

In January 2022, Jason Wei and a group of researchers at Google Brain published a paper with a finding that looked almost trivial. If you put step-by-step reasoning examples in your prompt, large language models get dramatically better at multi-step problems. Math word problems, logic puzzles, commonsense reasoning. Tasks where models had been failing went to near-human accuracy. They called it chain-of-thought prompting, and it changed how a lot of us write prompts.

The original insight

The paper, "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models," showed that few-shot prompts containing reasoning traces — not just final answers — unlock latent capability in large models.

The mechanic is easy to miss, so let's be precise about it. In few-shot prompting your prompt contains a worked example — a question and its answer — followed by the question you actually want solved. What chain-of-thought changes is not the question you ask. It changes how the example's answer is written. This is Figure 1 of the paper, with the model's reply marked:

Standard prompting — the example shows only the final answer:

Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls.
   Each can has 3 tennis balls. How many does he have now?
A: The answer is 11.

Q: The cafeteria had 23 apples. If they used 20 to make lunch and
   bought 6 more, how many apples do they have?
A: The answer is 27.        <- model's reply: WRONG

Chain-of-thought prompting — same questions, but the example now shows its reasoning:

Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls.
   Each can has 3 tennis balls. How many does he have now?
A: Roger started with 5 balls. 2 cans of 3 tennis balls each is
   6 tennis balls. 5 + 6 = 11. The answer is 11.

Q: The cafeteria had 23 apples. If they used 20 to make lunch and
   bought 6 more, how many apples do they have?
A: The cafeteria had 23 apples originally. They used 20 to make
   lunch, so they had 23 - 20 = 3. They bought 6 more apples, so
   they have 3 + 6 = 9. The answer is 9.        <- model's reply: RIGHT

The model imitates the format it was shown. Give it an example that jumps straight to a number and it jumps straight to a number — and flubs the two-step arithmetic. Give it an example that works through the steps and it works through the steps on the new problem too. A few extra words in the exemplar. On the GSM8K math benchmark, that change pushed PaLM 540B from 18% to 57% accuracy. That's not a marginal improvement. That's the difference between unusable and useful.

One important caveat from the paper: the trick only works at scale. Models below roughly 100 billion parameters showed almost no benefit. The reasoning capability looks like an emergent property of scale. The knowledge is there in smaller models, but you can't get it out through prompting alone.

Zero-shot CoT: the magic phrase

Later in 2022, Kojima et al. found something even weirder. You don't need carefully crafted examples. Just append "Let's think step by step" to the prompt and the model starts reasoning.

Q: A store has 4 shelves. Each shelf holds 8 boxes. Each box contains 6 items.
   How many items are in the store?

Let's think step by step.

The model produces intermediate steps: "4 shelves times 8 boxes is 32 boxes. 32 boxes times 6 items is 192 items." Zero-shot CoT is less reliable than few-shot — the chains are sometimes sloppy — but it costs zero prompt engineering effort. Five words, and the model tries harder.

Why does this work? The most plausible explanation is that LLMs absorbed millions of examples of step-by-step reasoning during training. Textbooks, tutorials, Stack Overflow answers, math solutions. The phrase "let's think step by step" activates a distribution of text that tends to include intermediate reasoning, and the model follows that distribution.

Watch the magic phrase work

Below are two real recorded runs of the same model (Gemini Flash, with its native thinking feature switched off in both runs, so what you see is pure prompting). Same question — count the letter "e" in a sentence — and the only difference is how each request opens. One begins "Answer with only the final number, no explanation". The other begins with the magic phrase itself: "Let's think step by step". Nothing is simulated and no LLM is called when you press play; these are recordings of the actual runs.

"Just the answer" produces a confident number in about a second — and it's wrong. Run it again and you get a different wrong number; separate attempts gave us 17, 18, 19 and 22. "Let's think step by step" makes the model spell the sentence out word by word, add up the per-word counts, and land on the right total: 21.

Counting letters is a deliberately unfair task for a language model — it reads tokens, not characters, so the direct answer is a guess dressed up as a fact. That's what makes it a clean demonstration: the step-by-step run doesn't make the model smarter, it makes the model write the problem down, and the answer falls out of the written steps.

Why chain-of-thought actually works

The deeper reason is computational. A Transformer generates one token at a time, and each token generation is a fixed-depth computation — a constant number of layers doing a constant number of operations. For a simple lookup like "What is the capital of France?", that's more than enough. For a multi-step problem, the model needs more compute than a single forward pass gives it.

Chain-of-thought spreads the computation across multiple token positions. Each reasoning step is a new token, and each token generation gets a full forward pass through the network. By writing out "5 + 6 = 11" as intermediate tokens, the model performs the addition during generation instead of trying to do it all in one shot.

Asking a model to jump from question to answer in a single step is like asking a person to multiply two five-digit numbers in their head. Writing out the intermediate steps gives the model the same scratch space a human gets with paper and pencil. That's exactly what you watched in the replay above — the numbered letter-by-letter breakdown is the scratch space, built one token at a time.

It also explains why chain-of-thought sometimes fails. If the model makes an error in an intermediate step, it will confidently carry that error through the rest of the chain. The chain is only as strong as its weakest link.

Variants and extensions

Chain-of-thought spawned a small family of techniques.

Self-consistency

Instead of generating a single reasoning chain, sample multiple chains with temperature greater than zero and take a majority vote on the final answer. If 7 out of 10 chains land on 192, that's probably the answer, even if 3 chains went off the rails.

Self-consistency adds a real chunk of accuracy. On GSM8K, it pushed PaLM 540B from 57% (single chain) to 74% (majority vote). The cost is obvious: you're running inference N times per question.

Tree of Thoughts

Instead of a linear chain, explore a tree of reasoning paths. At each step, generate multiple candidate next-steps, score which are most promising, and continue from the best ones. Beam search applied to reasoning.

Tree of Thoughts is good at problems with dead ends, where you might need to backtrack. The canonical example is the "Game of 24": given four numbers, use arithmetic operations to make 24. A linear chain commits to a wrong first move. A tree can explore several first moves in parallel.

Program-of-Thought

Instead of reasoning in natural language, have the model write code that solves the problem and then run the code. The model handles the problem formulation; a reliable interpreter handles the math.

Q: If a train travels at 60 mph for 2.5 hours, then 80 mph for 1.75 hours,
   what is the total distance?

# Let me write code to solve this:
distance_1 = 60 * 2.5   # 150 miles
distance_2 = 80 * 1.75  # 140 miles
total = distance_1 + distance_2  # 290 miles
print(total)  # 290

This eliminates arithmetic errors completely and is especially good for problems that require precise calculation.

Limitations

Chain-of-thought isn't a silver bullet.

Verbosity. Reasoning chains burn tokens, which means more latency and more cost. For simple questions, the overhead isn't worth it.

Hallucinated reasoning. The model can produce plausible-sounding intermediate steps that are logically invalid. The chain "looks right" but lands on the wrong conclusion. That's arguably worse than a direct wrong answer because it gives the user false confidence.

Faithfulness. There's an open debate about whether chain-of-thought reasoning reflects what the model is actually doing, or is a post-hoc rationalization. The model might arrive at an answer through its internal representations and then generate a reasoning chain that justifies it. Humans do this too.

Diminishing returns at extreme scale. As models get bigger and more capable, they solve more problems correctly even without chain-of-thought, and the gap closes.

From prompting to native reasoning

The most interesting evolution of chain-of-thought is that it has been absorbed into training. OpenAI's o1 and o3 models don't need you to prompt "think step by step." They do it natively, producing internal reasoning tokens before answering.

These reasoning models are trained with reinforcement learning to produce useful thinking traces. The reasoning happens on a hidden scratchpad. You see the final answer and optionally a summary of the reasoning, but not the raw chain. The model decides when and how much to reason instead of waiting on a prompt to trigger it.

This is a pattern that keeps repeating in AI: a prompting trick reveals a capability, researchers figure out how to bake the capability into training, and the next generation of models does it by default. Chain-of-thought was the insight. Reasoning models are the product.

What started as a prompting hack is now a foundational part of how the most capable systems think. The takeaway is plain: give the model room to work through the problem, and the answer gets better.