Ask for just the answer and the model says 17. Ask it to work step by step and it says 21. The sentence hasn't changed:

Effervescent elephants delivered these excellent lemon meringues everywhere.

There are 21 instances of the letter when you count both uppercase E and lowercase e. The second response gets there by counting each word and adding the results. That small change makes the calculation visible, which is useful even before you decide whether to trust it.

These are two recorded runs from the same letter-counting demonstration. One prompt opens with "Answer with only the final number, no explanation"; the other opens with "Let's think step by step, then give the final answer". The scene configuration uses Gemini with its thinking budget set to zero, so the steps appear in the answer itself. The saved configuration doesn't pin an exact model version.

The direct response gives 17; the step-by-step response gives 21, counting the initial uppercase E. Recorded completion offsets are 1.975 seconds and 3.185 seconds respectively. Playback adjusts the pacing for readability; pressing play replays saved events without calling a model.

The correct response breaks the count into 4 + 2 + 3 + 2 + 3 + 1 + 2 + 4. You can check each part against the sentence. A strictly lowercase count would be 20, a reminder that even a counting question needs a clear rule.

Two selected runs show what changed in the visible work. They don't tell us how often either prompt succeeds. To understand why this technique is worth testing, it helps to look at the larger experiment that introduced it.

The original insight

In 2022, Jason Wei and colleagues introduced chain-of-thought prompting. A few-shot prompt contains worked examples before the new question. Their change was to include intermediate steps in those examples. The following pair preserves the paper's tennis-ball and cafeteria example, with the model's reply marked:

Standard prompting. The example shows only the final answer:

Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls.
   Each can has 3 tennis balls. How many does he have now?
A: The answer is 11.

Q: The cafeteria had 23 apples. If they used 20 to make lunch and
   bought 6 more, how many apples do they have?
A: The answer is 27.        <- model's reply: WRONG

Chain-of-thought prompting. Same questions, but the example now shows its reasoning:

Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls.
   Each can has 3 tennis balls. How many does he have now?
A: Roger started with 5 balls. 2 cans of 3 tennis balls each is
   6 tennis balls. 5 + 6 = 11. The answer is 11.

Q: The cafeteria had 23 apples. If they used 20 to make lunch and
   bought 6 more, how many apples do they have?
A: The cafeteria had 23 apples originally. They used 20 to make
   lunch, so they had 23 - 20 = 3. They bought 6 more apples, so
   they have 3 + 6 = 9. The answer is 9.        <- model's reply: RIGHT

The first example invites a jump to the answer. The second demonstrates how to keep track of the quantities before combining them. In the paper's PaLM 540B experiment, eight worked examples raised accuracy on the GSM8K math-word-problem benchmark from 17.9% to 56.9%. That is a substantial improvement, with plenty of wrong answers remaining.

The study found its strongest benefits in large models. Its roughly 100-billion-parameter observation describes the models and tasks tested then, not a minimum requirement for every later model. Even its results vary by task: PaLM 62B improved on GSM8K. Model size alone isn't a useful pass/fail test for your prompt.

Asking for steps without examples

Later in 2022, Kojima and colleagues studied a simpler approach: ask for steps without supplying worked examples, then extract a final answer. This is zero-shot chain-of-thought. The letter-counting replay uses that general idea. A toy arithmetic prompt makes the instruction easy to see:

Q: A store has 4 shelves. Each shelf holds 8 boxes. Each box contains 6 items.
   How many items are in the store?

Let's think step by step.

A correct response would calculate 4 shelves times 8 boxes to get 32 boxes, then multiply by 6 to get 192 items. The prompt asks the model to produce useful intermediate quantities before committing to a total.

A scratchpad you can check

Think of multiplying two five-digit numbers with paper and pencil. Writing down partial results reduces what you must keep track of at once. Generated text can serve a similar purpose for a language model: earlier steps become part of the context used to generate later ones.

There is also more computation involved. A Transformer passes each newly generated token through its layers. Extra tokens create additional opportunities to develop and use intermediate results. A reasoning step can span many tokens, and the work per token depends on the architecture and context; it isn't a fixed allotment of arithmetic per sentence.

In the replay, the per-word counts provide that scratchpad. They also expose mistakes if the model makes them. Language models often receive word pieces rather than individual characters, so exact character counting can be awkward. Writing the words out helps organize this example, but it doesn't turn the model into a deterministic counter.

The same mechanism can carry a mistake forward. If the model counts three letters where there are four, the final sum can be perfectly consistent with a wrong starting point. More explanation can make that answer more persuasive. I find the visible steps useful because they give me something specific to check.

They are also not a guaranteed account of the model's internal decision process. Turpin and colleagues showed that models could produce plausible explanations without disclosing prompt biases that influenced their answers. An explanation can help you inspect a calculation without faithfully describing everything that caused the response.

Variants and extensions

Once intermediate work is available, you can sample alternatives, explore branches, or hand the calculation to an interpreter. Each addresses a different failure mode.

Self-consistency

Self-consistency samples several chains and chooses the most frequent final answer. In a hypothetical set of ten responses, seven answers of 192 would win the vote. The following pseudocode illustrates the procedure; it assumes an answer parser and a function for choosing the most common result.

answers = [] for _ in range(10): response = llm.generate(prompt, temperature=0.7) answer = extract_final_answer(response) answers.append(answer) final_answer = most_common(answers) # majority vote

In the self-consistency study, PaLM 540B's GSM8K accuracy rose from 56.5% with greedy chain-of-thought decoding to 74.4% with 40 sampled paths. Those are the paper's results, not a benchmark of the ten-sample pseudocode above. More samples cost more inference, and several chains can still share the same mistake.

Tree of thoughts

Tree of Thoughts explores candidate intermediate steps, evaluates them, and searches through promising branches with lookahead or backtracking. In the "Game of 24", the task is to combine four numbers using arithmetic to make 24. Trying several first moves can recover from a dead end that a single chain might keep pursuing. The benefit comes with extra generation and evaluation work.

Program of thoughts

Program of Thoughts asks the model to express the calculation in code, then uses an interpreter to execute it. This illustrative train problem shows the division of labor:

Q: If a train travels at 60 mph for 2.5 hours, then 80 mph for 1.75 hours,
   what is the total distance?

# Let me write code to solve this:
distance_1 = 60 * 2.5   # 150 miles
distance_2 = 80 * 1.75  # 140 miles
total = distance_1 + distance_2  # 290 miles
print(total)  # 290

The interpreter evaluates the arithmetic in the program. The model still has to choose the right quantities, units, and formula. If it confuses hours with minutes, valid Python will calculate the wrong distance very efficiently. Executed code gives you a dependable calculation only after the formulation is correct.

From prompting to native reasoning

Reasoning models brought intermediate computation into training. In its 2024 description of o1, OpenAI explained how reinforcement learning trained the model to use internal reasoning before responding. The user-facing answer or reasoning summary is separate from that hidden chain. Asking for a visible explanation is therefore different from enabling a model's native reasoning capability.

For a model that already reasons internally, adding "think step by step" should still earn its place through testing. A longer visible answer can add cost and reading time without improving the result. Simple lookups often need very little explanation; a calculation with several dependent quantities benefits from a checkable derivation.

Test the answer, then inspect the steps

The letter-counting replay makes a useful promise: intermediate work can make a problem easier to solve and an answer easier to inspect. Whether it improves your system is a question you can measure.

Take representative questions with independently checked answers. Compare a direct prompt with a prompt that requests useful intermediate work, keeping the model and generation settings fixed. Repeat the comparison across enough cases and runs to avoid mistaking one selected success for reliability. Measure final-answer accuracy, latency, and token use. Inspect incorrect responses to see whether the failure came from misunderstanding the question, choosing the wrong steps, or calculating them badly.

For exact letter counts, an ordinary string-counting function is the better operational tool. For a problem that needs interpretation before calculation, the model's decomposition can be useful, especially when you can verify or execute its steps. Give it room to work where that helps, and keep the independent check that tells you whether it did.