Prompting is still a skill

Consider an article summarizer with a simple instruction: "Summarize this article in three sentences." The source says a treatment helped a small group of patients, but the study had no control group. The summary reports the improvement and drops the limitation. It reads beautifully. It also leaves the reader with a stronger claim than the study supports.

This is an illustrative example, not a recorded experiment. It captures the problem that makes prompting a useful engineering skill: deciding what the system must preserve, explaining that requirement to the model, and checking whether the change works beyond the one example that annoyed you.

I think the debate about whether "prompt engineer" deserves its own job title misses that work. Query design remains a skill even when nobody on the team has "query designer" on a business card. Prompt design has the same practical test: does the system behave better for the people using it?

Give the model a decision it can follow

"You are an expert summarizer" doesn't tell the model what to do when brevity competes with accuracy. In our example, the missing instruction concerns which information can safely disappear.

A more useful revision would say: "Summarize the main finding in three sentences. Preserve any limitation that materially changes what the finding establishes, such as the absence of a control group. Do not turn an association into a causal claim. If the source does not establish a result, say so."

The intended summary would report both the observed improvement and the lack of a comparison group. That is an expected behavior we can test. We have not established that this wording produces it reliably.

Examples help make the rule concrete. Show a study where the limitation must survive, an ordinary announcement where no study limitation applies, and a source that contains no supported finding. Those cases also expose a possible overcorrection: a model that now adds a warning to everything. The examples should teach the decision, rather than make the model memorize a phrase. Anthropic's context engineering guidance similarly recommends diverse examples and clear instructions instead of accumulating rules for every imagined edge case.

There is no universal number of examples that gives the best improvement. Add examples that explain a missing distinction, then measure whether they help. A longer prompt has to earn its reading time and token cost.

The prompt includes the tools

Suppose the summarizer can fetch the full article after receiving an excerpt. A tool description such as "gets content" leaves several questions unanswered. Does it retrieve the full text? Can it read any URL? What happens when the source is unavailable?

A useful description would explain that the tool retrieves the article at the supplied URL, returns its text when available, and reports a retrieval failure otherwise. The system instructions would say when the excerpt is insufficient and prohibit claiming to have read content that the tool could not fetch. The tool's argument model defines what the application accepts; the description explains when and why the model should use it.

There is a documented example of this kind of correction. Anthropic reported that Claude was unnecessarily appending 2025 to web-search queries, biasing the results. Improving the tool description corrected that behavior. The report does not establish a general improvement percentage for prompt revisions, but it shows how a small instruction can change an observable tool-use mistake. See Writing effective tools for agents.

Structured output belongs in the same design. A schema can require separate fields for a finding and its limitations. JSON Schema, Pydantic models, or a grammar can describe the expected shape, while a supported constrained-generation mechanism can restrict what the model emits. Merely writing "return JSON" does not enforce that structure. Even when the shape is enforced, a neatly filled field can contain an unsupported claim. Check its meaning against the source.

Test the correction and the damage it could do

The first regression test is the failure that motivated the edit. The next tests are cases the old prompt handled correctly. For our summarizer, I would include sources with important limitations, sources without them, incomplete excerpts, and text containing instructions that should be treated as quoted material.

Decide what counts as success before comparing outputs. Does the summary preserve the main finding? Does it retain the qualification that changes the reader's interpretation? Does it invent a limitation? Does it respect the length requirement? Keep those checks separate so an attractive average cannot hide a serious regression.

An LLM judge can help grade many examples, but its judgments also need checking against human-reviewed cases. Otherwise, two models may agree that a misleading summary is excellent. Retain the source, output, and grading rationale so a failure can be inspected.

For an agent that retrieves, summarizes, checks, and delivers an answer, test the complete workflow too. A toy calculation explains why: if four required steps each succeed with probability 0.95, and their successes are independent, the probability that all four succeed is 0.95^4 = 0.8145, about 81%. This is not a measured agent reliability rate. Real steps can share a cause of failure, and retries or recovery can change the outcome. Nor is 95% automatically acceptable for the original single step; that depends on the consequences of its errors.

Let a model propose the revision

You can give a model the current prompt and the cases it failed, then ask it to suggest a correction. That is a useful way to generate candidates when you can describe the failure more easily than the fix.

The following pseudocode captures the loop:

The useful part is the comparison at the end. The risky part is treating one score as sufficient permission to promote the candidate. In an actual implementation, separate the cases used to generate the revision from held-out evaluation cases, inspect serious regressions, and compare cost and latency. Otherwise, the revision can learn the examples you showed it while getting worse elsewhere.

The loop is a design pattern, not a benchmark result. Report any improvement with the cases tested, the definition of success, and both prompts' scores. A percentage without those details makes the revision difficult to judge, especially if it does not distinguish a relative gain from a gain in percentage points.

More reasoning is another change to evaluate

Some failures need a better instruction. Others may benefit from a different inference process. These techniques give you distinct options:

• Chain-of-thought prompting encourages intermediate reasoning. Whether an explicit request helps depends on the model and task; "think step by step" is not a universal upgrade or a universally obsolete phrase.
• Tree of Thoughts explores and evaluates multiple reasoning paths, with opportunities to look ahead or backtrack. Its published experiments support particular planning and search tasks, not an improvement on every prompt.
• Self-consistency samples different reasoning paths and selects an answer based on agreement. The original work reports gains on arithmetic and commonsense reasoning benchmarks. Repeated agreement can still preserve a shared mistake.
• Reflection asks the model to critique and revise its answer. Self-Refine reports improvements across several tasks, while research on intrinsic self-correction shows that reasoning can also deteriorate without external feedback. Give the reviewer something concrete to check.

For the summarizer, checking the draft against the source's qualification may be more useful than generating several fluent alternatives. Extra calls cost tokens and time. Keep them when the measured improvement justifies that cost.

A valid answer can still be an unsafe action

The article being summarized is data, even if it contains a sentence telling the model to ignore its instructions. An attacker can place such directions in a document or tool response as well as in a user message. Anthropic's prompt-injection research describes this as an ongoing problem with defenses that reduce, rather than eliminate, successful attacks.

Separate trusted instructions from untrusted content using the API's message roles and clear boundaries. Sanitize inputs where that has a defined purpose, but stripping a few suspicious strings will not catch every instruction written in ordinary language. A separate classifier can flag possible attacks; its false positives, missed attacks, latency, and cost need evaluation too.

Validate outputs before using them. Then check the proposed action against permissions and application rules outside the model. A JSON object containing a recipient and a document can be perfectly valid while asking to send private material to the wrong person. The schema checks the shape of the request. The application must decide whether that recipient is authorized to receive that document.

These controls address different failures. None makes the others unnecessary, and a prompt telling an agent to be careful cannot replace an enforced permission boundary.

When the revision should ship

Keep the prompt, examples, tool definitions, and evaluation cases under version control. A file in the repository is often enough; a database or prompt-management service can also work if changes remain reviewable and reproducible. What matters is knowing which instructions produced an answer and being able to restore a previous version.

Record the model version and relevant settings with the evaluation. Recheck behavior when the model, tools, or context changes. For a candidate that passes offline tests, a limited rollout can reveal differences in real use. Track completion quality, user outcomes, per-call cost, and median and P95 latency, then keep a route back to the previous version.

I would ship our summarizer revision when it preserves the important qualification across representative cases, maintains the behavior that already worked, and meets the application's cost and response-time requirements. I would reject it if it fixes the opening example by inventing caveats everywhere else. A cleaner-looking prompt is not evidence of a better system.

That gives prompt design a concrete responsibility. Specify the behavior the reader needs, expose failures with examples, and make each revision answer to those examples and to cases it has never seen. The reader of the summary should come away understanding what the study found and what it could not establish. That is the result worth engineering.