Ask a movie-clip agent to find a scene, and a useful result is a link that opens at the right moment. Ask the same agent for a song, and its best response may be to stop before searching. Both requests sound easy. Only one belongs to the job it was given.

That choice makes autonomy interesting. Once software can choose its next action, we need to define what it may attempt, what evidence it needs, and when the task is finished. A model's willingness to keep working doesn't answer any of those questions.

The movie-clip agent below gives us a small, concrete way to examine them. Its recorded runs pair successful searches with an out-of-scope refusal. Follow the tool results as well as the final answer: they explain how a request becomes a sequence of actions.

Captured runs of the movie-clip agent, with model output, tool calls, and results. The player adjusts pauses and text pacing; event timestamps preserve the recorded timing. Movie requests can proceed through web search, YouTube search, timestamp lookup, and URL construction. The out-of-scope song request ends with a refusal. These examples show particular executions, not a measured success rate.

A goal needs a boundary

This agent handles films and television. For an unfamiliar quote, it can search the web to identify the film, search YouTube for a candidate video, ask a video-understanding tool where the moment appears, and construct a link to that timestamp. If the film is already known, its instructions allow it to begin with YouTube search. The next action depends on what remains unknown.

The tool called process_video deserves a close look. In this implementation it constructs a YouTube watch URL with a starting timestamp. It doesn't download, cut, or produce a new video file. That difference defines what completion means: the agent can deliver a link to a moment in an existing video, not promise an extracted MP4.

The song refusal follows the same idea. The agent has been assigned a film-and-television task, so a music request should end without a search. This is a useful observed behavior. It is also a prompt instruction, not an enforced restriction on what the search tools can search. A later run could disobey it. If that boundary must hold for every request, the application needs to enforce it outside the model.

I find it useful to separate a job description from access permissions. Telling someone to prepare a report explains their assignment. Giving them a database account determines what they can actually do. Agents need both, and the second should still hold when the first is misunderstood.

How the loop uses feedback

An LLM agent combines a model with an application that can execute actions and return their results. The model helps decide the next step; the application provides the tools, retains task state, and controls execution. It can continue without asking the user to specify every intermediate step. Long-term memory is optional: a short task can finish using only its current conversation and tool results.

A useful architectural distinction is whether code fixes the sequence in advance or the model chooses the next step as results arrive. Anthropic uses this distinction between workflows and agents in its guide to building effective agents. Neither design removes the need for ordinary software controls.

ReAct, introduced by Yao and colleagues in 2022, describes interleaving reasoning with actions and observations. Here is an illustrative sales-report example. The file names and revenue figures are invented to explain the sequence; this is not a recorded run or measured financial result.

Thought: The user wants me to find the latest sales report. I should search the file system.
Action: search_files(query="sales report Q3 2025")
Observation: Found 2 results: /reports/q3-sales-final.pdf, /drafts/q3-sales-v2.docx
Thought: The final version is likely the correct one. Let me open it.
Action: read_file(path="/reports/q3-sales-final.pdf")
Observation: [PDF content with revenue figures...]
Thought: I have the data. I can now summarize it for the user.
Action: respond("Q3 revenue was $4.2M, up 12% from Q2...")

The useful part is the dependence between steps. The file search supplies a path; reading that file supplies the figures. A filename containing "final" is a reason to inspect the document, not proof that it is the latest approved report. In a real reporting task, the agent would need to check the period and approval information before treating those figures as authoritative.

The movie example has the same dependency. A search result supplies a video ID, and timestamp lookup supplies a proposed location. The timestamp tool uses a model to examine the video, so its result is evidence to assess, not an infallible measurement. A plausible link alone cannot establish that the requested scene is there.

Text labeled "Thought" or "Observation" is also insufficient proof of execution. The application must actually run the requested function and return its result. The playback system records tool calls and returned results separately, which lets us inspect that distinction. Reading the model's explanation can help diagnose a run; it cannot guarantee that its next action is safe.

Tools determine what can happen

The model receives descriptions of the tools available to it. A simplified, provider-neutral declaration for a database tool might look like this:

{ "name": "execute_sql", "description": "Run a SQL query against the analytics database", "parameters": { "type": "object", "properties": { "query": { "type": "string", "description": "The SQL query to execute" } }, "required": ["query"] } }

The model proposes a structured call. The application validates and executes it, then returns the observation. The declaration tells the model how to ask; the database account and execution code determine what is permitted. Calling a tool "analytics" doesn't make its SQL read-only.

For an analysis agent, I would start with read-only credentials and restricted access to the data it needs. A read replica can separate analytical work from the primary database, but it doesn't by itself decide which tables the agent may read or keep expensive queries under control. Those are separate permissions and resource limits.

The same principle extends to other tasks. A coding agent needs access to files and a test runner to attempt a repair. A research agent needs retrieval tools and source documents to assemble evidence. Browser or desktop control lets an agent interact through clicks and forms where an API is unavailable. Sending email or deploying code adds consequences that need their own authorization. A persuasive final message is no substitute for checking what changed.

Memory should preserve what matters

For the movie task, useful state is concrete: which film has been identified, which videos were tried, what timestamp came back, and whether a tool failed. Keeping that information in the current context lets the model choose a different candidate instead of repeating the same unsuccessful search.

A longer task may need external storage. There are several ways to preserve it, each with a different purpose:

  • Vector search can retrieve related past interactions by meaning. It helps find a relevant memory, but similarity doesn't establish that the memory is correct or current.
  • Structured logs can store tool calls, arguments, results, and errors. They make it possible to inspect what happened and query the history precisely.
  • Stored summaries can restore the main facts of a previous session without loading every message. Compression can omit details, so consequential claims should remain traceable to their source.

For an agent that acts on data, I'd keep structured execution records even if summaries make the conversation easier to resume. "The report was updated" is less useful for verification than knowing which report changed and whether the write succeeded. Saving history doesn't automatically teach the agent anything; the application must retrieve the relevant information and make it available for the next decision.

These approaches can be combined, but a film lookup doesn't need an elaborate memory system to qualify as an agent. Add persistence when the task requires continuity. Loading an ever-growing history consumes context and can add latency and cost without helping the next choice.

Stopping is part of completing the task

The movie agent has several legitimate endings. It can return a supported answer and timestamped link, explain that it couldn't retrieve the clip, or decline a request outside its assigned subject. Repeating searches indefinitely is not a fourth kind of success.

Its scene configuration sets a maximum of eight tool-calling rounds. The capture code uses that value to bound the loop and recognizes finish as a request to end it. A round can contain multiple calls, so eight rounds is not an eight-call spending limit. A real budget also has to account for the work each tool performs.

The prompt tells the agent to confirm a timestamp before constructing a link. The URL-building function itself does not verify that an earlier timestamp lookup occurred. This is exactly where an instruction and an execution rule diverge. If that sequence is required, code should check the prerequisite before allowing the call.

I would evaluate this system against its possible endings: a known film, an ambiguous quote, an unavailable video, a failed tool, and a song request. Check whether the answer is supported, whether the actions stayed within scope, and whether the agent stopped for a valid reason. Tool unit tests can establish that a URL builder handles inputs correctly. They cannot establish that the model chose the right video.

Broader evaluations such as SWE-bench for software issues and WebArena for web tasks give researchers repeatable environments. A deployment still needs cases that exercise its own permissions, failure paths, and completion criteria.

More planning, or more agents?

Some tasks benefit from planning a sequence up front and revising it after a failure. Others require comparing alternatives. Tree of Thoughts explores multiple reasoning paths; it is a reasoning-search method, not a guarantee of safe tool execution. Reflexion stores textual reflections on feedback for later attempts, without updating model weights. Both are useful ideas to examine when a simple loop struggles with a specific task.

Dividing work among agents is another option. Separate retrieval and review tasks can be useful when each produces an independently checkable result. They also introduce handoffs: who owns the answer, what evidence gets passed along, and who resolves disagreement? If every worker edits the same file or relies on the same mistaken assumption, adding workers can create more coordination work without improving the result.

For the movie lookup, the short sequence already gives us clear evidence and a clear stopping point. I would add another agent only after identifying a job it can do independently and a way to verify its contribution. A larger team needs a reason beyond being possible.

What a useful agent owes the user

The movie request is complete when the answer identifies the requested material and provides a supported way to reach it, or honestly explains why it couldn't. The song request can be complete with a refusal. In either case, the user should understand what happened without mistaking a generated explanation for verified execution.

That gives us a practical way to judge autonomy. Define the task and its permitted actions, retain the observations needed to evaluate the result, and enforce limits in the application. Then test successful work alongside failures and legitimate refusals. An agent earns wider responsibility when those checks support it. Its ability to keep taking steps, by itself, tells us very little about whether we should let it.