Single-Agent & Tool-Calling Loop

Start with a concrete problem

When two tools tie in capability score, the selector must not default to the first item

In the first step of the single-agent loop, the system must choose exactly one tool from the registered tool list based on the capability set required by the goal. The file forge/agent/actions.py does not exist yet, so test collection aborts immediately with ModuleNotFoundError: No module named ‘forge.agent’.

The core decision in this stage is not merely creating that file; it is correctly handling capability-score ties inside select_action. If two tools match the same number of required capabilities, taking the first item after sorting does not prove the choice is unambiguous, so the function must raise ValueError(“ambiguous tool choice requires clarification”) to request external clarification.

The fault lab supplies a faultySource that already implements the goal check, capability ranking, and no-match check, but it deliberately removes the tie-checking branch. This causes test_ambiguous_tools_require_clarification to fail because the selector silently returns the first sorted tool instead of raising an exception.

Make a prediction

After select_action sorts the tool list by capability match count in descending order and finds ranked[0][0] == ranked[1][0] (the top two tools have the same score), what is the correct action?

• Return the tool at ranked[0] because sorting already determined priority
• Raise a ValueError containing “ambiguous” to signal that the choice is unclear and needs clarification
• Return a special Action object marked as pending
• Re-sort and pick the first tool alphabetically by name

Reasoning guide: Sorting only produces a sequence; when the highest score is shared by multiple tools, sorting alone cannot distinguish which tool is the correct choice. The correct behavior is to check for a tie before returning an Action and raise a ValueError so the outer loop knows clarification is required.

What it is

Action selection combines capability matching with uniqueness verification

The select_action function takes three inputs: a goal string, a set of required_capabilities, and a list of ToolSpec objects. It ranks each tool by computing the size of the intersection between the tool’s capabilities and required_capabilities, then selects the tool with the highest match count.

Uniqueness verification is a mandatory step that cannot be skipped: even when sorting produces a first-place tool, if a tie exists the selector cannot assert that the choice is deterministic. The tie check ranked[0][0] == ranked[1][0] must execute before returning an Action, otherwise the selector mistakes sort order for selection evidence.

When no tool matches any required capability, ranked[0][0] equals 0 and the function must raise a ValueError containing “no tool”, indicating that the current tool list cannot satisfy the goal.

What it is not

Action selection does not execute tools or decide stopping

Action selection only chooses one tool from the candidates and returns an Action object; it does not call the tool function, manage agent state, or decide whether the loop terminates. Those responsibilities belong to the harness and later stages of the agent loop.

Action selection is also not a simple sort-and-take-first operation. Sorting is an intermediate step; the sorted result must pass both the no-match check and the tie check before it can be converted into an Action return value.

Action selection does not handle tool permissions, side effects, or isolation. The harness handles those concerns when it actually executes the Action; select_action only makes a decision based on capability matching.

How it relates to neighboring concepts

Where action selection sits in the agent loop

In the core agent loop (goal/state, choose action, execute tool, observe result, verify, stop or replan), select_action corresponds to the choose-action step. Its output, Action(tool_name, reason), is passed to the execution layer to invoke the corresponding tool.

ToolSpec defines the static capability description of a tool (its name and capability set) and serves as input to select_action. Action is the data carrier for the selection result, containing tool_name and reason, and serves as input to the execution layer. Both are frozen dataclasses to ensure immutability.

When select_action raises a ValueError, the agent loop should catch it and decide whether to request user clarification (ambiguous case) or report a capability gap (no-tool case), rather than continuing execution.

Boundary decision for this stage: Action selection only ranks tools by capability-intersection size and verifies uniqueness, raising ValueError on ties instead of guessing; it does not execute tools, manage state, or decide stop conditions, which are responsibilities of the external harness and agent loop.

Start with a concrete problem

Tool Result Recording and the Conflation of Recoverable Errors

In a single-agent loop, tool execution results must be recorded as ordered observations so that subsequent action selection and verification can trace the full run history. Currently the file forge/agent/observations.py does not exist, so test collection fails with ModuleNotFoundError: No module named 'forge.agent.observations', preventing the entire test suite from running.

More critically, tool call result statuses are divided into ok, retryable_error, and fatal_error, where a recoverable transient error such as an HTTP 503 must not terminate the loop, while a fatal error such as a permission denial must set the terminal flag. If transient transport errors are conflated with the loop stop state, the agent will permanently halt on any temporary failure, unable to retry or replan.

Make a prediction

When LoopState.record receives status="retryable_error", what value should state.terminal be set to?

• True, because any error should stop the loop
• False, because a transient error is retryable and must not terminate the loop
• None, because the state is undefined
• An exception should be raised, because the error cannot be handled

Reasoning guide: The correct answer is False. retryable_error represents a recoverable transient failure such as a network timeout or 503, and the agent loop should have the opportunity to retry or choose an alternative action. Only fatal_error represents an unrecoverable terminal condition and should set terminal to True.

What it is

Responsibility Boundary of LoopState

LoopState is a dataclass responsible for recording each Observation produced by a tool execution in call order, and for managing whether the loop can continue accepting new observations through its terminal boolean flag. Observation is an immutable frozen dataclass containing sequence (an incrementing number starting at 1), tool_name, status, and payload, ensuring that once an observation is created it cannot be tampered with.

The record method is the core entry point of LoopState: it first checks the terminal flag and raises ValueError if already terminated; then validates that status is in the legal set; next computes the new sequence using len(self.observations) + 1 and creates an Observation; finally decides whether to set terminal to True based on status. This flow decouples state management from action selection — LoopState does not decide what to do next, it only records what happened and whether the loop can continue.

What it is not

What LoopState Does Not Do

LoopState does not execute tool calls, select the next action, perform retry logic, or interpret the semantic content of payload. It is a passive state container and event sequence recorder driven by the outer agent loop through its record method.

LoopState is also not a logging system or observability platform — it does not record timestamps, correlation IDs, or metrics, and only maintains an ordered observation list and a terminal flag. If tracing or metric aggregation is needed, that is the responsibility of the Observability module, not LoopState.

How it relates to neighboring concepts

Relationship to Other Agent Loop Components

LoopState forms an upstream-downstream relationship with select_action and ToolSpec from Stage 01: select_action decides the next action based on the observation history in LoopState, and after tool execution the result is written back to LoopState via record, closing the loop. The terminal flag directly affects the loop exit condition — when it is True, the outer loop should stop calling record and end the run.

The sequence field of Observation provides deterministic ordering for outer verification and evaluation, allowing tests to assert the result of the Nth tool call precisely and enabling trace replay. The three-valued status classification (ok/retryable_error/fatal_error) makes error severity explicit, so loop control logic can distinguish retry from termination.

Boundary decision for this stage: LoopState only records observations and manages the terminal flag; it does not select actions, execute tools, or perform retries. retryable_error keeps terminal=False, fatal_error sets terminal=True, and after terminal=True the record method raises ValueError.

Start with a concrete problem

Undeclared arguments must be rejected before calling the handler

In the single-agent loop, execute_tool is responsible for validating the argument contract before calling the concrete handler. Currently forge/agent/execution.py does not exist, so the test file tests/test_stage.py raises ModuleNotFoundError at import time and all three tests fail during collection.

The driving question is: why must undeclared arguments be rejected before calling the handler? If extra arguments are passed straight through to the handler, the handler may crash due to an unexpected keyword argument, or in a more dangerous case it may execute an unauthorized side effect such as receiving sudo=True.

The task for this stage is to create forge/agent/execution.py from scratch, define the ToolContract and ExecutionResult dataclasses, and implement the execute_tool function so that it raises ValueError when arguments are missing or contain undeclared fields, and returns a wrapped ExecutionResult when arguments are fully valid.

Make a prediction

When execute_tool receives arguments {"path": "a.txt", "sudo": True} but ToolContract only declares required=frozenset({"path"}) with no optional set, what happens if the code only checks for missing arguments and then calls handler(**arguments) directly?

• The handler executes normally and ignores the extra argument
• The handler raises TypeError because it receives an unexpected keyword argument sudo
• execute_tool raises ValueError mentioning unexpected
• The system automatically strips the sudo argument and continues

Reasoning guide: The correct answer is that the handler raises TypeError. Python’s **arguments unpacking passes every key as a keyword argument, so if the handler signature does not accept sudo it will crash. This proves that unexpected arguments must be checked actively before calling the handler rather than relying on the handler to defend itself.

What it is

Argument contract validation boundary

execute_tool is the isolation layer between the Agent Loop and the concrete tool implementation. Its sole responsibility is to validate the incoming arguments dictionary against the required and optional parameter sets declared in ToolContract, call the handler and wrap the result if they match, or raise ValueError if they do not.

Set operations are the core of validation: missing = set(contract.required) - set(arguments) finds required parameters that are absent, and unexpected = set(arguments) - set(contract.required) - set(contract.optional) finds extra parameters that are neither required nor optional. Both checks must complete before the handler is called.

What it is not

Responsibility boundary of execute_tool

execute_tool does not implement retry logic, does not manage LoopState, does not perform idempotency checks, and does not care about the business logic inside the handler. It is only a strict structural gateway.

It is also not a generic argument filter that silently strips unexpected parameters and then calls the handler, because silently dropping arguments would conceal the caller’s erroneous intent and could allow an unauthorized capability to execute.

How it relates to neighboring concepts

Relationship to the Agent Loop and Verification

After the Agent Loop uses select_action to decide which tool to call, it hands control to execute_tool. execute_tool guarantees that only arguments matching the contract reach the handler, which provides a deterministic foundation for subsequent Verification.

ExecutionResult wraps the handler’s return value into a structured result carrying tool_name, so the Observation layer can uniformly record and trace tool call outputs without needing to know the concrete type of the raw return value.

Boundary decision for this stage: execute_tool calls the handler and returns ExecutionResult only after the argument contract passes validation; any missing or unexpected argument must terminate with ValueError before the handler is called, never silently filtered or passed through.

Start with a concrete problem

Why you must not retry all exceptions indiscriminately

In a single-agent loop, tool execution can encounter two fundamentally different kinds of failures: transient failures such as a temporarily busy service, and permanent failures such as a permission denial. If the retry policy does not distinguish between these two classes and treats all exceptions as retryable, a forbidden operation will be attempted three times, wasting the retry budget and masking the real error.

This stage requires creating the forge/agent/retry.py file to implement the run_with_retry function. This function must only catch TransientToolError for bounded retries and must immediately re-raise when it encounters a PermanentToolError. The current test suite in tests/test_stage.py defines three tests, but because the forge.agent.retry module does not yet exist, the tests fail during collection with a ModuleNotFoundError.

Make a prediction

When run_with_retry receives an operation that always raises PermanentToolError, how many times should a correct implementation call that operation?

• 1 time, then immediately re-raise the exception
• 3 times, until the max_attempts budget is exhausted
• 0 times, directly returning a failed result
• 2 times, saving one budget for subsequent recovery

Reasoning guide: The correct answer is 1 time. PermanentToolError represents a permanent failure where retrying will not change the outcome, so it must be re-raised immediately upon the first catch without consuming the remaining retry budget.

What it is

The precise semantics of bounded retry

run_with_retry is a standalone function responsible only for retry policy. It accepts a zero-argument operation callable and a max_attempts bound, returning a RetryResult containing the final value and the actual number of attempts. Its core logic calls operation inside a for loop, continuing only when it catches a TransientToolError and the current attempt count has not reached the limit.

RetryResult uses a frozen=True dataclass to ensure the return value is immutable, allowing callers to safely inspect result.value and result.attempts to verify the execution trace. When max_attempts is exhausted and the last attempt still raises TransientToolError, that exception is re-raised to the caller for handling.

What it is not

Boundaries and responsibility limits of the retry policy

run_with_retry is not responsible for selecting tools, managing agent loop state, or checking operation idempotency; it is purely an execution wrapper. It does not log, maintain traces, or interact with external observability systems, as those responsibilities belong to the Harness and Observability modules.

The retry policy is not an unbounded model loop that continues until success; max_attempts provides a hard upper bound. When max_attempts < 1, the function directly raises a ValueError, refusing to execute any meaningless attempts.

How it relates to neighboring concepts

The causal relationship between exception classification and retry decisions

TransientToolError and PermanentToolError both inherit from RuntimeError, but their handling paths within the retry loop are completely different. The except clause catches only TransientToolError, which allows PermanentToolError and other unlisted exceptions to naturally propagate through the loop and immediately terminate retries.

This classification strategy based on exception type directly binds failure nature to retry decisions: a transient failure means a retry might succeed, while a permanent failure means a retry will certainly fail. The Agent Loop relies on this precise classification to decide whether to continue retrying or report an unrecoverable error to the upper layer.

Boundary decision for this stage: run_with_retry only retries TransientToolError with a bound; it immediately re-raises upon encountering PermanentToolError or any other exception, without selecting tools, managing loop state, or checking idempotency.

Start with a concrete problem

Registry behavior when the same idempotency key arrives with a different request fingerprint

In a single-agent loop, tool calls may be triggered multiple times due to network retries or duplicate user submissions. If every call executes a write operation, duplicate side effects occur, such as writing the same file twice or sending the same email twice. An idempotency registry prevents this duplicate execution by recording the idempotency key and the request fingerprint. When the same idempotency key arrives again, the registry must check whether the request fingerprint exactly matches the stored fingerprint. If the fingerprint differs, a different request has reused the same key, and the registry must reject execution and raise an exception.

The file forge/agent/idempotency.py does not exist yet, so the tests fail at import time with ModuleNotFoundError. You need to create this file and implement the IdempotencyRegistry class so that all three tests pass.

Make a prediction

When execute_once(“op-1”, “hash-b”, operation) is called and the registry already stores fingerprint “hash-a” for key “op-1”, what should the registry do?

• Execute operation and overwrite the old result with the new one
• Return the cached first result and mark it as a duplicate
• Raise ValueError indicating the request fingerprint does not match
• Ignore the new request and return nothing

Reasoning guide: The correct answer is to raise ValueError. The idempotency key identifies an operation slot, while the request fingerprint identifies the specific request content. If the key is the same but the fingerprint differs, two different requests have accidentally reused the same key, and the registry must reject to prevent semantic errors.

What it is

What the idempotency registry is

IdempotencyRegistry is an in-memory registry implemented as a dataclass, storing a mapping from idempotency keys to (fingerprint, value) tuples in its _entries dictionary. Its execute_once method takes an idempotency key, a request fingerprint, and a callable operation, returning a (value, bool) tuple. The boolean False means this is the first execution and the operation was actually called, while True means a cache hit occurred and the operation was not called. This mechanism lets the agent loop safely avoid duplicate side effects during retries or repeated calls.

What it is not

What the idempotency registry is not

IdempotencyRegistry does not select tools, does not implement retry logic, and does not manage agent loop state transitions. It does not validate whether the operation return type conforms to a tool contract, and it does not persist to disk or a database. It is a local, in-process deduplication guard whose lifetime is bound to a single registry instance.

How it relates to neighboring concepts

Relationship to other agent loop components

In the agent loop, IdempotencyRegistry sits on the tool execution path, typically intercepting before execute_tool calls the actual side effect. It complements the retry mechanism in retry.py: retry handles transient failures, while the idempotency registry handles duplicate requests. ToolContract defines the parameter and result contract for a tool, while the idempotency registry ensures that repeated calls under the same contract do not produce multiple side effects. Observability components can log each execute_once hit or rejection event to trace the agent execution path.

Boundary decision for this stage: The boundary of the idempotency registry is: it only prevents duplicate side effects using keys and fingerprints, does not select tools, does not retry, and does not manage loop state; empty keys or empty fingerprints must be rejected because they cannot uniquely identify an operation slot.

Start with a concrete problem

Producing an answer alone is not verified success

In a single-agent loop, the model may have already generated an “answer” action, but this does not mean the task goal has been externally verified. You must now create the forge/agent/stopping.py file and implement the StopController class so the loop makes explicit stop decisions based on the externally supplied verified flag, fatal_error, the step_count budget, and repeated action patterns in action_history.

Before this file is implemented, the test module fails with a ModuleNotFoundError because it cannot find forge.agent.stopping. In the faulty version, the code incorrectly treats the last action in action_history being “answer” as verified_success, causing the loop to stop before acceptance checks pass.

The driving question for this stage is: why is producing an answer alone insufficient to trigger a verified_success stop? The core reason is that output production is a model behavior, while verification is externally inspectable evidence, and the two must not be conflated.

Make a prediction

When decide(step_count=1, verified=False, fatal_error=False, action_history=["answer"]) is called, what should StopController return?

• Return stop=True, reason=“verified_success” because the model already produced an answer
• Return stop=False, reason=“continue” because external verification has not passed
• Return stop=True, reason=“step_budget_exhausted” because the step budget is used up
• Return stop=True, reason=“fatal_error” because unverified means failure

Reasoning guide: The correct answer is to return stop=False. The verified parameter being False means the external acceptance check has not passed yet, so even if “answer” appears in action_history, the loop must not stop. Producing an answer and verifying success are two independent stages.

What it is

Responsibility boundary of StopController

StopController is a component solely responsible for stop decisions. It accepts four external inputs: step_count, verified, fatal_error, and action_history, and returns a StopDecision dataclass containing a stop boolean and a reason string.

The priority order of stop conditions is: first check whether verified is True, then check fatal_error, then check whether step_count has reached the max_steps budget limit, and finally check whether the repetition count of the last action in action_history has reached repeat_limit. If none are met, it returns stop=False and reason="continue".

What it is not

What StopController does not do

StopController does not execute any tool calls, manage observation results, or handle retry logic. It does not read model outputs directly or judge the quality of answer content; it relies entirely on boolean flags passed in by the external harness.

It is not a mechanism for letting the model loop indefinitely, nor is it an independent persona. The authority to stop belongs to the external runtime environment, and StopController merely aggregates these constraints into a single explicit decision object.

How it relates to neighboring concepts

Relationship to other Agent Loop components

StopController sits in the Agent Loop after verification and before action selection. After the harness executes a tool and observes the result, the Verification component checks whether the result advanced the goal, then passes the verified boolean to StopController.

action_history is maintained by LoopState and records the full sequence of actions taken so far. step_count is also part of LoopState. StopController detects whether the loop is stuck in repetition by checking the occurrence count of the last action in action_history.

Boundary decision for this stage: StopController only makes stop decisions based on externally supplied verified, fatal_error, step_count, and action_history; it does not execute tools, manage observations, retry, or infer answer correctness on its own.

Start with a concrete problem

Loop fails to stop after verification passes, causing extra tool calls

At the start of this stage, the file forge/agent/runner.py does not exist, so the test file tests/test_stage.py raises ModuleNotFoundError while importing from forge.agent.runner import run_agent, causing collection to abort with exit code 2. You must create this file and implement the run_agent function, composing the LoopState and StopController modules from previous stages into a bounded loop.

In the fault lab, an implementation of run_agent already exists but contains a precise defect: when StopController.decide returns decision.stop as True with reason set to verified_success, the code does not return and instead continues the loop. This causes test_runner_stops_immediately_after_verified_output to fail because the step callback is invoked 3 times instead of the expected 2, and result.reason becomes repeated_action instead of verified_success.

The core engineering decision is that run_agent must return immediately for any decision where decision.stop is True, regardless of whether the reason is verified_success, repeated_action, or step_budget_exhausted. Ignoring any stop condition breaks the bounded-loop guarantee and may trigger unnecessary side effects after verification has already passed.

Make a prediction

When verify(output) returns True and StopController.decide returns decision.stop == True, what should run_agent do immediately?

• Return a LoopResult immediately without calling step again
• Continue the loop until max_steps is exhausted
• Raise an exception to abort the loop
• Call step again to confirm the result

Reasoning guide: The correct choice is to return immediately. Verification passing means the goal is achieved, and continuing the loop wastes step budget and may trigger an incorrect repeated_action stop reason.

What it is

Composition of a bounded agent loop

run_agent is a combinator that accepts a step callback and a verify function, executing within a max_steps limit: call step to get an action and output, record it in LoopState, and use StopController to decide whether to stop. run_agent itself does not select tools, retry, or check idempotency; those responsibilities belong to the step callback and previously implemented modules.

Each step of the loop produces a StopDecision where the stop boolean is the sole termination signal. The core responsibility of run_agent is to faithfully execute this signal: once stop is True, regardless of the reason, it must immediately exit the loop and return the current state.

What it is not

What run_agent does not do

run_agent is not an autonomous decision-maker; it does not decide which tool to call—that is determined by the external step callback based on the step number. run_agent also does not perform error retries or idempotency checks, as those belong to run_with_retry and IdempotencyRegistry.

run_agent is not an unbounded loop or an independent persona. It does not continue running because a model feels more information is needed; stop conditions are entirely constrained by StopController and the max_steps parameter, and any stop decision must be executed immediately.

How it relates to neighboring concepts

Responsibility boundaries between modules

LoopState is responsible for recording action history and observations, while StopController decides whether to stop based on step count, verification result, and action history. run_agent chains the two together: record first, then decide, and let the decision drive the control flow.

The step callback is the action source, verify is the acceptance source, StopController is the stop arbiter, and run_agent is the executor. This separation keeps run_agent logic simple: execute, record, decide, then return or continue.

Boundary decision for this stage: The boundary of run_agent is: return unconditionally for any decision where decision.stop == True without distinguishing reason; loop steps are strictly limited by max_steps; it does not handle tool selection, retries, or idempotency checks.