AI Tools, Models & Task Judgment

Start with a concrete problem

How to classify tasks using only three boolean features?

You are building a task router that must decide downstream handling based on objective task properties. The repository currently lacks forge/task_types.py, so tests cannot import classify_task.

The tests require classify_task to accept three boolean parameters: mutates_state (whether it changes state), needs_sources (whether it requires source evidence), and output_schema (whether it requires structured output), and return the correct TaskType enum member.

The key decision is priority when multiple features are true. For example, a task that both mutates state and needs sources should be classified as ACT, not RETRIEVE.

Make a prediction

If classify_task receives mutates_state=True, needs_sources=True, output_schema=True, which TaskType should it return?

• TaskType.ACT
• TaskType.RETRIEVE
• TaskType.EXTRACT
• TaskType.GENERATE

Reasoning guide: The correct answer is TaskType.ACT. Because mutates_state indicates the task will change external state, such tasks have side effects and must be prioritized as action tasks, even if they also need sources and structured output.

What it is

Task classification is a pure function decision tree

classify_task is a pure function: given three boolean inputs, it checks conditions in a fixed order and returns a TaskType enum value.

The decision order is: first check mutates_state, then needs_sources, then output_schema, and finally default to GENERATE.

This order reflects task risk: state-changing tasks are highest risk and must be handled first; tasks needing only sources are next; tasks requiring only structured output are next; remaining tasks are classified as generation.

What it is not

Task classification does not depend on model capabilities or user intent

Classification relies solely on three boolean inputs and does not involve model selection, cost, latency, or user preferences.

It does not judge whether a task is ‘difficult’ or ‘important’, nor does it consider the specific tools or environment for execution.

The classification result contains no confidence or probability; it is a deterministic mapping.

How it relates to neighboring concepts

Relationship between TaskType enum and classify_task

TaskType defines four possible categories: EXTRACT, GENERATE, RETRIEVE, ACT.

classify_task returns one of these enum members based on input features, and tests use the is operator to verify the same enum object is returned.

The enum inherits from str, so each member is both an enum and a string, facilitating serialization and comparison.

Boundary decision for this stage: When mutates_state is True, return TaskType.ACT regardless of other features; otherwise, if needs_sources is True, return TaskType.RETRIEVE; otherwise, if output_schema is True, return TaskType.EXTRACT; otherwise return TaskType.GENERATE.

Start with a concrete problem

Why would a model without tool-call support be incorrectly selected?

In Stage 01, we classified tasks into types, such as action tasks that require tool calling. Now we need to select a suitable model for each task, but candidate models have different capabilities.

Suppose we have a text model that only supports text modality and does not support tool calling. If we define task requirements as containing modality:text and tool_calls, the current code incorrectly considers this model as satisfying the requirements because it only checks modalities and ignores tool-call capability.

This error leads the system to select a model that cannot perform actions, causing task failure. We need to implement a capability matching function that ensures the model’s declared capabilities cover all task requirements.

Make a prediction

In the following code, the supports function currently only checks structured_output and modality requirements. If the requirement set contains tool_calls but the model has tool_calls=False, what will the function return?

• True, because the modality check passes
• False, because tool-call capability is missing
• Raises an exception, because the requirement is unrecognized
• None, because the function does not handle this case

Reasoning guide: The correct answer is ‘True, because the modality check passes’. The current code does not check the tool_calls requirement, so as long as modalities match, it returns True. This causes models without tool-call support to be incorrectly considered as satisfying the requirements.

What it is

Capability matching is a set coverage check

Capability matching checks whether the model’s declared capability set covers the task requirement set. Each requirement is a string, such as modality:text for text modality and tool_calls for tool-call capability.

Model capabilities are represented by the ModelCapability dataclass, which includes modalities (a set of modalities), structured_output (whether structured output is supported), and tool_calls (whether tool calling is supported).

The supports function iterates over the requirement set and checks each requirement against the model’s capabilities. If all requirements are satisfied, it returns True; otherwise, it returns False.

What it is not

Capability matching does not evaluate quality, cost, or latency

Capability matching only cares whether the model has the basic capabilities required for the task; it does not consider output quality, inference speed, cost, or other non-functional metrics.

For example, a model may support tool calling but have low accuracy in tool calls; this is outside the scope of capability matching. Capability matching answers ‘can it do it’, not ‘how well does it do it’.

Similarly, capability matching does not check whether the model is suitable for a specific domain, such as medicine or law, unless those domain requirements are explicitly represented as requirements.

How it relates to neighboring concepts

Relationship between capability matching, task classification, and model selection

Task classification (Stage 01) determines the task’s requirement set, e.g., action tasks require tool_calls. Capability matching uses this requirement set to filter candidate models.

Model selection further considers quality, cost, etc., among models that pass capability matching. Capability matching is a pre-filtering step for model selection.

The ModelCapability dataclass encapsulates the model’s static capability declarations, and the supports function is the concrete implementation of capability matching. Together they form the basis of the capability matrix.

Boundary decision for this stage: Capability matching only checks whether the model’s declared capabilities cover the requirement set; it does not involve quality, cost, or latency. If the requirement contains tool_calls, the model must explicitly declare tool_calls=True to pass matching.

Start with a concrete problem

Why is the cheapest model not always the best choice?

In task routing, if you select a model based solely on cost, you may pick a candidate that fails to meet the quality bar, causing the task to fail or produce unusable output.

This stage requires implementing a function that first filters out candidates whose quality score is below a minimum quality floor, then selects the model with the lowest estimated cost from the remaining candidates.

In the starting code, the file forge/economics.py does not exist, so the tests fail immediately when importing CandidateScore and choose_with_quality_floor.

You need to create this file and implement the data class and selection function so that two tests pass: one test verifies that a cheap candidate below the quality floor is rejected, and another test verifies that an explicit exception is raised when no candidate meets the quality floor.

Make a prediction

When implementing choose_with_quality_floor, which of the following orders is correct?

• First sort by cost, then check if quality meets the bar
• First filter out candidates below the quality floor, then select by cost
• Sort by a weighted combination of quality and cost simultaneously
• Pick a candidate at random

Reasoning guide: The correct order is to first filter out candidates below the quality floor, then select by cost. If you sort by cost first, you might pick a cheap model that fails the quality constraint.

What it is

What the quality-cost frontier is

The quality-cost frontier is a decision method: choose the candidate with the lowest estimated cost while satisfying a minimum quality requirement.

It consists of two steps: feasibility filtering (quality >= quality floor) and optimization (minimize estimated cost).

Feasibility filtering ensures that all candidates entering the cost comparison meet the quality constraint, and optimization finds the lowest-cost candidate among the feasible set.

What it is not

What the quality-cost frontier is not

It is not pure cost minimization, because the cheapest candidate may fail the quality bar.

It is not quality maximization, because the highest-quality candidate may be too expensive and exceed the budget.

It does not involve other constraints such as context windows or latency; this stage only considers quality scores and cost estimates.

How it relates to neighboring concepts

Relationships with other concepts

The CandidateScore data class encapsulates the model name, quality score, input cost, and output cost, and provides an estimated_cost method to compute the projected cost.

The choose_with_quality_floor function receives a list of candidates, a quality floor, input token count, and output token count, and returns the selected candidate.

Quality filtering must occur before cost calculation; otherwise, optimization may select an infeasible candidate.

Boundary decision for this stage: When no candidate meets the quality floor, the function must raise a ValueError rather than returning None or an arbitrary candidate, because the caller needs to know explicitly that no feasible option exists.

Start with a concrete problem

Why would a seemingly feasible model be rejected by the context window?

In Stage 03, you filtered candidate models by quality and cost, but you have not yet checked whether they can actually handle a given request.

Even a model with high quality and low cost cannot be used for a task if its context window cannot accommodate both input and output, or if its latency exceeds the budget.

In this stage, you will implement the is_feasible function, which determines whether a model is feasible based on context capacity and latency budget.

The file forge/constraints.py does not exist yet, so the tests fail at import time; you need to implement the module from scratch.

Make a prediction

Before implementing is_feasible, predict: if input_tokens=3800, output_reserve=500, and the model’s context_tokens=4096, is the model feasible?

• Feasible, because the input token count 3800 is less than the context window 4096
• Infeasible, because the total tokens of input plus output reserve 4300 exceeds the context window 4096
• Feasible, because output reserve does not consume the context window
• Infeasible, because output reserve must be calculated separately and cannot be added to input

Reasoning guide: The correct answer is “Infeasible, because the total tokens of input plus output reserve 4300 exceeds the context window 4096”. The context window must accommodate both input and output; output reserve is the space needed for the model to generate a response and must be included in the total token count.

What it is

What feasibility checking is

Feasibility checking is a binary decision: given a model profile and request parameters, can the model complete inference within constraints?

It only considers hard constraints: context window capacity and latency budget, not quality or cost.

The is_feasible function takes a RuntimeProfile (containing context_tokens and p95_latency_ms) and request parameters input_tokens, output_reserve, and latency_budget_ms, and returns a boolean.

What it is not

What feasibility checking is not

It is not quality assessment: even if a model is feasible, it may produce poor output, but that is handled by other stages.

It is not cost calculation: feasibility checking does not care about price, only whether the model can complete within constraints.

It is not a probabilistic judgment: feasibility is deterministic; if any constraint is violated, it returns False.

How it relates to neighboring concepts

Relationship to other concepts

Feasibility checking is a filter in the model selection process, applied after the quality-cost frontier.

It uses the RuntimeProfile dataclass, which encapsulates the model’s runtime attributes.

The result of is_feasible influences subsequent routing decisions; only feasible models can enter the final selection.

Boundary decision for this stage: Feasibility checking only answers “can it complete within constraints”, not “is it worth selecting”. If the context window or latency budget is not satisfied, the model must be eliminated, even if it is excellent in other aspects.

Start with a concrete problem

Who should execute a task that needs fresh data and changes external state?

In the previous stage, you filtered models by latency and context constraints; now you must decide whether a task should be handled by the model directly, a read-only tool, or a write tool.

The codebase currently lacks forge/boundaries.py, so the test file tests/test_stage.py fails at import with ModuleNotFoundError: No module named 'forge.boundaries' when it tries to import Executor and choose_executor.

You need to create this module and implement the choose_executor function: it takes two boolean parameters needs_fresh_data and changes_external_state, and returns an Executor enum member.

The key decision is: when a task both needs fresh data and changes external state, side effects must take priority, returning Executor.WRITE_TOOL, not READ_TOOL.

Make a prediction

If needs_fresh_data=True and changes_external_state=True, which executor should choose_executor return?

• Executor.READ_TOOL, because it needs fresh data
• Executor.WRITE_TOOL, because changing external state must be done by a write tool
• Executor.MODEL, because the model can handle both
• Raise an exception, because the two conditions conflict

Reasoning guide: The correct answer is Executor.WRITE_TOOL. Changing external state means the task has side effects and must be executed by a tool with write permissions; a read-only tool cannot safely complete such a task, even if it also needs fresh data.

What it is

Executor selection is a priority decision

choose_executor is a pure function that decides the executor type based on two boolean task attributes: whether it changes external state and whether it needs fresh data.

The decision order is: check side effects first, then data freshness, and finally default to the model. This ensures any side-effecting operation is never incorrectly assigned to a read-only tool or the model.

What it is not

It is not a concrete tool implementation or permission system

This stage only returns an enum value indicating which type of executor should be used; it does not actually invoke tools, check permissions, or handle errors.

The Executor enum is just a label, not a specific function or API; actual tool implementations and permission controls are handled by later stages or an external harness.

How it relates to neighboring concepts

Relationship to existing modules

forge/boundaries.py depends on concepts from previously defined task_types.py, capabilities.py, economics.py, and constraints.py, but this stage only uses two boolean inputs and does not directly import those modules.

The test file tests/test_stage.py imports Executor and choose_executor from forge.boundaries, so the module must exist and the function signature must be correct.

Boundary decision for this stage: This stage only decides the executor type (model, read tool, write tool); it does not involve specific tool implementations or permissions. If a task changes external state, it must return Executor.WRITE_TOOL regardless of whether it needs fresh data.

Start with a concrete problem

How do we fairly compare candidate models and reject those missing critical results?

You have already built task classification, capability matrix, quality-cost frontier, latency constraints, and model-tool boundary in previous stages. Now you need to implement a benchmark suite that compares different candidate models using the same set of weighted test cases, and ensures that a candidate is rejected when critical task results are missing.

The file forge/benchmark.py does not exist yet, so the tests test_missing_critical_case_rejects_candidate and test_weighted_score_uses_measured_cases fail with an import error. You need to create this file, define the BenchmarkCase dataclass and the score_candidate function, and make both tests pass.

Specifically, score_candidate receives a list of cases and a results dictionary. It must check that all cases marked critical=True have results; if any critical case is missing, it must raise a ValueError containing ‘critical’. For cases that have results, it computes a weighted average using the case weights.

Make a prediction

When implementing score_candidate, what should the function do if a critical case’s result is missing?

• Ignore the missing critical case and compute weighted average only on available results
• Raise ValueError indicating missing critical benchmark case
• Treat the missing critical case score as 0 and continue
• Return None to indicate inability to score

Reasoning guide: The correct action is to raise ValueError. Critical cases are mandatory tasks; missing results mean the candidate model has not completed necessary evaluation, so a valid score cannot be given. Ignoring or treating as 0 would incorrectly let the candidate pass.

What it is

Benchmark scoring is a weighted average calculation with a completeness check

The benchmark scoring function receives a set of case definitions (each with id, weight, and critical flag) and the candidate model’s scores on those cases. It first verifies that all critical cases have results, then computes a weighted average over the cases that have results.

The core of this model is that completeness of critical cases is a prerequisite for scoring, and weights reflect the relative importance of different cases. Missing critical cases mean the candidate has not completed necessary evaluation, so it must be rejected.

What it is not

It is not a simple average, nor a fault-tolerant calculation that ignores missing data

Benchmark scoring is not a simple average over all cases because different cases have different importance and must be weighted. It is also not fault-tolerant; it cannot silently skip missing cases, especially critical ones.

It does not involve evaluation of model capabilities or costs; it only performs mathematical calculation based on provided cases and results. Any missing critical case must be explicitly reported as an error, not given a seemingly plausible score.

How it relates to neighboring concepts

Relationship with task classification, capability matrix, and constraints

Benchmark case ids typically correspond to task types (e.g., ‘schema’, ‘summary’) from Stage 01 task classification. Case weights may reflect task importance or frequency, related to Stage 02 capability matrix and Stage 03 quality-cost frontier.

The critical flag may come from Stage 04 constraints, such as certain tasks must meet minimum performance. Benchmark scoring results can be used for subsequent model selection, so it must reliably reject incomplete candidates.

Boundary decision for this stage: When a critical case is missing, raise ValueError to reject the candidate; when no critical case is missing but all cases lack results, also raise ValueError to avoid division by zero.

Start with a concrete problem

Why can’t the selection report just say “best model”?

In the previous stage, you used benchmark tests to obtain scores for each candidate model and selected the best model for the current task. Now you need to write the selection result into a report for team review.

If the report only says “we selected the balanced model”, reviewers cannot know why other models were rejected, nor under what conditions this choice needs to be re-evaluated.

The goal of this stage is to write the build_report function that records the selected model, benchmark score, rejection reasons, assumptions, and retest conditions, forming an auditable decision document.

Make a prediction

Before starting implementation, predict: what happens if the build_report function only returns the selected model name and a phrase “best model”?

• The test will pass because the report already states the selection result.
• The test will fail because the report lacks rejection reasons and retest conditions.
• The test will fail because the report does not include the benchmark score.
• The test will pass because reviewers can infer rejection reasons themselves.

Reasoning guide: The correct answer is the second option. The test test_report_preserves_rejections_and_retest_triggers explicitly requires the report to contain rejection reasons in the rejected dictionary and retest conditions in the retest_when list. Writing only “best model” will cause a KeyError because the report does not have the key cheap.

What it is

What a selection report is

A selection report is an immutable data structure that fixes the key evidence in the decision process, including the selected model, benchmark score, reasons for each rejected model, assumptions made at the time of selection, and future triggers for re-evaluation.

Each field in the report comes from the input parameters of build_report; the function itself does no new calculations, only organizes and validates this information.

What it is not

What a selection report is not

A selection report is not free text; it cannot just say “we selected the best model”. It must structurally record all rejection reasons and retest conditions so that reviewers can reconstruct the decision process.

The report also does not recalculate scores or introduce external data; it only uses the scores and reasons provided by the caller.

How it relates to neighboring concepts

Relationships with other components

build_report receives the scores dictionary from the benchmark stage, as well as the rejected reasons and retest_when conditions produced by the task constraint stage.

The reasons field in the report must include the selected model’s benchmark score in the format benchmark_score=0.870, so reviewers can verify the score source.

Boundary decision for this stage: Report construction uses only the provided scores, rejection reasons, assumptions, and retest conditions; it does not involve new calculations or external data. If the selected model has no benchmark score, the function must raise ValueError because the selection basis cannot be verified.