Diagnostic & Evidence Map

Start with a concrete problem

Why do we need three distinct scenarios?

Forge is a multi-scenario agent platform, but the current codebase only contains forge/init.py with no scenario definitions. The test test_catalog_covers_three_distinct_scenarios requires scenario_catalog() to return a dictionary with exactly the keys software, office, and music, and each scenario must have a unique user, non-empty request, and non-empty outcome.

If only two scenarios are defined, the test fails because the assertion set(catalog) == {“software”, “office”, “music”} is not satisfied. You need to create forge/scenarios.py, define a Scenario data class, and implement scenario_catalog() to return three distinct scenario instances.

Make a prediction

Before implementing, predict: if scenario_catalog() returns only software and office scenarios, what will happen to test_catalog_covers_three_distinct_scenarios?

• The test passes because two scenarios are sufficient.
• The test fails because the assertion requires exactly three keys.
• The test fails because missing the music scenario causes insufficient users.
• The test passes because users are different.

Reasoning guide: The correct answer is the second option. The test explicitly asserts set(catalog) == {“software”, “office”, “music”}; missing music causes a set mismatch and the test fails.

What it is

What a scenario catalog is

A scenario catalog is a mapping from scenario IDs to Scenario objects, each containing id, user, request, and outcome fields. It demonstrates Forge’s applicability across different domains (software, office, music).

The scenario catalog is static and deterministic: every call to scenario_catalog() returns the same three scenarios, independent of external state or randomness.

What it is not

What a scenario catalog is not

A scenario catalog is not a complete implementation specification; it only describes users, requests, and observable outcomes without concrete execution steps or technical details.

A scenario catalog is not dynamically generated or read from a database or configuration file; it is hardcoded constant data in forge/scenarios.py.

How it relates to neighboring concepts

Relationships with other components

The scenario catalog is part of the Forge product brief and provides input for subsequent acceptance tests. Tests test_catalog_covers_three_distinct_scenarios and test_scenarios_describe_observable_outcomes directly verify the structure and content of the scenario catalog.

The Scenario data class uses dataclass(frozen=True) to ensure scenario objects are immutable and prevent accidental modification.

Boundary decision for this stage: The scenario catalog must contain three scenarios with distinct users, requests, and outcomes, but it does not need to include implementation details.

Start with a concrete problem

User stories expand without limit: missing scope boundary

In the software delivery scenario, frontend maintainers often complain about unclear acceptance criteria, making patches unreviewable. If user outcomes only describe desired results without explicit exclusions, the team keeps adding new requirements and the story never finishes.

Currently forge/users.py does not exist, so tests test_user_outcome_requires_a_scope_boundary and test_complete_user_outcome_is_accepted fail with an import error. We need to implement the UserOutcome dataclass and validate_user_outcome function to enforce that every user outcome includes at least one out_of_scope item.

Make a prediction

If validate_user_outcome only checks that role and desired_result are non-empty and allows out_of_scope to be an empty tuple, what will happen?

• test_user_outcome_requires_a_scope_boundary will pass because an empty tuple is a valid input.
• test_user_outcome_requires_a_scope_boundary will fail because a missing scope boundary should raise ValueError.
• test_complete_user_outcome_is_accepted will fail because an empty tuple causes a validation error.
• Both tests will pass because the validation function does not need to check out_of_scope.

Reasoning guide: The correct answer is the second option. The test test_user_outcome_requires_a_scope_boundary uses pytest.raises to expect a ValueError; if the validation function does not check out_of_scope, no exception is raised and the test fails. This reminds us that a scope boundary is a necessary part of a user outcome.

What it is

User outcome is a testable contract with boundaries

UserOutcome is a frozen dataclass with five fields: scenario_id, role, pain, desired_result, and out_of_scope. It describes a role’s pain, desired result, and explicit exclusions in a specific scenario.

The validate_user_outcome function checks that role, pain, and desired_result are non-empty, and that out_of_scope contains at least one element. Only outcomes satisfying these conditions pass validation and become reliable inputs for subsequent work.

What it is not

User outcome is not an unbounded wish list

A user outcome is not just a list of desired results without limits. If out_of_scope is allowed to be empty, the user story expands without limit and the team cannot determine when it is done.

A user outcome is also not a self-evaluation of model output. Validation must rely on externally checkable fields and rules, not subjective opinions.

How it relates to neighboring concepts

Relationship between verification and user outcomes

The validation function validate_user_outcome is the gatekeeper for user outcomes. It ensures every user outcome includes an explicit exclusion boundary, preventing scope creep.

The tests test_user_outcome_requires_a_scope_boundary and test_complete_user_outcome_is_accepted verify that a missing boundary raises an exception and a complete outcome is accepted, providing double protection for the validation function.

Boundary decision for this stage: When out_of_scope is an empty tuple, validate_user_outcome must raise ValueError because a user outcome without a scope boundary cannot prevent unbounded expansion.

Start with a concrete problem

Validator treats optional fields as required, causing valid requests to be rejected

You are defining input/output contracts for Forge scenarios. The contract must clearly specify which inputs are required, which are optional, and what output artifacts and acceptance checks must be produced.

In the current code, the IOContract.validate() method incorrectly computes missing fields from optional_inputs instead of required_inputs. This causes a valid request containing only required fields to be reported as missing the optional field deadline, while a request actually missing the required field workspace is ignored.

The test test_only_required_inputs_are_reported_missing expects: when request and workspace are provided, return an empty list; when only request is provided, return [‘workspace’]. But the current implementation returns [‘deadline’] and [], completely opposite.

Make a prediction

In IOContract.validate(), from which input set should the missing fields list be computed?

• required_inputs
• optional_inputs
• The union of required_inputs and optional_inputs
• The intersection of required_inputs and optional_inputs

Reasoning guide: The correct answer is required_inputs. The purpose of validation is to ensure all required inputs are present; missing optional inputs should not be reported as errors. If computed from optional_inputs, optional fields are treated as required, causing valid requests to be rejected.

What it is

Input/output contract is a machine-checkable boundary for scenarios

IOContract is an immutable dataclass containing four tuples: required_inputs, optional_inputs, output_artifacts, and acceptance_checks.

The validate(payload) method takes a dictionary and returns a list of missing required inputs. If output_artifacts or acceptance_checks is empty, it raises ValueError.

What it is not

The contract is not a comprehensive validation of all fields

The contract does not check for the presence of optional inputs, nor does it check the type or format of input values. It only reports missing required inputs.

The contract does not execute acceptance checks; it only ensures that the acceptance checks list is non-empty.

How it relates to neighboring concepts

Relationship between contract, scenarios, and users

Scenarios (scenarios.py) and users (users.py) are consumers of the contract. The contract defines the input conditions that must be satisfied before a scenario executes.

Acceptance checks are machine-checkable conditions that must pass after scenario completion, and output artifacts are files or data that the scenario must generate.

Boundary decision for this stage: Validation reports only missing required inputs; missing optional inputs are not errors. Output artifacts and acceptance checks must be non-empty, otherwise the contract is invalid.

Start with a concrete problem

A vague error cannot distinguish three failure types

The file forge/failures.py does not exist yet, so tests/test_stage.py fails during collection with ModuleNotFoundError when importing IntakeStatus and classify_intake.

Even if we create the file and implement a simple classifier, returning a single generic error status would fail test_destructive_request_requires_approval, which asserts that a request with destructive=True must return NEEDS_APPROVAL, and test_missing_and_unsupported_inputs_are_distinct, which asserts that missing workspace returns NEEDS_CLARIFICATION and media_type=‘video’ returns UNSUPPORTED.

Therefore, the learner must implement a classifier that distinguishes different failure categories rather than masking all problems with one vague error.

Make a prediction

Before implementing classify_intake, predict: for payload={“request”:“clean repo”,“workspace”:“repo”,“destructive”:True}, what status should the classifier return?

• READY, because request and workspace are present
• NEEDS_APPROVAL, because destructive is True
• NEEDS_CLARIFICATION, because other fields are missing
• UNSUPPORTED, because destructive is not a supported media type

Reasoning guide: The correct answer is NEEDS_APPROVAL. Destructive operations are potentially harmful and must pass through an approval boundary; they cannot be marked READY just because basic fields are present.

What it is

Failure classification is a decision tree

classify_intake checks inputs in a fixed order: first whether required fields are missing, then whether the operation is destructive, and finally whether the media type is supported.

Each check corresponds to a distinct IntakeStatus enum value, allowing callers to take different subsequent actions based on the status.

What it is not

It is not generic error handling

Failure classification does not return a generic error string or raise an exception; it uses enum values to precisely express the failure reason.

It also does not execute approval or clarification flows; it only marks the status, and later components handle the actual process.

How it relates to neighboring concepts

Relationship between statuses and check order

The check order determines priority: the missing-field check comes first because without necessary information later judgments cannot be made; the destructive check precedes the media-type check because destructive operations require approval even if the media type is supported.

The IntakeStatus enum defines all possible states, and classify_intake must return one of them.

Boundary decision for this stage: The classifier only makes decisions based on static fields of the input payload; it does not access external systems or modify any state, and it only returns a clear IntakeStatus.

Start with a concrete problem

Completion check incorrectly reports success when required evidence is missing

The file forge/evidence.py does not exist yet, so test collection fails with ModuleNotFoundError because the test file tries to import all_checks_pass and record_evidence.

Even if we create the file and implement a simple all_checks_pass that only checks the passed field of existing records, it will ignore missing check IDs from the required set.

In the fault fixture, all_checks_pass returns True for input that has only a schema record and is missing failure-case, while test_missing_required_evidence_prevents_success expects False.

The root cause is that completion is based only on present records, not the declared acceptance set, so partial evidence is incorrectly treated as complete.

Make a prediction

If all_checks_pass only checks the passed field of each record in the input list, what will it return when the required set contains a check ID that is not in the records?

• True, because all existing records passed
• False, because a required check is missing
• KeyError, because the check ID cannot be found
• None, because it cannot decide

Reasoning guide: The correct answer is True. If the implementation only iterates over existing records and checks passed, it will not notice the missing ID in required, so it will incorrectly conclude that all required checks passed. This is exactly the defect in the fault fixture.

What it is

Contract for evidence records and completion checks

EvidenceRecord is an immutable dataclass with four fields: check_id, artifact, digest, and passed, where digest is the SHA-256 hash of the content bytes as a hex string.

The record_evidence function creates an evidence record, requiring check_id and artifact to be non-empty, otherwise it raises ValueError, then computes the content hash and returns the record.

The all_checks_pass function takes a list of records and a required set, and must verify that every check ID in required exists in the records and the corresponding record’s passed is True.

This contract ensures that a success claim is backed by a complete and passing evidence set, not just a partial set.

What it is not

Not just checking existing records

all_checks_pass must not only check the passed field of each record in the input list, because that would ignore missing check IDs from the required set.

It is also not a semantic validation of evidence content; it only makes a formal judgment based on check ID existence and the passed boolean.

The evidence digest is not based on filename or path, but on content bytes, so identical content under different filenames produces the same digest.

How it relates to neighboring concepts

Relationship between evidence, tests, and acceptance

The test test_missing_required_evidence_prevents_success constructs a list with only a schema record and passes required set {“schema”,“failure-case”}, expecting all_checks_pass to return False.

The test test_evidence_digest_is_content_addressed verifies that identical content under different artifact names yields the same digest, ensuring the digest is content-addressed.

The defective implementation in the fault fixture only checks existing records’ passed, causing the test to fail; the fix must check both the containment of required and the passed status of each required record.

Boundary decision for this stage: all_checks_pass must require that the required set is a subset of the check_id set in records, and that every required record’s passed is True; any missing or failing required check causes False.

Start with a concrete problem

Compute gaps from evidence and sort by dependency weight

The forge directory already contains scenarios.py, users.py, io_contracts.py, failures.py, and evidence.py, but gaps.py is missing, causing a ModuleNotFoundError during test collection.

You need to create forge/gaps.py, implement the Capability dataclass and assess_gaps function, and make the tests test_blocking_gaps_are_prioritized and test_demonstrated_capability_is_not_a_gap pass.

The tests require assess_gaps to return a list of missing capability IDs sorted in descending order by dependency weight, with higher-weight capabilities first.

Additionally, if a capability’s evidence level already meets or exceeds its required level, it must not appear in the result.

Make a prediction

Before implementing assess_gaps, predict: if you sort missing capabilities by dependency weight in ascending order, what will test_blocking_gaps_are_prioritized return?

• Returns [‘python’, ‘slides’], test passes
• Returns [‘slides’, ‘python’], test fails
• Returns [‘http’, ‘python’, ‘slides’], test fails
• Returns [‘python’, ‘slides’, ‘http’], test fails

Reasoning guide: The correct answer is that it returns [‘slides’, ‘python’] and the test fails. Ascending order puts the lower-weight slides first, while the test expects the higher-weight python first.

What it is

Capability gap assessment model

assess_gaps takes a list of capabilities and a dictionary of evidence levels, filters out capabilities whose evidence level is below their required level, and returns the IDs of the missing capabilities.

The missing capabilities are sorted in descending order by dependency weight so that higher-weight capabilities, which may block later modules, are addressed first.

The Capability dataclass has three fields: id, required_level, and dependency_weight, where dependency_weight indicates how much other capabilities depend on this one.

What it is not

Not a simple list filter

This model does not return all capability IDs; it only returns those whose evidence level is below the required level.

It does not sort alphabetically or arbitrarily; it must sort by dependency weight in descending order to ensure high-priority capabilities are handled first.

It does not modify evidence levels or capability definitions; it only computes gaps and sorts them.

How it relates to neighboring concepts

Relationships with other modules

assess_gaps depends on the Capability dataclass to define the capability structure and uses the evidence_levels dictionary to obtain current evidence levels.

The sorting result directly influences subsequent learning path planning, as higher-weight capabilities should be scheduled first.

This module works with evidence.py, where evidence levels come from actual tests or evaluations, not subjective ratings.

Boundary decision for this stage: When evidence level equals or exceeds required_level, the capability is not a gap; when sorting, only missing capabilities are considered, and they must be sorted by dependency_weight in descending order.

Start with a concrete problem

Why do we need topological sorting?

In the diagnostic module, multiple capabilities have dependencies, for example engineering depends on diagnostic, and agent-loop depends on engineering.

If the learning route does not respect these dependencies, learners may encounter agent-loop content first without the engineering and diagnostic foundations, leading to confusion.

Therefore we need a function build_route() that takes a capability dependency graph and returns a topological order satisfying all dependencies.

When the dependency graph contains a cycle, such as A depends on B and B depends on A, no linear order can satisfy both, so it must be explicitly rejected.

Make a prediction

Given the dependency graph {“diagnostic”: set(), “engineering”: {“diagnostic”}, “agent-loop”: {“engineering”}}, what would be the result if we output nodes in alphabetical order?

• [‘agent-loop’, ‘diagnostic’, ‘engineering’]
• [‘diagnostic’, ‘engineering’, ‘agent-loop’]
• [‘engineering’, ‘diagnostic’, ‘agent-loop’]
• [‘diagnostic’, ‘agent-loop’, ‘engineering’]

Reasoning guide: Alphabetical order would produce [‘agent-loop’, ‘diagnostic’, ‘engineering’], but agent-loop depends on engineering, and engineering depends on diagnostic, so agent-loop appearing first violates the dependencies.

What it is

Topological sorting is dependency-driven linearization

Topological sorting arranges the nodes of a dependency graph into a linear sequence such that every dependency edge points from a prerequisite to a successor.

In build_route(), we maintain a set of remaining nodes and repeatedly output only those nodes whose dependency set is empty, then remove those output nodes from the dependency sets of other nodes.

This process repeats until all nodes are output; if an iteration has no ready nodes but remaining nodes still exist, a dependency cycle is present.

What it is not

Topological sorting is not an arbitrary order

Topological sorting does not output nodes in alphabetical, insertion, or random order; it must satisfy all dependency constraints.

It is also not the direct result of depth-first search or breadth-first search, although those algorithms can assist in implementing topological sorting.

Topological sorting does not guarantee uniqueness: when multiple nodes are ready simultaneously, any order among them is acceptable as long as dependencies are preserved.

How it relates to neighboring concepts

Relationship between dependency graph and route

The dependency graph is the input and the route is the output; each node in the route must appear after all of its dependency nodes.

If the dependency graph contains a cycle, no valid topological sort exists, so build_route() must raise a ValueError.

In the diagnostic module, the capability dependency graph is provided by modules such as forge/gaps.py, and the route generated by build_route() will guide subsequent learning stages.

Boundary decision for this stage: When the dependency graph contains a cycle, build_route() must raise a ValueError containing the word ‘cycle’ rather than returning a partial route or looping indefinitely.