End-to-End Portfolio Delivery

Start with a concrete problem

Why does a non-empty check fail to prevent path escape?

In forge/portfolio/scenarios.py, ScenarioContract currently only checks whether the scenario name is allowed and whether the input, output, and acceptance tuples are non-empty.

When the input path is ../private.csv, validate() does not raise an error because the path is non-empty and the scenario office is valid.

This means the contract can reference files outside the package, breaking the closure of the delivery package and potentially leaking sensitive data from the workspace.

This stage requires completing the path confinement logic so that any absolute path or path containing .. is rejected.

Make a prediction

Before completing the path check, predict: if we only check that input paths are non-empty, what happens when calling ScenarioContract("office", ("../private.csv",), ("brief.md",), ("citations present",)).validate()?

• Raises ValueError because the path contains ..
• Returns normally because the path is non-empty and the scenario is valid
• Raises ValueError because the scenario office is not allowed
• Raises TypeError because the path is not a string

Reasoning guide: The current code only checks non-emptiness and does not check path content, so ../private.csv is accepted and validate() returns normally. This is exactly why the test test_scenario_path_escape_is_rejected fails.

What it is

Contract validation is a combination of completeness and path confinement

ScenarioContract.validate() must check three things: the scenario name is in the allowed set, all tuples are non-empty, and input paths do not escape the package directory.

Path confinement is achieved by rejecting absolute paths (starting with /) and paths containing .., ensuring inputs can only come from within the package.

What it is not

Contract validation is not just checking non-emptiness

Checking non-emptiness alone cannot prevent paths like ../private.csv because the path is non-empty and the scenario is valid.

Contract validation is also not a check of file content or output correctness; it only ensures the contract’s completeness and path safety.

How it relates to neighboring concepts

Relationship between path check, scenario, and completeness

The scenario check ensures the contract belongs to a known scenario, the completeness check ensures necessary fields exist, and the path check ensures input sources are controlled.

All three are indispensable: if the path check is omitted, the contract may reference files outside the package; if the completeness check is omitted, the contract may lack necessary fields.

Boundary decision for this stage: The path check must reject both absolute paths and paths containing .., because absolute paths directly point to arbitrary locations in the filesystem, while .. can escape the current directory. Rejecting only one of them can still be bypassed.

Start with a concrete problem

Delivery result missing required output is accepted

In Stage 01, we defined ScenarioContract, which specifies which output files and acceptance checks each scenario must deliver. Now we need to implement verify_delivery to ensure the delivery result fully satisfies the contract.

The starting code does not yet have forge/portfolio/integration.py, so tests fail at import. The learner must create this file and implement the DeliveryResult dataclass and verify_delivery function.

A common mistake is to only check whether the result has any output, ignoring the complete set of required outputs. For example, the software scenario contract requires outputs patch.diff and test.txt, but the result only contains patch.diff; checking non-emptiness would incorrectly accept this incomplete delivery.

The goal of this stage is to understand that verification must perform set subtraction against contract requirements, not just check non-emptiness. By implementing correct validation logic, we ensure all required outputs and acceptance checks are present and trace_id is non-empty.

Make a prediction

What happens if verify_delivery only checks that result.outputs is non-empty?

• It correctly rejects the delivery missing test.txt.
• It incorrectly accepts the delivery missing test.txt.
• It crashes due to a type error.
• It ignores acceptance checks.

Reasoning guide: The correct answer is the second option. Checking only non-emptiness cannot detect missing required outputs because the result has at least one output file and passes the check. The correct approach is to subtract the actual output set from the contract’s required output set; if the difference is non-empty, there are missing items.

What it is

Contract coverage verification

Contract coverage verification is a set-based validation method: compute the set difference between the contract’s required outputs and acceptance checks and the actual outputs and passed checks in the result.

If the difference is empty, all requirements are satisfied; if non-empty, there are missing items and the delivery is incomplete.

This verification is explicit and checkable, not relying on subjective judgment, and can precisely identify missing items.

What it is not

Not a non-emptiness check

A non-emptiness check only verifies that the result has any output, not that all required outputs are present.

It cannot detect missing specific files or checks, so it incorrectly accepts incomplete deliveries.

Contract coverage verification requires checking each required item individually, not just a blanket existence check.

How it relates to neighboring concepts

Relationship with ScenarioContract

ScenarioContract defines the expected_outputs and acceptance sets, and verify_delivery uses these sets as the baseline for validation.

DeliveryResult stores the actual delivered outputs and passed checks, and verify_delivery compares the two.

Validation passes when: scenario name matches, all expected_outputs are in result.outputs, all acceptance are in result.passed_checks, and trace_id is non-empty.

Boundary decision for this stage: Verification must perform set subtraction against contract requirements, not just check non-emptiness. If only non-emptiness is checked, missing required outputs or acceptance checks are missed, leading to incorrect acceptance of incomplete deliveries.

Start with a concrete problem

Why do identical inputs produce different manifests?

You have just completed the delivery verification in Stage 02, and now you need to package the release files into a manifest.

The test test_same_inputs_produce_same_manifest requires that two calls to build_manifest("1.0", {"a.txt": b"a"}) return exactly the same dictionary.

However, the file forge/portfolio/packaging.py has not been created yet, so the test collection phase reports ModuleNotFoundError.

You need to implement a function that produces identical output for identical input, which is the foundation of reproducible releases.

Make a prediction

When implementing build_manifest, which of the following practices is most likely to break reproducibility?

• Sorting file paths before generating the manifest
• Including the current timestamp in the manifest
• Using SHA-256 to hash file contents
• Fixing key order during JSON serialization

Reasoning guide: Including a timestamp makes each build’s manifest different even if the input files are identical, so it will fail test_same_inputs_produce_same_manifest. Sorting and fixed serialization order are key to ensuring determinism.

What it is

What a reproducible manifest is

A reproducible manifest is a deterministic function: given the same version number and file contents, it always generates exactly the same manifest dictionary.

It depends only on the version number and the sorted file entries, and contains no time-varying metadata.

The manifest digest manifestSha256 is a hash of a canonical JSON string, so identical inputs necessarily yield the same digest.

What it is not

What a reproducible manifest is not

It is not a log that includes build time, build serial number, or random numbers; such non-deterministic information would cause identical inputs to produce different digests.

It is not simply outputting the file dictionary as-is, because dictionary ordering may be unstable and must be explicitly sorted.

It does not verify whether file contents are correct; it only records file paths, hashes, and sizes.

How it relates to neighboring concepts

Relationship to other concepts

A reproducible manifest extends the delivery verification from Stage 02: after verification passes, the manifest freezes the release content.

It relies on a hash function (SHA-256) to uniquely identify file contents; any content change alters the hash.

Sorting is crucial for determinism because JSON serialization key order can affect the final string.

Boundary decision for this stage: If the manifest includes a timestamp, identical inputs will produce different digests, so all non-deterministic metadata must be excluded, keeping only the version and sorted file entries.

Start with a concrete problem

Demo script missing evidence path becomes unverifiable

You are preparing a demo script for HeatStack Forge with three steps: show request, execute command, and show test results.

The current validate_demo only checks the narration field, ignoring command and evidence path, so a step missing an evidence path still passes validation.

The test test_demo_without_evidence_path_is_rejected expects a ValueError containing ‘incomplete’ when the evidence path is empty, but the current code accepts this incomplete step.

You need to modify the validate_demo function in forge/portfolio/demo.py to ensure narration, command, and evidence path are all non-empty for every step.

Make a prediction

What happens if validate_demo only checks the narration field?

• Steps missing an evidence path are accepted, making the demo unverifiable
• Steps missing a command are rejected, but steps missing an evidence path are accepted
• All steps are rejected because the narration field might be empty
• The demo script cannot run because required fields are missing

Reasoning guide: The correct answer is the first option. Checking only narration allows steps with missing evidence paths to pass because the validation logic does not cover command and evidence path.

What it is

Demo validation is a completeness check

Demo validation ensures every step in the demo script has verifiable evidence by checking that narration, command, and evidence path are non-empty.

It uses the all() function to check all three fields simultaneously, raising a ValueError if any is empty.

What it is not

Demo validation is not content quality assessment

Demo validation does not check whether the narration is eloquent, the command is optimal, or the evidence file actually exists; it only checks that fields are non-empty.

It also does not check logical ordering between steps or uniqueness of step IDs; those are responsibilities of other validation logic.

How it relates to neighboring concepts

Relationship between demo validation and tests

The test test_complete_demo_passes verifies that a complete demo passes, while test_demo_without_evidence_path_is_rejected verifies that a demo missing an evidence path is rejected.

The validate_demo function is central to both tests; it must satisfy both passing and rejection conditions.

Boundary decision for this stage: Demo validation must require narration, command, and evidence path for every step; missing any of them makes the demo unverifiable.

Start with a concrete problem

Why does a record with only decision ID and choice fail validation?

In the previous stage, we verified demo script step completeness with DemoStep and validate_demo, but architecture decision records (ADRs) were not yet included in validation.

The current test test_decision_without_alternatives_is_rejected expects that when alternatives is an empty tuple, DecisionRecord.validate() must raise a ValueError containing 'tradeoff'.

However, if validate() only checks that decision_id and choice are non-empty, a record without alternatives, consequences, and evidence will be accepted, causing the test to fail.

You need to implement DecisionRecord.validate() to ensure it rejects architecture decision records that lack tradeoff rationale.

Make a prediction

Before implementing DecisionRecord.validate(), predict: what happens if we only check that decision_id and choice are non-empty?

• All tests pass because decision ID and choice are sufficient.
• test_decision_without_alternatives_is_rejected fails because empty alternatives is not rejected.
• test_complete_decision_record_passes fails because a complete record is also rejected.
• Both tests fail because validate() is not implemented.

Reasoning guide: The correct answer is the second option. Checking only decision_id and choice allows a record with empty alternatives to pass validation, triggering the AssertionError in the test.

What it is

What is architecture decision record (ADR) validation?

ADR validation is an externally checkable rule that ensures each decision record contains not only decision ID and choice, but also context, alternatives, consequences, and evidence.

DecisionRecord.validate() checks that these fields are non-empty and that the choice is not listed among the alternatives, guaranteeing the record includes sufficient tradeoff rationale.

What it is not

What is ADR validation not?

It is not asking the model for another opinion about its own decision, nor is it merely checking that a decision ID exists.

It does not care whether the decision content is correct; it only cares whether the record completely presents the decision’s context, alternatives, consequences, and evidence.

How it relates to neighboring concepts

How ADR validation relates to other concepts

ADR validation is a concrete application of the Verification concept: it checks one result with rules and tests, rather than measuring behavior across runs with datasets and metrics.

It relates to Observability because a complete ADR provides decision context and evidence, helping explain why the system is designed this way.

It relates to the Agent Loop’s stopping condition: if the decision record is incomplete, validation fails, and the loop should continue gathering information.

Boundary decision for this stage: Boundary decision: validation must rely on externally inspectable evidence (non-empty fields, choice not in alternatives), not on the model’s subjective judgment.

Start with a concrete problem

The evidence index must prevent conflicting hashes for the same path

In portfolio delivery, we need to organize evidence produced by different scenarios (such as files, reports, test results) into a searchable index so that it can be quickly looked up by capability or scenario later.

The file forge/portfolio/evidence_index.py has not been created yet, so the tests test_evidence_is_grouped_by_capability and test_conflicting_hash_for_same_path_is_rejected fail with an import error.

If we simply append each evidence entry to a list without checking whether the same path has already been recorded with a different content hash, the index becomes self-contradictory: the same file path corresponds to two different content fingerprints, and downstream consumers cannot determine which one is real.

Therefore, the core decision in this stage is: maintain a path-to-hash mapping while building the index, and as soon as a different hash is found for the same path, immediately raise a ValueError containing conflicting, rather than silently overwriting or coexisting.

Make a prediction

In build_index, if the same path appears with two different sha256 values, which of the following approaches best satisfies the immutable identity requirement of an evidence index?

• Append the second entry directly, letting the list contain both hashes for the consumer to decide.
• Overwrite the previous hash with the later one, keeping the path unique.
• Raise a ValueError when a conflict is detected, with the error message containing conflicting.
• Ignore the second entry and keep only the first hash.

Reasoning guide: An evidence index must guarantee a one-to-one correspondence between path and content hash; otherwise, the same path pointing to two different contents makes it impossible for downstream consumers to determine the true evidence. Therefore, when a conflict is detected, it should be rejected by raising an exception rather than overwriting or coexisting.

What it is

The evidence index is an immutable mapping from path to hash

The evidence index groups each evidence entry by capability and records the path, scenario, type, and content hash within each group.

It also maintains a global path-to-hash mapping used to check whether a path has already appeared with a consistent hash when inserting a new entry.

If the same path has the same hash, it can be safely referenced again; if the hash differs, the content has changed or there is a conflict, and it must be rejected.

What it is not

The evidence index is not a simple append-only list

It cannot allow the same path to carry multiple different hashes, because that would make the index untrustworthy.

It is also not a storage structure that automatically deduplicates or overwrites; conflicts must be explicitly raised as exceptions so that the caller can decide how to handle them.

It does not verify the correctness of the hash itself; it only maintains the consistency constraint between path and hash.

How it relates to neighboring concepts

Relationship between the index, evidence items, and capability grouping

EvidenceItem is an immutable data class with five fields: path, sha256, scenario, capability, and kind.

build_index receives a tuple of EvidenceItem and returns a dictionary: keys are capability names, and values are lists of evidence entries under that capability.

During construction, the seen dictionary records the first hash for each path; if a later entry has the same path but a different hash, a conflict exception is triggered.

Boundary decision for this stage: If and only if the same path already exists with a different hash, a ValueError containing conflicting must be raised; if the hash is the same, repeated references are allowed.

Start with a concrete problem

The release gate must check all required scenarios and sections

The file forge/portfolio/release.py does not exist yet, so the tests test_complete_portfolio_publishes and test_missing_music_scenario_blocks_release fail during import.

You need to implement the PortfolioRelease dataclass and the publish function so that a complete portfolio can be published, while a portfolio missing the music scenario is blocked with a ValueError containing ‘music’.

The release gate cannot merely check that the scenario set is non-empty; it must validate against the required scenario and section sets.

Make a prediction

If the publish function only checks whether release.scenarios is non-empty, what happens when a portfolio containing only the software and office scenarios is passed?

• Publishing succeeds because the scenario set is non-empty
• A ValueError is raised because the music scenario is missing
• An empty string is returned
• A TypeError is raised

Reasoning guide: The correct answer is ‘Publishing succeeds because the scenario set is non-empty’. Checking only non-emptiness fails to detect missing required scenarios, which is exactly the defect injected in the fault fixture.

What it is

The release gate is a completeness check against required sets

The release gate compares the scenario set included in the portfolio with the required scenario set defined by the REQUIRED_SCENARIOS constant; any missing scenario causes the release to fail.

Similarly, it checks the required section set REQUIRED_SECTIONS to ensure that key sections such as architecture, demo, and evidence index are present.

Only when both sets have no missing elements does publish return a release identifier.

What it is not

The release gate is not a simple non-empty check

The release gate does not consider a portfolio complete merely because the scenario set has at least one element; it must contain all three scenarios: software, office, and music.

It is also not a quality score for the portfolio, but a hard validation of the release contract; any missing element blocks the release.

The release gate does not generate missing content; it only detects and reports missing items.

How it relates to neighboring concepts

Relationship between the release gate and portfolio components

The PortfolioRelease dataclass encapsulates the version, scenario set, section set, and artifact hash, and the publish function uses these fields for validation.

The constants REQUIRED_SCENARIOS and REQUIRED_SECTIONS define the release contract, and the test cases construct inputs and assert behavior based on this contract.

The defect in the fault fixture is that it only checks scenario non-emptiness, causing a portfolio missing the music scenario to be incorrectly published.

Boundary decision for this stage: The release gate must check scenarios and sections against required sets; any missing element blocks release; it cannot merely check non-emptiness or the presence of any scenario.