Architecture position

Production Engineering

Use evaluation, observability, deployment, and evidence to prove the system is reliable under real constraints.

How do we prove reliability and complete deployment, portfolio evidence, and interview communication?

Governed by

Concept calibration

Verification & Evaluation

What it is: Use tests, schemas, citations, state, and acceptance rules to verify outputs, then measure system behavior across datasets.
What it is not: It is not asking a model for another opinion about its own output; verification must resolve to externally inspectable evidence.

Observability

What it is: Use structured logs, traces, metrics, and version data to explain what happened during a system run.
What it is not: It is not just terminal output; logs without correlation IDs, state transitions, and privacy boundaries are hard to diagnose.

Harness

What it is: The runtime control environment that hosts models, tools, state, permissions, logs, budgets, human approval, and recovery.
What it is not: It can constrain risk and preserve evidence, but it cannot directly prevent model hallucinations.

Evaluation, Observability, Deployment & Reliability

Build eval sets, tracing, quality and cost metrics, deployment, rollback, and incident drills.

One thing to completeProduce a deployable, observable, and rollback-ready Forge release candidate.

Before you begin, Forge already hasThe accepted solution from Module 15, Cross-Platform Adapters & Migration, is this chapter’s starting point.

After this chapter, Forge canCreate an observable release covering quality, latency, cost, failure, and rollback.

Smallest recovery pointRun local evals and structured traces until failures are diagnosable.

Download this chapter labPython 3.12 · pytest · Pydantic · SQLite · deterministic Mock

Stage capability chain

01Define evaluation cases with stable IDs, inputs, and acceptance criteria while rejecting duplicates02Compute accuracy, citation coverage, and tool success from case-level results while retaining denominators03Record parent/child spans while keeping one trace ID consistent across a run04Aggregate run cost and latency and produce an actionable release gate for budget violations05Bind evaluation, migration, and security checks to an immutable release manifest before promotion06Compare baseline and canary error/latency metrics and recommend rollback under explicit thresholds07Record detection, mitigation, recovery, and review timestamps while validating recovery objectives and evidence

Each step adds one verifiable capability; every later step begins from code and tests accepted in the previous step.

Concept calibration\n\nPut the concepts used in this module back inside their engineering boundaries before you work with the code.\n\n
\n\n### Verification & Evaluation evaluation\n\n- What it is: Verification checks one result with tests, schemas, citations, state, or acceptance rules; evaluation measures behavior across runs with datasets and metrics.\n- What it is not: It is not asking the model for another opinion about its own answer; pass or fail must resolve to externally inspectable evidence.\n- Relationship to adjacent concepts: The agent loop relies on verification to stop, RAG needs layered evaluation, planning needs node acceptance, and observability supplies run records.\n- Where it lives in HeatStack Forge: Forge preserves contract tests and evidence for each run, then compares quality, latency, cost, permission, and recovery across fixed scenarios.\n- Typical misuse and correction: A common misuse treats one aggregate score as the whole conclusion. Define layered metrics, failure samples, and version baselines, then review product risk.\n\n### Observability observability\n\n- What it is: Structured logs, traces, metrics, state transitions, and version information that explain a run and support failure diagnosis and version comparison.\n- What it is not: It is not merely terminal output or unlimited input recording; logs without correlation IDs, state transitions, or privacy boundaries are hard to diagnose and may leak data.\n- Relationship to adjacent concepts: The harness emits runtime events, the agent loop and tool calls form traces, evaluation aggregates metrics, and memory and recovery depend on versioned records.\n- Where it lives in HeatStack Forge: Forge creates an operation_id for each task and records stage, tool, latency, cost, error class, state change, and evidence path while filtering sensitive data.\n- Typical misuse and correction: A common misuse adds logs only after failure. Define events and correlation fields with the contract, then use fault injection to prove the timeline can be reconstructed.\n\n### Harness harness\n\n- What it is: The runtime control environment that hosts models, tools, state, permissions, logs, budgets, human approval, isolation, and recovery.\n- What it is not: It can reduce risk, reject actions, and preserve evidence, but it cannot guarantee that model output is always correct or replace domain acceptance checks.\n- Relationship to adjacent concepts: The agent loop runs inside the harness; tool, memory, and MCP connections obey its policies; observability records execution and evaluation judges behavior.\n- Where it lives in HeatStack Forge: The Forge harness manages workspace, tools, approval queue, budget, checkpoints, audit logs, and rollback, separating model suggestions from real side effects.\n- Typical misuse and correction: A common misuse treats a system prompt as a security boundary. Enforce permissions, paths, network access, secrets, budgets, and rollback outside the model.\n\n

STAGE 01

Define evaluation cases with stable IDs, inputs, and acceptance criteria while rejecting duplicates

Start with a concrete problem

Evaluation cases need stable identifiers, otherwise duplicates break traceability

You are building an evaluation dataset to hold multiple evaluation cases. Each case contains an input and an expected output used to verify system behavior.

If two cases share the same case_id, later statistics and tracing become ambiguous: you cannot tell which input produced which result, and you might accidentally overwrite an existing case.

Currently, EvalDataset.validate() only checks that the version and case list are non-empty; it does not check that case_id values are unique.

Therefore, when test_duplicate_case_ids_are_rejected constructs two cases with case_id “same”, validate() does not raise an exception, causing the test to fail.

Make a prediction

In EvalDataset.validate(), besides checking that version and cases are non-empty, what additional check is needed to reject duplicate case IDs?

Check that every case’s expected field is non-empty
Check that all case_id values are unique
Check that version starts with ‘v’
Check that input contains specific keys

Reasoning guide: The correct answer is to check that all case_id values are unique. The test test_duplicate_case_ids_are_rejected expects a ValueError containing “unique” when two cases have the same ID. Other checks may be useful but are not required by this test.

What it is

An evaluation dataset is a versioned collection of unique cases

EvalDataset contains a version string and a tuple of cases, each with a unique case_id, input, and expected output.

The validate() method checks dataset integrity before use: version and cases are non-empty, case IDs are unique, and each case’s expected is non-empty.

Unique IDs are stable identifiers for evaluation cases, ensuring each case can be independently tracked and referenced.

What it is not

An evaluation dataset does not execute evaluations or compute metrics

It only stores and validates case definitions; it does not run models or compare outputs.

It does not guarantee that case content is business-correct, only that it satisfies structural constraints.

It is not a database; it does not automatically persist or provide query interfaces.

How it relates to neighboring concepts

Relationship between EvalCase and EvalDataset

EvalCase is a single evaluation case containing case_id, input, and expected.

EvalDataset aggregates multiple EvalCase objects and adds version information.

The validate() method iterates over all cases, checking ID uniqueness and non-empty expected to ensure the dataset is usable.

Boundary decision for this stage: When a dataset contains duplicate case_id values, validate() must raise a ValueError because duplicate IDs make evaluation results untraceable.

Experiment: evaluation-observability-deployment-01-versioned-evaluation-dataset
Input: Software delivery: add a feature to a frontend repository, modify code, and add tests.
Evidence: evidence/evaluation-observability-deployment/01-versioned-evaluation-dataset/
Gate: Pending acceptance

Worked example

Goal: Demonstrate creating EvalCase and EvalDataset and calling validate() to pass the valid case test.

First, create a valid EvalCase with a unique case_id, an input dictionary, and an expected output dictionary.

Then, place this case into an EvalDataset with a version string.

Finally, call validate(); it should not raise any exception, indicating the dataset is valid.

Implementation steps

Define an EvalCase instance: case = EvalCase(“case-1”, {“q”: “x”}, {“status”: “ok”})
Define an EvalDataset instance: dataset = EvalDataset(“v1”, (case,))
Call dataset.validate() and confirm no exception is raised.

Verified solution code: forge/reliability/__init__.py

"""Evaluation and production reliability primitives."""

Verified solution code: forge/reliability/dataset.py

from dataclasses import dataclass

@dataclass(frozen=True)
class EvalCase:
    case_id: str
    input: dict[str, object]
    expected: dict[str, object]

@dataclass(frozen=True)
class EvalDataset:
    version: str
    cases: tuple[EvalCase, ...]

    def validate(self) -> None:
        ids = [case.case_id for case in self.cases]
        if not self.version.strip() or not self.cases:
            raise ValueError("dataset version and cases are required")
        if len(ids) != len(set(ids)):
            raise ValueError("evaluation case IDs must be unique")
        if any(not case.expected for case in self.cases):
            raise ValueError("every case needs acceptance data")

python -m pytest -q tests/test_stage.py

Run the real tests

..                                                                       [100%]
2 passed in <duration>s

Why this implementation

EvalCase uses a dataclass, which automatically generates init and eq methods.

EvalDataset.validate() currently only checks version and cases non-empty, so a valid case passes.

This test verifies the basic dataset creation and validation flow.

Guided practice

Complete the duplicate ID check in validate() to make test_duplicate_case_ids_are_rejected pass.

Open forge/reliability/dataset.py.
In the validate() method, after checking version and cases non-empty, add code to check case_id uniqueness.
Hint: use a list comprehension to extract all case_id values, then compare the list length with the set length.
If duplicates are found, raise a ValueError with a message containing “unique”.
Run the test to confirm it passes.

Progressive hints

Use [case.case_id for case in self.cases] to get all IDs.
Use len(ids) != len(set(ids)) to detect duplicates.
Raise ValueError(“evaluation case IDs must be unique”).

Success criteria

test_duplicate_case_ids_are_rejected passes
validate() raises a ValueError containing “unique” when duplicate IDs are present

Fault injection and diagnosis

The current code’s validate() does not check case_id uniqueness. Running test_duplicate_case_ids_are_rejected fails with AssertionError: duplicate evaluation cases were accepted because two cases with the same ID are accepted without error.

This stage’s goal is to implement the duplicate ID check. You have learned how to extract IDs and compare set sizes, so you can locate the missing check in validate() and add the fix.

Verified fault contract

Target test: test_duplicate_case_ids_are_rejected
Faulty file: forge/reliability/dataset.py

Diagnosis order

Run the test command python -m pytest -q ../tests/test_stage.py::test_duplicate_case_ids_are_rejected and observe the failure output.
Examine the test code: it constructs two cases with case_id “same” and expects validate() to raise a ValueError.
Inspect forge/reliability/dataset.py: validate() only checks version and cases non-empty, not ID uniqueness.
Therefore, validate() does not raise, the else branch executes, and AssertionError is raised.

Fix principle: Add an ID uniqueness check in validate(): extract all case_id values into a list ids, then compare len(ids) and len(set(ids)); if they differ, raise ValueError(“evaluation case IDs must be unique”). After fixing, run python -m pytest -q ../tests/test_stage.py::test_duplicate_case_ids_are_rejected and confirm the test passes.

python -m pytest -q ../tests/test_stage.py::test_duplicate_case_ids_are_rejected

Transfer challenge

Create an EvalDataset with three distinct cases using office scenario data, ensuring version ‘office-v1’ and unique IDs with non-empty expected.

Changed condition: Change the case data from generic examples to office scenario data and set the version to ‘office-v1’, while keeping three cases with unique IDs.

Write a Python script or interactive code that creates three EvalCase instances with office scenario inputs and expected outputs, for example: case1: input {‘query’: ‘Summarize meeting minutes’}, expected {‘summary’: ‘…’}; case2: input {‘query’: ‘Extract action items’}, expected {‘actions’: […]}; case3: input {‘query’: ‘Generate report’}, expected {‘report’: ‘…’}. Ensure each case_id is unique, such as ‘office-1’, ‘office-2’, ‘office-3’. Then place them into an EvalDataset with version ‘office-v1’ and call validate() to confirm no exception.

Success criteria

The created EvalDataset has version ‘office-v1’, contains three cases, each with a unique case_id and non-empty expected.
Calling validate() does not raise an exception, and prints or returns a success message.

Run the real tests

python -m pytest -q tests/test_stage.py

Run the real tests

..                                                                       [100%]
2 passed in <duration>s

Design explanation question: Why do evaluation cases need stable IDs? What happens if two cases have the same ID?

A strong answer should include

Stable IDs ensure each evaluation case can be uniquely identified and tracked, preventing result confusion.
If two cases have the same ID, evaluation results cannot be distinguished by input, leading to potential data overwrites or statistical errors.

Mastery evidence

Passes both test_valid_dataset_has_versioned_cases and test_duplicate_case_ids_are_rejected.
Can explain why unique IDs are needed: to prevent evaluation result confusion and accidental overwrites.
Implemented ID uniqueness check in validate() using set comparison of lengths.

STAGE 02

Compute accuracy, citation coverage, and tool success from case-level results while retaining denominators

Start with a concrete problem

How can we aggregate meaningful overall quality metrics from multiple case results?

You already have the CaseResult dataclass that records pass status, citation requirements, and tool calls for a single case, but there is no function yet to aggregate these case results into overall quality metrics.

If you manually compute accuracy and citation coverage, it is error-prone, and when a metric has no applicable cases (for example, no citation requirements), a zero denominator leads to division by zero or an incorrect zero score.

In this stage you need to implement the summarize() function that takes a tuple of CaseResult objects and returns a dictionary containing case count, accuracy, citation coverage, and tool success, while correctly handling zero denominators.

Make a prediction

Before implementing summarize(), predict: when a metric’s denominator is zero (for example, no citation requirements), what value should that metric return?

Return 0.0 because there are no successful cases
Return 1.0 to indicate the metric is not applicable and should be considered successful
Raise an exception because it cannot be computed
Return None to indicate missing data

Reasoning guide: The correct answer is to return 1.0. A zero denominator means there are no applicable cases for that metric, so the system should not be penalized; this vacuous truth semantics is common in quality metrics, for example citation coverage should be considered fully satisfied when no citations are required.

What it is

What quality metric aggregation is

Quality metric aggregation is the process of combining discrete case-level results (pass/fail, valid citations, successful tool calls) into overall ratios, while preserving each metric’s denominator (total applicable cases) to reflect system performance in relevant scenarios.

The summarize() function takes a tuple of CaseResult objects and returns a dictionary where accuracy is the proportion of correct cases, citation_coverage is the proportion of valid citations out of required citations, and tool_success is the proportion of successful tool calls out of attempted calls.

What it is not

What quality metric aggregation is not

It is not a detailed review of individual cases or a subjective evaluation of model output; it only performs arithmetic aggregation based on structured fields already recorded in CaseResult.

It does not involve tracing, budgets, or any runtime context, and it does not decide whether cases should be included; those decisions are made by the upstream evaluation process.

How it relates to neighboring concepts

Relationship to other concepts

CaseResult is the input dataclass, summarize() consumes it and produces a metrics dictionary; ratio() is a helper function that safely computes ratios, handling zero denominators.

These metrics provide data for later observability dashboards and evaluation reports, but they do not themselves produce logs or traces.

Boundary decision for this stage: When the denominator is zero, the metric should return 1.0 (vacuous truth) rather than 0.0 or raising an exception, because zero applicable cases means there is no failure and the system should not be penalized.

Experiment: evaluation-observability-deployment-02-quality-metrics
Input: Software delivery: add a feature to a frontend repository, modify code, and add tests.
Evidence: evidence/evaluation-observability-deployment/02-quality-metrics/
Gate: Pending acceptance

Worked example

Goal: Demonstrate how summarize() computes accuracy and citation coverage from two CaseResult objects.

Suppose you have two cases: the first case passes, requires 2 citations and both are valid, attempts 1 tool and succeeds; the second case fails, requires 1 citation and it is valid, attempts 1 tool but fails.

We call summarize() and observe the returned dictionary, where accuracy should be 0.5 (one pass, one fail), and citation coverage should be 2/3 (2 valid out of 3 required citations).

Implementation steps

Create two CaseResult instances with the appropriate arguments.
Call summarize((case1, case2)).
Check the accuracy and citation_coverage values in the returned dictionary.

Verified solution code: forge/reliability/metrics.py

from dataclasses import dataclass

@dataclass(frozen=True)
class CaseResult:
    passed: bool
    citations_required: int = 0
    citations_valid: int = 0
    tools_attempted: int = 0
    tools_succeeded: int = 0

def ratio(numerator: int, denominator: int) -> float:
    return 1.0 if denominator == 0 else numerator / denominator

def summarize(results: tuple[CaseResult, ...]) -> dict[str, float | int]:
    if not results:
        raise ValueError("results are required")
    return {
        "cases": len(results),
        "accuracy": ratio(sum(item.passed for item in results), len(results)),
        "citation_coverage": ratio(sum(item.citations_valid for item in results), sum(item.citations_required for item in results)),
        "tool_success": ratio(sum(item.tools_succeeded for item in results), sum(item.tools_attempted for item in results)),
    }

python -m pytest -q tests/test_stage.py

Run the real tests

..                                                                       [100%]
2 passed in <duration>s

Why this implementation

accuracy is computed as the number of passed cases divided by total cases: 1/2 = 0.5.

citation_coverage is computed as total valid citations divided by total required citations: (1+1)/(2+1) = 2/3 ≈ 0.6667.

tool_success is computed as total successful tool calls divided by total attempted tool calls: (1+0)/(1+1) = 0.5.

Guided practice

Fix the ratio() function to return 1.0 when the denominator is zero, making test_not_applicable_metric_is_not_failure pass.

Open forge/reliability/metrics.py and locate the ratio() function.
The current implementation is return numerator / denominator if denominator else 0.0; change it to return 1.0 when denominator == 0.
Run the test python -m pytest -q tests/test_stage.py::test_not_applicable_metric_is_not_failure to verify the fix.

Progressive hints

Hint 1: Use a conditional expression or if statement to check if the denominator is zero.
Hint 2: Return 1.0 instead of 0.0 because zero applicable cases should be considered success.
Hint 3: Ensure summarize() uses the corrected ratio().

Success criteria

ratio(0, 0) returns 1.0.
test_not_applicable_metric_is_not_failure passes.

Fault injection and diagnosis

In the fault lab, we provide a defective version of forge/reliability/metrics.py where the ratio() function returns 0.0 when the denominator is zero. This causes test_not_applicable_metric_is_not_failure to fail because a case with no citation requirements is incorrectly reported as having citation coverage 0.

In this stage you have understood vacuous truth semantics and know how to modify ratio() to handle zero denominators. By diagnosing this fault, you will reinforce this concept and learn how to locate and fix similar metric calculation errors.

Verified fault contract

Target test: test_not_applicable_metric_is_not_failure
Faulty file: forge/reliability/metrics.py

Diagnosis order

Run the faulty command python -m pytest -q ../tests/test_stage.py::test_not_applicable_metric_is_not_failure and observe the assertion failure: assert 0.0 == 1.0.
Inspect the ratio() function in forge/reliability/metrics.py and notice it returns 0.0 when the denominator is zero.
Understand the test expectation: for a case with no citation requirements, citation coverage should be 1.0 because the metric is not applicable and should not penalize the system.
Fix the ratio() function to return 1.0 when the denominator is zero.

Fix principle: Change the ratio() function from return numerator / denominator if denominator else 0.0 to return 1.0 if denominator == 0 else numerator / denominator. This returns 1.0 when the denominator is zero, indicating vacuous success.

python -m pytest -q ../tests/test_stage.py::test_not_applicable_metric_is_not_failure

Transfer challenge

Now use music scenario data to test your understanding. Create three CaseResult instances, one with no citation requirements (i.e., citations_required=0), and the other two with citation requirements where the number of valid citations equals the required number.

Changed condition: Change the case data from the software delivery scenario to the music creation scenario, and adjust citation requirements to include one not-applicable case.

In a Python script, create three CaseResult instances: the first case passes with no citation requirements (CaseResult(True)), the second case passes with 2 required citations and 2 valid citations, and the third case passes with 1 required citation and 1 valid citation. Then call summarize() and print the result.

Success criteria

The printed dictionary should have citation_coverage equal to 1.0 because all applicable cases have valid citations and the not-applicable case does not affect coverage.
cases should be 3 and accuracy should be 1.0 (all cases pass).

Run the real tests

python -m pytest -q tests/test_stage.py

Run the real tests

..                                                                       [100%]
2 passed in <duration>s

Design explanation question: Why should a metric with zero denominator be considered successful? Explain the rationale for vacuous truth semantics in quality metrics and how it differs from returning 0.0.

A strong answer should include

A zero denominator means there are no applicable cases, so the metric has no opportunity to fail and the system should not be penalized.
Returning 1.0 indicates full satisfaction of the requirement, while returning 0.0 would incorrectly imply poor performance on that metric.
Vacuous truth semantics is common in logic and mathematics, for example a universal statement over an empty set is true.

Mastery evidence

Passes both test_metrics_use_case_level_counts and test_not_applicable_metric_is_not_failure.
Can explain why a metric with zero denominator should return 1.0 (vacuous success) rather than 0.0.
Correctly implements ratio() and summarize() in forge/reliability/metrics.py while preserving the case count.

STAGE 03

Record parent/child spans while keeping one trace ID consistent across a run

Start with a concrete problem

The forge/reliability/ directory currently contains only metrics.py; there is no tracing.py, so the test file tests/test_stage.py fails immediately when importing Span and validate_trace.

You need to create forge/reliability/tracing.py from scratch, defining the Span dataclass and the validate_trace function so that both tests pass.

test_valid_parent_child_trace requires accepting two spans that share trace_id="t1" and have a correct parent-child relationship; test_cross_trace_child_is_rejected requires rejecting a child span from a different trace.

If you only check span ID uniqueness and ignore trace ID consistency, a cross-trace child span will be incorrectly accepted, polluting the trace data.

Make a prediction

When implementing validate_trace, why is it necessary to check that all spans have the same trace_id in addition to checking span ID uniqueness?

Because the trace ID is the unique identifier of a trace; inconsistency prevents correlating all spans of the same run.
Because the trace ID must be the same as the span ID.
Because the trace ID is only used for sorting and does not affect trace correctness.
Because the trace ID can be set arbitrarily and does not need to be checked.

Reasoning guide: The correct option is the first one. The trace ID identifies a complete run, and all spans belonging to that run must share the same trace ID; otherwise, the call chain cannot be reconstructed. The other options misunderstand the role of the trace ID.

What it is

A trace is a tree of spans

One run corresponds to one trace, identified by a unique trace_id; each operation within the trace is recorded as a span, and spans form parent-child relationships via parent_id.

validate_trace checks the structural integrity of the trace: span IDs must be unique, all spans must have the same trace_id, every parent_id must point to an existing span, and durations cannot be negative.

What it is not

Trace validation does not involve metrics or budgets

validate_trace only checks structure and ID consistency; it does not compute any metrics (such as total duration or average latency) and does not check budgets or resource consumption.

It does not generate spans or send them to any storage or visualization system; it only statically validates an already generated set of spans.

How it relates to neighboring concepts

Relationship between spans and trace

A trace contains multiple spans, each recording an operation (e.g., run, tool.call) and its duration.

The root span has parent_id set to None; other spans’ parent_id must point to a span ID within the same trace.

validate_trace enforces that all spans belong to the same trace by checking that the set of trace IDs has size 1, and it ensures valid parent-child relationships by checking that each parent_id exists in the set of span IDs.

Boundary decision for this stage: When a cross-trace child span is found, validate_trace must raise a ValueError, because accepting it would make trace data impossible to correlate correctly and lead to errors in later analysis.

Experiment: evaluation-observability-deployment-03-correlated-tracing
Input: Software delivery: add a feature to a frontend repository, modify code, and add tests.
Evidence: evidence/evaluation-observability-deployment/03-correlated-tracing/
Gate: Pending acceptance

Worked example

Goal: Demonstrate creating two spans and calling validate_trace() to pass the valid parent-child trace test.

First, define the Span dataclass with five fields: trace_id, span_id, parent_id, name, and duration_ms.

Then implement the basic checks in validate_trace: raise an error for an empty trace and check span ID uniqueness.

Finally, create two spans: a root span Span("t1", "root", None, "run", 20) and a child span Span("t1", "tool", "root", "tool.call", 8), and call validate_trace; it should not raise an exception.

Implementation steps

Create the file forge/reliability/tracing.py.
Define the Span dataclass using @dataclass(frozen=True) to make it immutable.
Implement the validate_trace function, first checking for an empty trace, then checking span ID uniqueness.
In the test, construct two spans sharing trace_id="t1" and call validate_trace to verify it passes.

Verified solution code: forge/reliability/tracing.py

from dataclasses import dataclass

@dataclass(frozen=True)
class Span:
    trace_id: str
    span_id: str
    parent_id: str | None
    name: str
    duration_ms: int

def validate_trace(spans: tuple[Span, ...]) -> None:
    if not spans:
        raise ValueError("trace is empty")
    if len({span.trace_id for span in spans}) != 1:
        raise ValueError("trace IDs must match")
    ids = {span.span_id for span in spans}
    if len(ids) != len(spans):
        raise ValueError("span IDs must be unique")
    for span in spans:
        if span.parent_id is not None and span.parent_id not in ids:
            raise ValueError("span parent is missing")
        if span.duration_ms < 0:
            raise ValueError("duration cannot be negative")

python -m pytest -q tests/test_stage.py

Run the real tests

..                                                                       [100%]
2 passed in <duration>s

Why this implementation

The Span dataclass uses frozen=True to ensure spans are immutable after creation, preventing accidental modification of trace data.

validate_trace first checks span ID uniqueness to prevent duplicate IDs from causing parent-child confusion.

In a valid parent-child trace, both spans have the same trace_id, and the child span’s parent_id points to the root span’s span_id, so validation passes.

Guided practice

Add a trace ID consistency check in validate_trace() to make test_cross_trace_child_is_rejected pass.

In validate_trace, before or after the span ID uniqueness check, add code to verify that all spans have the same trace_id.
Use a set comprehension {span.trace_id for span in spans} to get all trace IDs; if the set size is not 1, raise ValueError("trace IDs must match").
Run the test python -m pytest -q tests/test_stage.py::test_cross_trace_child_is_rejected and confirm it passes.

Progressive hints

Hint: A set size other than 1 means there are multiple different trace IDs.
Hint: The error message must contain “trace IDs” because the test asserts that.
Hint: The order of checks does not affect correctness, but it is recommended to check trace ID consistency before span ID uniqueness.

Success criteria

validate_trace raises a ValueError when encountering spans with different trace IDs.
test_cross_trace_child_is_rejected passes.

Fault injection and diagnosis

The failure fixture fixtures/failure.json injects a defect: validate_trace only checks span ID uniqueness and omits trace ID consistency. When the test test_cross_trace_child_is_rejected runs, it passes two spans: Span("t1","root",None,"run",20) and Span("t2","tool","root","tool.call",8), which have different trace IDs, but the function does not raise an exception, causing the test to fail.

This stage focuses on understanding and implementing trace ID consistency checks. By analyzing the code in the failure fixture, learners can see the consequence of omitting this check and can add the correct validation logic based on the test requirements.

Verified fault contract

Target test: test_cross_trace_child_is_rejected
Faulty file: forge/reliability/tracing.py

Diagnosis order

Run the faulty command python -m pytest -q ../tests/test_stage.py::test_cross_trace_child_is_rejected and observe the test failure: AssertionError: cross-trace span was accepted.
Inspect the implementation of validate_trace in forge/reliability/tracing.py and notice that it only checks span ID uniqueness, with no trace ID consistency check.
Understand the test’s intent: when spans have different trace IDs, a ValueError should be raised, and the error message must contain “trace IDs”.
Add the check to validate_trace: if len({span.trace_id for span in spans}) != 1: raise ValueError("trace IDs must match").

Fix principle: In the validate_trace function, add a trace ID consistency check before the span ID uniqueness check: use a set comprehension to collect all trace IDs, and if the set size is not 1, raise ValueError("trace IDs must match"). This way, when spans with different trace IDs are passed, the function correctly rejects them.

python -m pytest -q ../tests/test_stage.py::test_cross_trace_child_is_rejected

Transfer challenge

Office research scenario: You are working on a document analysis task and need to record three spans: a root span representing the entire run, a tool span for document parsing, and an LLM span for summary generation. All spans share the same trace_id ‘office-trace-1’ and form a correct parent-child relationship.

Changed condition: Extend from a two-span parent-child trace to a three-span office scenario, introducing new span names (doc.parse, llm.summarize) and different durations while maintaining trace_id consistency and correct parent-child relationships.

In a Python interactive environment or a separate script (e.g., office_trace.py), create three Span objects: root span Span("office-trace-1", "root", None, "run", 30), tool span Span("office-trace-1", "tool", "root", "doc.parse", 12), and LLM span Span("office-trace-1", "llm", "tool", "llm.summarize", 18). Then call validate_trace to verify that these three spans form a valid trace.

Success criteria

Running python -c "from forge.reliability.tracing import Span, validate_trace; validate_trace((Span('office-trace-1','root',None,'run',30), Span('office-trace-1','tool','root','doc.parse',12), Span('office-trace-1','llm','tool','llm.summarize',18)))" does not raise an exception.
After running, no exception is raised, and all three spans have trace_id ‘office-trace-1’, with parent-child relationships root->tool->llm.

Run the real tests

python -m pytest -q tests/test_stage.py

Run the real tests

..                                                                       [100%]
2 passed in <duration>s

Design explanation question: Why must validate_trace check that all spans have the same trace_id? What specific consequences would omitting this check have for the tracing system?

A strong answer should include

The trace ID identifies a complete run; all spans belonging to that run must share the same trace ID, otherwise the scattered spans cannot be correlated.
If the check is omitted, spans from different runs could be incorrectly merged into the same trace, causing confusion in the call chain and leading to wrong conclusions in later analysis (e.g., latency statistics, error localization).
In distributed systems, mixing spans from different traces breaks trace integrity, making it impossible to accurately reconstruct the request path.

Mastery evidence

Passing both test_valid_parent_child_trace and test_cross_trace_child_is_rejected proves that validate_trace correctly handles valid traces and cross-trace child spans.
Being able to explain the necessity of trace ID consistency: all spans must share the same trace ID, otherwise trace data cannot be correlated.
In the office scenario transfer challenge, successfully creating three spans and validating them demonstrates the ability to apply trace validation in a different context.

STAGE 04

Aggregate run cost and latency and produce an actionable release gate for budget violations

Start with a concrete problem

How can we determine if a run is within cost and latency budgets?

Before release, we need a clear budget-checking function that takes actual cost, actual latency, and a budget object, and returns the names of dimensions that exceed the budget.

The codebase currently lacks the forge/reliability/budgets.py file, so tests fail at import time.

The test test_over_budget_dimensions_are_named expects that when cost 1.5 exceeds budget 1.0, the function returns ('cost',), while test_exact_budget_is_allowed expects an empty tuple when cost 1.0 and latency 1000 exactly equal the budget.

You need to create the file, define the Budget dataclass and check_budget function, and ensure the comparison logic is correct.

Make a prediction

If the actual cost exactly equals the budget limit, what should check_budget return?

A tuple containing ‘cost’, because reaching the limit means exceeding the budget
An empty tuple, because being equal to the limit is still within the allowed range
A tuple containing ‘latency’, because latency might also equal the limit
Raise an exception, because equality is a boundary case

Reasoning guide: The budget limit is the maximum allowed value; being equal to it means not exceeding, so it should not be considered a violation. Returning an empty tuple indicates all dimensions are within budget.

What it is

What budget checking is

Budget checking is a pure function that compares actual measurements against budget limits and returns the names of all dimensions that exceed the budget.

It only performs numeric comparison and does not involve release gates or rollback decisions.

A violation occurs only when the actual value is strictly greater than the limit; equality is allowed.

What it is not

What budget checking is not

It is not a release gate system; it does not automatically block deployment or trigger rollback.

It does not handle negative inputs; negative measurements should be considered invalid and raise an exception.

It does not modify the budget object or measurements; it only returns the violating dimensions.

How it relates to neighboring concepts

Relationships with other components

The Budget dataclass defines the budget limits, and check_budget uses these limits for comparison.

The test file tests/test_stage.py imports Budget and check_budget to verify their behavior.

This module is independent of the tracing module but may be called by the release process in the future.

Boundary decision for this stage: Budget checking only compares numbers, not release gates or rollback; negative inputs should raise an exception, and equality to the limit is allowed.

Experiment: evaluation-observability-deployment-04-cost-latency-budgets
Input: Software delivery: add a feature to a frontend repository, modify code, and add tests.
Evidence: evidence/evaluation-observability-deployment/04-cost-latency-budgets/
Gate: Pending acceptance

Worked example

Goal: Demonstrate how check_budget() identifies over-budget dimensions.

Suppose we have a budget Budget(max_cost=1.0, max_latency_ms=1000), actual cost 1.5, and latency 900.

Cost 1.5 is greater than limit 1.0, so it should return ('cost',); latency 900 is less than 1000, so no violation.

We implement the check_budget function using strict greater-than comparisons and return a tuple of violating dimensions.

Implementation steps

Create the forge/reliability/budgets.py file.
Define the Budget dataclass with max_cost and max_latency_ms fields.
Implement check_budget to check if cost is greater than budget.max_cost; if so, add ‘cost’ to the violations list.
Check if latency is greater than budget.max_latency_ms; if so, add ‘latency’.
Return the violations tuple.

Verified solution code: forge/reliability/budgets.py

from dataclasses import dataclass

@dataclass(frozen=True)
class Budget:
    max_cost: float
    max_latency_ms: int

def check_budget(cost: float, latency_ms: int, budget: Budget) -> tuple[str, ...]:
    violations: list[str] = []
    if cost < 0 or latency_ms < 0:
        raise ValueError("measurements cannot be negative")
    if cost > budget.max_cost:
        violations.append("cost")
    if latency_ms > budget.max_latency_ms:
        violations.append("latency")
    return tuple(violations)

python -m pytest -q tests/test_stage.py

Run the real tests

..                                                                       [100%]
2 passed in <duration>s

Why this implementation

When cost 1.5 is greater than budget 1.0, the violations list contains ‘cost’, returning ('cost',).

Latency 900 is less than 1000, so ‘latency’ is not added, and the result is correct.

Guided practice

Fix comparison operators to return an empty tuple when equal to budget, making test_exact_budget_is_allowed pass.

Open forge/reliability/budgets.py.
Inspect the comparison operators in the check_budget function.
Change >= to > to ensure equality to the limit is not a violation.
Run the test python -m pytest -q tests/test_stage.py::test_exact_budget_is_allowed.

Progressive hints

Note that the current code may use >=, which treats equality as a violation.
The budget limit is the maximum allowed value; equality should be allowed.
After the change, when cost equals budget.max_cost, ‘cost’ should not be added.

Success criteria

test_exact_budget_is_allowed passes.
check_budget(1.0, 1000, Budget(1.0, 1000)) returns an empty tuple.

Fault injection and diagnosis

The file forge/reliability/budgets.py contains a faulty implementation of check_budget that uses >= comparisons. When you run the test test_exact_budget_is_allowed, it fails because cost 1.0 and latency 1000 are incorrectly flagged as violations. Your task is to diagnose why the test fails and fix the defect.

This stage has taught the boundary condition of budget checking: equality to the limit should be allowed. By inspecting the comparison operators in forge/reliability/budgets.py, you can locate and fix this defect.

Verified fault contract

Target test: test_exact_budget_is_allowed
Faulty file: forge/reliability/budgets.py

Diagnosis order

Run the test python -m pytest -q ../tests/test_stage.py::test_exact_budget_is_allowed and observe the failure.
Examine the check_budget function in forge/reliability/budgets.py.
Identify that the comparison operators use >=, which treats equality as a violation.
Conclude that the fix is to change >= to >.

Fix principle: Change cost >= budget.max_cost to cost > budget.max_cost and latency_ms >= budget.max_latency_ms to latency_ms > budget.max_latency_ms in the check_budget function.

python -m pytest -q ../tests/test_stage.py::test_exact_budget_is_allowed

Transfer challenge

In an office research scenario, a run has cost 2.0 dollars and latency 1500 milliseconds, with budget limits 2.0 and 1500 respectively.

Changed condition: Change the budget limits from 1.0 and 1000 to 2.0 and 1500, while preserving the boundary condition of equality to the limit.

Define Budget(max_cost=2.0, max_latency_ms=1500) for the office scenario, call check_budget(2.0, 1500, budget), and verify it returns an empty tuple.

Success criteria

check_budget(2.0, 1500, Budget(2.0, 1500)) returns an empty tuple.
check_budget(2.5, 1600, Budget(2.0, 1500)) returns (‘cost’, ‘latency’).

Run the real tests

python -m pytest -q tests/test_stage.py

Run the real tests

..                                                                       [100%]
2 passed in <duration>s

Design explanation question: Why should being equal to the budget be allowed? What should check_budget do if cost or latency is negative?

A strong answer should include

Being equal to the budget means not exceeding the maximum allowed value, so it should not be considered a violation.
Negative measurements are invalid and should raise a ValueError exception.

Mastery evidence

Passes both test_over_budget_dimensions_are_named and test_exact_budget_is_allowed.
Can explain why being equal to the budget limit is allowed rather than a violation.
Implemented the Budget dataclass and check_budget function in forge/reliability/budgets.py using strict greater-than comparisons.

STAGE 05

Bind evaluation, migration, and security checks to an immutable release manifest before promotion

Start with a concrete problem

A release candidate missing the security gate is promoted

The file forge/reliability/deployment.py does not exist yet, so both tests in tests/test_stage.py fail at import with ModuleNotFoundError: No module named 'forge.reliability.deployment'.

Even if we create the file and implement a simple promote function that only checks for a version and returns a promotion string, test_missing_security_gate_blocks_promotion will still fail because a manifest lacking the security gate is incorrectly labeled as promoted.

You need to implement an immutable ReleaseManifest dataclass and a promote function that validates the release identity (non-empty version and artifact SHA256 length 64) and checks that all required gates (tests, evaluation, security, migration) are present, otherwise raising a ValueError containing the missing gate names.

Make a prediction

When implementing the promote function, what happens if you only check that manifest.version is non-empty and then return the promotion string?

Both tests pass because having a version is sufficient for promotion.
test_complete_release_is_promoted passes, but test_missing_security_gate_blocks_promotion fails because a manifest missing the security gate is incorrectly promoted.
Both tests fail because an exception should be raised when the security gate is missing.
test_missing_security_gate_blocks_promotion passes, but test_complete_release_is_promoted fails.

Reasoning guide: The correct answer is the second option. Checking only for a version does not prevent a manifest missing the security gate from being promoted, so the second test fails. You need to compare passed_gates with the required gate set and raise a ValueError when gates are missing.

What it is

What release promotion gates are

Release promotion gates are a set comparison between an immutable release manifest (ReleaseManifest) and a required gate set (REQUIRED_GATES).

The promote function first validates the release identity (non-empty version and artifact SHA256 length 64), then computes missing gates = required gates - passed gates, and if the missing set is non-empty it raises a ValueError containing the missing gate names; otherwise it returns a promotion string starting with promoted:.

What it is not

What release promotion gates are not

Release promotion gates do not perform actual deployment, rollback, or any infrastructure operations; they only check whether the gate set is complete.

They are also not a comprehensive evaluation of release quality; they only verify that all required gates have passed, without caring about specific metrics or scores within each gate.

How it relates to neighboring concepts

Relationship to other concepts

Release promotion gates build on the reliability checking ideas from the Stage 04 budgets module, but focus on pre-release gate set validation.

They relate to observability because promotion decisions should be based on observable run records and test results, not on the model’s subjective judgment.

They complement the Harness’s isolation and permission controls: the Harness handles runtime security, while release promotion gates handle static pre-release gate checks.

Boundary decision for this stage: Release promotion only checks the gate set, not actual deployment or rollback; if any required gate is missing, it must raise an exception to block promotion.

Experiment: evaluation-observability-deployment-05-release-promotion-gates
Input: Software delivery: add a feature to a frontend repository, modify code, and add tests.
Evidence: evidence/evaluation-observability-deployment/05-release-promotion-gates/
Gate: Pending acceptance

Worked example

Goal: Demonstrate creating a ReleaseManifest with all required gates and calling promote() successfully.

We first define the REQUIRED_GATES constant containing the four gates: tests, evaluation, security, migration.

Then we create the ReleaseManifest dataclass with frozen=True to make it immutable, containing version, artifact_sha256, and passed_gates fields.

In the promote function, we first validate that the version is non-empty and the artifact SHA256 length is 64, otherwise raise a ValueError.

Then we compute missing gates; if the missing set is non-empty, raise a ValueError containing the missing gate names; otherwise return the promotion string.

Implementation steps

Create the file forge/reliability/deployment.py.
Define REQUIRED_GATES = frozenset({"tests", "evaluation", "security", "migration"}).
Define the ReleaseManifest dataclass with the @dataclass(frozen=True) decorator.
Implement the promote function with identity validation and gate check logic.
Run the test command to verify both tests pass.

Verified solution code: forge/reliability/deployment.py

from dataclasses import dataclass

REQUIRED_GATES = frozenset({"tests", "evaluation", "security", "migration"})

@dataclass(frozen=True)
class ReleaseManifest:
    version: str
    artifact_sha256: str
    passed_gates: frozenset[str]

def promote(manifest: ReleaseManifest) -> str:
    if not manifest.version or len(manifest.artifact_sha256) != 64:
        raise ValueError("release identity is invalid")
    missing = REQUIRED_GATES - manifest.passed_gates
    if missing:
        raise ValueError("release gates missing: " + ", ".join(sorted(missing)))
    return f"promoted:{manifest.version}:{manifest.artifact_sha256[:12]}"

python -m pytest -q tests/test_stage.py

Run the real tests

..                                                                       [100%]
2 passed in <duration>s

Why this implementation

REQUIRED_GATES defines all gates that must pass for release promotion, using frozenset to ensure immutability.

ReleaseManifest uses frozen=True to make it immutable, preventing accidental modification during promotion.

promote first validates the release identity, then checks gate completeness, ensuring only releases meeting all conditions are promoted.

Guided practice

Add missing gate check in promote() to make test_missing_security_gate_blocks_promotion pass.

Open the file forge/reliability/deployment.py.
Define the REQUIRED_GATES constant containing all required gates.
In the promote function, compute missing = REQUIRED_GATES - manifest.passed_gates.
If missing is non-empty, raise a ValueError with a message containing the missing gate names.
Run the test command to verify both tests pass.

Progressive hints

Use set difference to compute missing gates.
The error message should contain the missing gate names, e.g., security.
Ensure the promote function does not return a promotion string when gates are missing.

Success criteria

test_complete_release_is_promoted passes.
test_missing_security_gate_blocks_promotion passes.

Fault injection and diagnosis

The failure fixture fixtures/failure.json describes a release promotion bypass vulnerability: the promote function only checks for a version and returns a promotion string, causing a manifest missing the security gate to be incorrectly promoted. Running the fault test test_missing_security_gate_blocks_promotion fails because the test expects a ValueError containing security, but no exception is raised.

The core mechanism of this stage is gate set comparison, so it can diagnose and fix this fault. By checking whether the promote function compares passed_gates with the required gate set, you can locate the defect and add the missing gate check.

Verified fault contract

Target test: test_missing_security_gate_blocks_promotion
Faulty file: forge/reliability/deployment.py

Diagnosis order

Run the fault test command python -m pytest -q ../tests/test_stage.py::test_missing_security_gate_blocks_promotion and observe the failure output.
Inspect the promote function in forge/reliability/deployment.py and notice it only checks manifest.version is non-empty before returning the promotion string.
Confirm that there is no comparison between passed_gates and the required gate set, allowing a manifest missing the security gate to be promoted.
Fix: add the REQUIRED_GATES constant and in promote compute missing gates; if missing, raise a ValueError.

Fix principle: In forge/reliability/deployment.py, define REQUIRED_GATES = frozenset({"tests", "evaluation", "security", "migration"}) and add a missing gate check in promote: missing = REQUIRED_GATES - manifest.passed_gates; if missing is non-empty, raise ValueError("release gates missing: " + ", ".join(sorted(missing))).

python -m pytest -q ../tests/test_stage.py::test_missing_security_gate_blocks_promotion

Transfer challenge

Music creation scenario: you are preparing a music release and need to ensure all gates pass before promotion.

Changed condition: Use music scenario data to create ReleaseManifest(version='2.0.0', artifact_sha256='b'*64, passed_gates={'tests','evaluation','security','migration'}), call promote(), and verify the returned string starts with promoted:2.0.0.

In forge/reliability/deployment.py, create a ReleaseManifest instance with the given music scenario data, call the promote() function, and verify that the returned string starts with promoted:2.0.0.

Success criteria

The created ReleaseManifest instance has version 2.0.0, artifact SHA256 as 64 b characters, and passed gates containing tests, evaluation, security, migration.
Calling promote() returns a string starting with promoted:2.0.0.

Run the real tests

python -m pytest -q tests/test_stage.py

Run the real tests

..                                                                       [100%]
2 passed in <duration>s

Design explanation question: Why is the security gate required? If multiple gates are missing, what should the error message include?

A strong answer should include

The security gate ensures the release has undergone security review, preventing code that has not passed security checks from entering production.
If multiple gates are missing, the error message should include all missing gate names so developers can identify and fix all issues at once.

Mastery evidence

Passes both test_complete_release_is_promoted and test_missing_security_gate_blocks_promotion.
Can explain the role of the required gate set REQUIRED_GATES and why the security gate is required.
Can correctly implement the missing gate check in promote, raising a ValueError containing the missing gate names.

STAGE 06

Compare baseline and canary error/latency metrics and recommend rollback under explicit thresholds

Start with a concrete problem

When should an automatic rollback be recommended for a canary version?

When deploying a new version, we typically route a small portion of traffic to a canary instance and observe its health metrics.

If the canary’s error rate or latency is significantly higher than the baseline, the system should automatically recommend a rollback rather than waiting for manual judgment.

However, statistical fluctuations in small samples are large, and triggering a rollback based on only a few requests would cause false alarms and waste deployment resources.

Therefore, we need to implement a function that, given a minimum sample size, compares baseline and canary error rates and latency, and provides a rollback recommendation.

Make a prediction

When implementing the rollback decision function, which approach is most reasonable?

Immediately roll back if the canary error rate is higher than baseline, without considering sample size.
First check if the canary request count meets the minimum sample size; if not, return insufficient sample, otherwise compare error rate and latency.
Only compare latency, ignoring error rate, because latency better reflects user experience.
Let the model decide whether to roll back regardless of metrics.

Reasoning guide: The correct approach is to first check the minimum sample size to avoid false rollbacks due to small-sample statistical fluctuations; then compare error rate and latency against thresholds.

What it is

What the rollback decision function is

The rollback decision function is a pure function that takes baseline health metrics, canary health metrics, and a minimum request count, and returns whether rollback is needed along with a tuple of reasons.

It uses explicit threshold rules: error rate exceeding twice the baseline and at least 2%, or latency exceeding 1.5 times the baseline, triggers a rollback recommendation.

The function does not perform the actual rollback; it only provides the decision basis for upper-level systems to take action.

What it is not

What the rollback decision function is not

It is not an operation that actually performs a rollback; it does not modify deployment state or send notifications.

It is not a statistical testing tool; it does not compute confidence intervals or perform hypothesis testing, only simple threshold comparisons.

It does not collect metric data; the input health metrics are provided by external systems.

How it relates to neighboring concepts

Relationship with other components

The function depends on the Health data class, which encapsulates request count, error count, and p95 latency.

It is directly called by two test cases in tests/test_stage.py to verify its behavior.

In the deployment pipeline, this function sits in the canary analysis phase and provides decisions for subsequent rollback operations.

Boundary decision for this stage: Rollback decisions are based only on health metrics, not actual rollback or incident recording; when the sample size is insufficient, return an insufficient-sample state to avoid misjudgment.

Experiment: evaluation-observability-deployment-06-canary-rollback-decision
Input: Software delivery: add a feature to a frontend repository, modify code, and add tests.
Evidence: evidence/evaluation-observability-deployment/06-canary-rollback-decision/
Gate: Pending acceptance

Worked example

Goal: Demonstrate how rollback_required() identifies a canary with significantly increased error rate.

Assume baseline health metrics are Health(requests=1000, errors=5, p95_ms=200), i.e., error rate 0.5%, latency 200ms.

Canary health metrics are Health(requests=200, errors=20, p95_ms=400), error rate 10%, latency 400ms.

Canary request count 200 is greater than the minimum sample size 100, so it enters metric comparison.

Error rate 10% is greater than twice the baseline error rate (1%) and greater than 2%, so ‘error-rate’ reason is added; latency 400ms is greater than 1.5 times baseline latency (300ms), so ‘latency’ reason is added.

Finally returns (True, (‘error-rate’, ‘latency’)), recommending rollback.

Implementation steps

Create baseline Health object: Health(1000, 5, 200).
Create canary Health object: Health(200, 20, 400).
Call rollback_required(baseline, canary).
Check that returned needed is True and reasons contains ‘error-rate’ and ‘latency’.

Verified solution code: forge/reliability/rollback.py

from dataclasses import dataclass

@dataclass(frozen=True)
class Health:
    requests: int
    errors: int
    p95_ms: int

    @property
    def error_rate(self) -> float:
        return self.errors / self.requests if self.requests else 0.0

def rollback_required(baseline: Health, canary: Health, min_requests: int = 100) -> tuple[bool, tuple[str, ...]]:
    if canary.requests < min_requests:
        return False, ("insufficient-sample",)
    reasons=[]
    if canary.error_rate > max(0.02, baseline.error_rate * 2): reasons.append("error-rate")
    if canary.p95_ms > baseline.p95_ms * 1.5: reasons.append("latency")
    return bool(reasons), tuple(reasons)

python -m pytest -q tests/test_stage.py

Run the real tests

..                                                                       [100%]
2 passed in <duration>s

Why this implementation

This example shows that when both canary error rate and latency deteriorate significantly, the function returns True and gives two reasons.

Threshold rules: error rate exceeds twice the baseline and at least 2%, latency exceeds 1.5 times the baseline.

Note: In the example, the canary request count 200 meets the minimum sample size 100, so the insufficient sample check is not triggered.

Guided practice

Add minimum sample check in rollback_required() to make test_small_sample_does_not_trigger_automatic_rollback pass.

Open forge/reliability/rollback.py file.
At the beginning of the rollback_required function, add: if canary.requests < min_requests: return False, (“insufficient-sample”,).
Ensure the subsequent metric comparison logic remains unchanged.
Run the test: python -m pytest -q tests/test_stage.py::test_small_sample_does_not_trigger_automatic_rollback.

Progressive hints

Hint 1: Check if canary.requests is less than min_requests.
Hint 2: The returned reasons tuple should contain ‘insufficient-sample’.
Hint 3: Don’t forget to return needed as False.

Success criteria

test_small_sample_does_not_trigger_automatic_rollback passes.
test_bad_canary_requests_rollback still passes.

Fault injection and diagnosis

In the fault fixture, rollback.py lacks the minimum sample size check, causing a canary sample with only 3 requests to trigger an automatic rollback. The test test_small_sample_does_not_trigger_automatic_rollback fails because the function returns True instead of False. The executable fault fixture modifies forge/reliability/rollback.py and is reliably reproduced by test_small_sample_does_not_trigger_automatic_rollback.

The core of this stage is to implement a rollback decision function with a minimum sample size check. By comparing the faulty code with the correct implementation, learners can understand the importance of the sample size check and fix the defect.

Verified fault contract

Target test: test_small_sample_does_not_trigger_automatic_rollback
Faulty file: forge/reliability/rollback.py

Diagnosis order

Run the faulty test: python -m pytest -q ../tests/test_stage.py::test_small_sample_does_not_trigger_automatic_rollback, observe the assertion failure.
Inspect the rollback_required function and notice that it does not check if canary.requests is less than min_requests.
Compare with the correct implementation, add the sample size check at the beginning of the function and return insufficient-sample.

Fix principle: Add at the beginning of rollback_required: if canary.requests < min_requests: return False, (“insufficient-sample”,). This way, when the sample size is insufficient, the function returns False and reason ‘insufficient-sample’, avoiding false rollbacks.

python -m pytest -q ../tests/test_stage.py::test_small_sample_does_not_trigger_automatic_rollback

Transfer challenge

In the office research scenario, you need to evaluate the performance of a canary version. The baseline health metrics are Health(requests=500, errors=10, p95_ms=200), and the canary health metrics are Health(requests=400, errors=40, p95_ms=250).

Changed condition: Use a different baseline/canary dataset (office research scenario), but keep the threshold logic of rollback_required() unchanged.

Create baseline and canary Health objects using the given office scenario data, call rollback_required(), and verify the return value is (True, (‘error-rate’,)).

Success criteria

Calling rollback_required(Health(500,10,200), Health(400,40,250)) returns needed=True.
The returned reasons tuple is (‘error-rate’,), not containing ‘latency’, because latency 250ms does not exceed 1.5 times baseline latency (300ms).

Run the real tests

python -m pytest -q tests/test_stage.py

Run the real tests

..                                                                       [100%]
2 passed in <duration>s

Design explanation question: Why shouldn’t small samples trigger automatic rollback? Explain the role of the sample size check in rollback decisions and its impact on system stability.

A strong answer should include

In small samples, the estimates of error rate and latency have high variance, making it easy to misjudge due to random fluctuations.
Setting a minimum sample size ensures that metrics are statistically meaningful, reducing false rollbacks and improving deployment stability.

Mastery evidence

Pass both test_bad_canary_requests_rollback and test_small_sample_does_not_trigger_automatic_rollback.
Can explain why small samples should not trigger automatic rollback and how the sample size check prevents false positives.
In the office scenario transfer challenge, correctly use rollback_required() and verify return (True, (‘error-rate’,)).

STAGE 07

Record detection, mitigation, recovery, and review timestamps while validating recovery objectives and evidence

Start with a concrete problem

Why is an ordered timeline a prerequisite for trustworthy recovery metrics?

In an incident drill, the four timestamps for detection, mitigation, recovery, and review must be recorded in chronological order; otherwise, the computed recovery time is meaningless.

The file forge/reliability/incident_drill.py does not exist yet, so the tests fail immediately when importing DrillRecord and verify_drill; therefore, the module must be created from scratch.

If we only compute recovered_at - detected_at without checking timestamp order, an impossible sequence like detected_at=100, mitigated_at=80 would be accepted, leading to a false recovery objective judgment.

This stage requires implementing verify_drill(), which must validate that the timeline is monotonically increasing and evidence is non-empty before computing recovery time and judging the objective.

Make a prediction

When implementing verify_drill(), what should the function do if the timestamps are out of order?

Still compute recovery time but return objectiveMet=False
Raise a ValueError and reject the record
Automatically reorder the timestamps and then compute
Ignore the order issue and only check if evidence is empty

Reasoning guide: The correct approach is to raise a ValueError, because timeline order is a prerequisite for the validity of recovery metrics and cannot be silently corrected or ignored.

What it is

What incident drill verification is

Incident drill verification is a pure function that takes a DrillRecord and a recovery objective in seconds, returning a dictionary with recovery seconds and whether the objective was met.

It first checks that the four timestamps are monotonically increasing, then checks that the evidence tuple is non-empty, and only then computes recovery time and compares it to the objective.

This verification depends only on input data and does not involve actual deployment or rollback operations, so it can be safely used for drill review.

What it is not

What incident drill verification is not

It is not an actual incident response system; it does not trigger alerts or execute rollbacks, only performs static checks on drill records.

It does not generate timestamps or evidence; those must be provided by external systems, and verification only judges their plausibility.

It does not guarantee that the recovery objective is met; it objectively computes and returns a boolean based on the input data.

How it relates to neighboring concepts

Relationship to other components

DrillRecord is the data carrier, using dataclass(frozen=True) to ensure immutability and prevent accidental modification during verification.

verify_drill() depends on the fields of DrillRecord but is independent of any storage or network layer, making it easy to unit test.

This module sits alongside other reliability components like rollback.py, collectively forming an incident handling toolkit, but this stage focuses only on verification logic.

Boundary decision for this stage: The verification function must reject invalid input before computing any metrics; otherwise, subsequent metrics would be built on untrustworthy data.

Experiment: evaluation-observability-deployment-07-incident-drill-evidence
Input: Software delivery: add a feature to a frontend repository, modify code, and add tests.
Evidence: evidence/evaluation-observability-deployment/07-incident-drill-evidence/
Gate: Pending acceptance

Worked example

Goal: Demonstrate how verify_drill() computes recovery time from an ordered timeline and judges the objective.

Suppose a drill record is DrillRecord(0, 30, 90, 120, ("trace.json",)) with a recovery objective of 120 seconds.

The timestamps 0→30→90→120 are monotonically increasing, and the evidence tuple contains "trace.json", so validation passes.

Recovery time is computed as 90 - 0 = 90 seconds, which is less than or equal to the objective of 120 seconds, so objectiveMet is True.

Implementation steps

Create the file forge/reliability/incident_drill.py.
Define the DrillRecord dataclass with five fields.
Implement verify_drill(): first check timestamp order and evidence, then compute recovery time.
Run the test test_ordered_drill_measures_recovery to verify the output.

Verified solution code: forge/reliability/incident_drill.py

from dataclasses import dataclass

@dataclass(frozen=True)
class DrillRecord:
    detected_at: int
    mitigated_at: int
    recovered_at: int
    reviewed_at: int
    evidence: tuple[str, ...]

def verify_drill(record: DrillRecord, recovery_objective_seconds: int) -> dict[str, int | bool]:
    times = (record.detected_at, record.mitigated_at, record.recovered_at, record.reviewed_at)
    if list(times) != sorted(times):
        raise ValueError("incident timestamps are out of order")
    if not record.evidence:
        raise ValueError("drill evidence is required")
    recovery_time = record.recovered_at - record.detected_at
    return {"recoverySeconds": recovery_time, "objectiveMet": recovery_time <= recovery_objective_seconds}

python -m pytest -q tests/test_stage.py

Run the real tests

..                                                                       [100%]
2 passed in <duration>s

Why this implementation

The timestamp check uses list(times) != sorted(times) to determine if the sequence is ordered; if not equal, it raises a ValueError.

The evidence check ensures record.evidence is non-empty, preventing drill records without evidence from being accepted.

Recovery time uses only detected_at and recovered_at, independent of mitigation and review times, but the latter two must exist and be ordered.

Guided practice

Add a time order check in verify_drill() to make test_out_of_order_timeline_is_rejected pass.

Open forge/reliability/incident_drill.py and inspect the current implementation.
Before computing recovery time, add a timestamp order check.
If the timestamps are out of order, raise a ValueError with a message containing "order".
Run the test test_out_of_order_timeline_is_rejected to confirm it passes.

Progressive hints

You can use sorted() to compare whether the timestamp tuple is ordered.
The error message must contain "order", otherwise the test assertion will fail.
Should the check be placed before or after the evidence check? Consider which error is more fundamental.

Success criteria

test_out_of_order_timeline_is_rejected passes.
test_ordered_drill_measures_recovery still passes.

Fault injection and diagnosis

The current verify_drill() in forge/reliability/incident_drill.py only computes recovery time without validating timestamp order. When running test_out_of_order_timeline_is_rejected, it passes DrillRecord(100, 80, 140, 160, ("trace.json",)), where mitigation time 80 is earlier than detection time 100, but the function still returns objectiveMet=True, causing the test to fail.

This stage has already taught the necessity of timeline validation, and the final implementation of verify_drill() includes an order check. By locating this fault, you can understand why out-of-order timelines must be rejected and how to fix it.

Verified fault contract

Target test: test_out_of_order_timeline_is_rejected
Faulty file: forge/reliability/incident_drill.py

Diagnosis order

Run python -m pytest -q ../tests/test_stage.py::test_out_of_order_timeline_is_rejected and observe the failure.
Inspect the source of verify_drill() and notice the missing timestamp order check.
Add an order check before computing recovery time and raise a ValueError.
Re-run the test to confirm it passes.

Fix principle: In verify_drill(), add a timestamp order check: times = (record.detected_at, record.mitigated_at, record.recovered_at, record.reviewed_at), then if list(times) != sorted(times): raise ValueError("incident timestamps are out of order").

python -m pytest -q ../tests/test_stage.py::test_out_of_order_timeline_is_rejected

Transfer challenge

In an office research scenario, an incident drill for a document processing failure has the following record: detection time 10 seconds, mitigation time 40 seconds, recovery time 100 seconds, review time 150 seconds, and evidence file office-trace.json.

Changed condition: Change the drill scenario from software delivery to office research, using different timestamps and evidence filename.

Create DrillRecord(detected_at=10, mitigated_at=40, recovered_at=100, reviewed_at=150, evidence=('office-trace.json',)), call verify_drill(record, 120), and verify that it returns {'recoverySeconds':90, 'objectiveMet':True}.

Success criteria

The dictionary returned by verify_drill has recoverySeconds equal to 90.
The returned dictionary has objectiveMet as True because 90 seconds is less than the objective of 120 seconds.

Run the real tests

python -m pytest -q tests/test_stage.py

Run the real tests

..                                                                       [100%]
2 passed in <duration>s

Design explanation question: Why must timestamps be monotonically increasing? What would be the consequence for recovery time calculation if out-of-order timestamps were allowed?

A strong answer should include

Monotonically increasing timestamps are a fundamental constraint of an incident timeline; otherwise, the order of events cannot be determined.
If out-of-order timestamps were allowed, recovery time could be negative or incorrect, leading to a distorted recovery objective judgment.
The verification function must reject invalid input to ensure the trustworthiness of subsequent metrics.

Mastery evidence

Passes both test_ordered_drill_measures_recovery and test_out_of_order_timeline_is_rejected.
Can explain why timestamp order checking is a prerequisite for computing recovery metrics.
Correctly creates a DrillRecord in the office scenario and verifies the recovery objective.

Complete the chapter

Create an observable release covering quality, latency, cost, failure, and rollback.

FORGE / LOCAL CHECK

Local lab self-check

Not started
2Reading
3Lab downloaded
4Test result read
5Local check passed

verification.json is parsed only in this browser and is never uploaded. This checks the lab contract; it is not server certification or third-party endorsement.

Choose verification.json

Primary layer

Depends on

Provides to

Governed by

Concept calibration

Verification & Evaluation

Observability

Harness

Evaluation, Observability, Deployment & Reliability

Start with a concrete problem

Evaluation cases need stable identifiers, otherwise duplicates break traceability

Make a prediction

What it is

An evaluation dataset is a versioned collection of unique cases

What it is not

An evaluation dataset does not execute evaluations or compute metrics

How it relates to neighboring concepts

Relationship between EvalCase and EvalDataset

Start with a concrete problem

How can we aggregate meaningful overall quality metrics from multiple case results?

Make a prediction

What it is

What quality metric aggregation is

What it is not

What quality metric aggregation is not

How it relates to neighboring concepts

Relationship to other concepts

Start with a concrete problem

Spans in one run must share the same trace ID

Make a prediction

What it is

A trace is a tree of spans

What it is not

Trace validation does not involve metrics or budgets

How it relates to neighboring concepts

Relationship between spans and trace

Start with a concrete problem

How can we determine if a run is within cost and latency budgets?

Make a prediction

What it is

What budget checking is

What it is not

What budget checking is not

How it relates to neighboring concepts

Relationships with other components

Start with a concrete problem

A release candidate missing the security gate is promoted

Make a prediction

What it is

What release promotion gates are

What it is not

What release promotion gates are not

How it relates to neighboring concepts

Relationship to other concepts

Start with a concrete problem

When should an automatic rollback be recommended for a canary version?

Make a prediction

What it is

What the rollback decision function is

What it is not

What the rollback decision function is not

How it relates to neighboring concepts

Relationship with other components

Start with a concrete problem

Why is an ordered timeline a prerequisite for trustworthy recovery metrics?

Make a prediction

What it is

What incident drill verification is

What it is not

What incident drill verification is not

How it relates to neighboring concepts

Relationship to other components