Interview, System Design & Industry Judgment

Start with a concrete problem

Interview project narrative lacks verifiable evidence

You tell an interviewer about a project: after long tasks failed, you added a recovery mechanism and achieved a 100% recovery rate.

The interviewer asks, “How was that 100% verified? Do you have code, tests, or logs?” You can only describe it verbally, with no local evidence to show.

The current codebase has no forge/interview package, so you cannot use ProjectClaim to bind evidence and reject unsupported claims.

You need to create a ProjectClaim dataclass whose validate() method requires at least one immutable local evidence artifact, otherwise it raises a ValueError containing ‘evidence’.

Make a prediction

What happens if ProjectClaim.validate() only checks that the four narrative fields are non-empty and ignores the evidence tuple?

• A claim without evidence is accepted, and the test test_claim_without_evidence_is_rejected fails.
• A claim without evidence is rejected, and the test passes.
• The program crashes because evidence is undefined.
• Nothing happens because evidence is not required.

Reasoning guide: The correct answer is the first option. If validate() does not check evidence, an empty tuple () passes validation, so the expected ValueError is not raised and the test fails.

What it is

Evidence-bound project claim

ProjectClaim is an immutable dataclass that binds the four parts of a project narrative (situation, task, action, result) to an evidence tuple.

The validate() method enforces two rules: narrative fields must be non-empty, and the evidence tuple must contain at least one local file path.

Local evidence paths point to immutable artifacts such as code, tests, logs, or data files that an interviewer can inspect after the interview.

What it is not

Not a fluent story or online links

It is not a plain dataclass with only narrative text and no evidence; claims without evidence are rejected.

It does not accept web links as evidence because link content can change or disappear, violating immutability.

It does not guarantee that project results are correct; it only ensures that claims are backed by inspectable local evidence.

How it relates to neighboring concepts

Constraint relationship between evidence and narrative

Each path in the evidence tuple points to a local file that substantiates the outcome claimed in the result field.

validate() checks evidence after checking narrative fields, ensuring that only complete and evidenced claims pass.

The test test_evidenced_claim_passes uses evidence/recovery.json as valid evidence, while test_claim_without_evidence_is_rejected uses an empty tuple to verify rejection logic.

Boundary decision for this stage: If the evidence tuple is empty or contains a path starting with ‘http’, validate() must raise a ValueError; only local file paths are considered valid evidence.

Start with a concrete problem

Select the lowest-cost model that satisfies all constraints from multiple candidates

You are designing a model routing component for an interview system that must choose the lowest-cost model satisfying quality, latency, cost, and modality constraints from multiple candidates.

The codebase currently lacks forge/interview/model_tradeoffs.py, so tests cannot import ModelOption and choose_model, causing a collection error.

You need to create this file and implement the model selection logic so that test_cheapest_eligible_model_is_selected and test_required_modality_is_enforced pass.

If you select solely by cost and ignore modality constraints, you will incorrectly choose a text-only model for an image-capable task, which is the failure this stage prevents.

Make a prediction

When implementing choose_model, what happens if you select the model with the lowest cost while ignoring the required_modalities constraint?

• The test test_required_modality_is_enforced fails because text-cheap is selected even though it lacks image modality.
• The test still passes because the lowest-cost model always satisfies all constraints.
• The program raises an exception because modality constraints cannot be checked.
• The test fails because choose_model cannot handle multiple models.

Reasoning guide: The correct answer is the first option. Ignoring modality constraints leads to selecting the text-only model text-cheap for an image-capable task, causing test_required_modality_is_enforced to fail. Modality constraints must be hard filters, not post-hoc checks.

What it is

What the model tradeoff matrix is

The model tradeoff matrix is a decision function that takes a list of candidate models and constraints, returns the lowest-cost model satisfying all constraints, and lists the names of rejected models.

It narrows the candidate set through hard filters on quality, latency, cost, and modality, then selects the optimal model by cost among the remaining options.

If no model satisfies all constraints, the function raises a ValueError to clearly indicate that the requirements cannot be met.

What it is not

What the model tradeoff matrix is not

It is not a simple cost comparator; it cannot ignore dimensions like quality, latency, or modality.

It is not a post-hoc check mechanism; modality constraints must be hard conditions during filtering, otherwise an incorrect model may be selected.

It does not guarantee that a model satisfying all constraints exists, so it must handle the no-solution case.

How it relates to neighboring concepts

Relationships with other concepts

The model tradeoff matrix is part of routing decisions in system design and relies on the ModelOption dataclass to encapsulate model attributes.

It relates to verification and evaluation: test cases assert the selected model name to verify the correctness of the decision logic.

It relates to observability: the function returns rejected model names to facilitate logging and explaining the decision process.

Boundary decision for this stage: Model selection must satisfy quality, latency, cost, and modality constraints simultaneously; any omission leads to incorrect routing. Modality constraints must be hard filters, not post-hoc checks.

Start with a concrete problem

Why does a seemingly normal tool definition hide risk?

You are adding tool design capability to the interview system design module. The file forge/interview/tool_design.py does not exist yet, so the tests test_read_tool_has_no_findings and test_idempotent_write_requires_key fail at import time.

You need to implement the ToolDesign dataclass and the review_tool function to make the tests pass. The real challenge is that tool review cannot only check name and argument consistency; it must also consider side-effect semantics.

For example, an idempotent write tool like create_issue without an explicit idempotency key argument could cause duplicate side effects on repeated requests. Your review logic must detect this overbroad capability.

Make a prediction

Before implementing review_tool, predict: for ToolDesign("create_issue", "Create an issue", ("title",), ("title",), "write", True), what should the review function return?

• Return an empty tuple because the name and arguments are valid.
• Return a tuple containing "missing-idempotency-key" because the idempotent write tool lacks an idempotency key.
• Return a tuple containing "invalid-schema" because required_args is not a subset of allowed_args.
• Return a tuple containing "unknown-side-effect" because the side_effect value is not in the allowed set.

Reasoning guide: The correct answer is the second option. Tool review must check side-effect semantics: for idempotent write or destructive tools, an explicit idempotency key argument is required. This tool is a write operation and idempotent, but allowed_args does not include idempotency_key, so it should report a missing idempotency key.

What it is

What tool contract review is

Tool contract review is a static check of the structure and semantics of a tool definition to discover potential risks. It takes a ToolDesign object and returns a tuple of finding strings.

The review must cover identity clarity, argument consistency, side-effect type, and idempotency key requirements. Each check is independent and can combine to produce multiple findings.

What it is not

What tool contract review is not

It is not runtime behavior verification; it does not actually call the tool or inspect execution results. It only performs static analysis based on the tool definition itself.

It is also not a simple consistency check of tool names and argument formats; side-effect semantics and idempotency are core parts of the review.

How it relates to neighboring concepts

Relationship to other concepts

Tool contract review is a safety boundary in agent design: the harness can call the review function before executing a tool to reject tools with overbroad capabilities.

It relates to verification and evaluation: review results can serve as verification evidence for tool usability, but the review itself does not replace runtime verification.

Boundary decision for this stage: Tool review must consider side-effect semantics: for idempotent write or destructive tools, an explicit idempotency key argument is required. If the idempotency key is missing, the review should report missing-idempotency-key.

Start with a concrete problem

RAG system frequently gives unsupported answers, but only prompts are being adjusted

You inherit a RAG question-answering system where online metrics show a 20% unsupported answer rate, yet the team keeps modifying prompts without improvement.

You need to implement a diagnostic function diagnose that takes RagMetrics (recall, citation precision, unsupported answer rate) and returns the improvement actions to take.

The codebase currently lacks forge/interview/rag_reasoning.py, so tests test_low_recall_points_to_retrieval and test_unsupported_answers_require_abstention_gate fail to import.

Your task is to create that file, define the RagMetrics dataclass and diagnose function, and make the diagnostic logic distinguish retrieval, citation, and abstention issues.

Make a prediction

When a RAG system has a high unsupported answer rate (e.g., 20%), what is the most reasonable diagnostic action?

• Only improve retrieval to increase recall
• Only improve citation to increase citation precision
• Add an abstention gate to avoid unsupported answers
• Hold baseline and continue observing

Reasoning guide: A high unsupported answer rate means the model generates answers even when evidence is lacking. Improving retrieval or citation alone cannot directly stop this behavior; an explicit abstention gate is required to intercept such outputs.

What it is

RAG diagnostic model

The RAG diagnostic model decomposes system problems into three independently measurable dimensions: recall (whether retrieval finds relevant documents), citation precision (whether generated content is supported by retrieved evidence), and unsupported answer rate (whether the model answers when evidence is insufficient).

Each dimension maps to a specific improvement action: low recall points to improve-retrieval, low citation precision points to improve-grounding, and high unsupported answer rate points to add-abstention-gate.

The diagnose function checks each dimension against thresholds and returns all required actions; if all metrics are normal, it returns hold-baseline.

What it is not

Not just prompt tuning

RAG diagnosis is not merely adjusting prompt wording; prompt tuning cannot fix structural issues like insufficient retrieval recall or missing citation evidence.

It is also not using a single composite score to represent system health; recall, citation, and abstention metrics must be examined separately because their failure modes and remedies differ.

The diagnostic result is not a model’s subjective opinion but deterministic rules based on quantifiable metrics and fixed thresholds.

How it relates to neighboring concepts

Mapping between metrics and actions

When recall recall_at_k is below 0.8, the retriever fails to rank relevant documents in the top k results, requiring improvements to retrieval strategy or index.

When citation precision citation_precision is below 0.9, the generated content cites irrelevant or incorrect documents, requiring improvements to grounding or citation verification.

When unsupported answer rate unsupported_answer_rate is above 0.05, the model generates answers despite insufficient evidence, requiring an abstention gate that returns ‘unable to answer’ when confidence is low.

The three metrics are independent; an anomaly in one does not imply anomalies in others. The diagnostic function must check each and accumulate all needed actions.

Boundary decision for this stage: When the unsupported answer rate exceeds 5%, an abstention gate must be added even if recall and citation precision are normal; otherwise the system will continue producing unsupported answers and erode user trust.

Start with a concrete problem

Why is listing threats alone not enough?

In an interview or system design discussion, you might quickly list a few threats such as ‘tool escape’ or ‘prompt injection’ and consider the threat model complete.

But without checking whether these threats cover all critical assets, you may miss important protected objects like API keys.

This stage requires you to implement a validation function that must ensure the threat model covers all required assets, otherwise it raises ValueError.

The starting code does not yet have the file forge/interview/security_reasoning.py, so the tests fail at import time.

Make a prediction

Before implementing validate_threat_model, which behavior do you think is correct?

• As long as the threat list is non-empty, the threat model is valid.
• You must check that every required asset is covered by at least one threat entry, otherwise reject.
• You only need to check that threat IDs are unique; asset coverage is not important.
• Threat models should be reviewed by humans; code does not need automatic validation.

Reasoning guide: The correct option is the second: the threat model must cover all required assets. If any asset is omitted, the validation function should raise ValueError because an uncovered asset means there is unanalyzed risk.

What it is

Threat model as a coverage contract

A threat model is a mapping from assets to threats: every critical asset must have at least one threat entry describing its risk, boundary, mitigation, and verification.

The validate_threat_model function enforces this contract by comparing the set of assets in threat entries with the required asset set.

If there are missing assets, the function raises a ValueError containing the missing asset names, preventing incomplete threat models from passing.

What it is not

Not a free-form threat list

A threat model is not just listing a few threats arbitrarily; it must systematically cover all assets that need protection.

Checking only threat ID uniqueness is insufficient because even with unique IDs, asset coverage may still be incomplete.

A threat model is also not a static document; it should be automatically verifiable by code so that omissions are quickly detected when the asset set changes.

How it relates to neighboring concepts

Relationship among assets, threats, and validation

Assets are objects that need protection, such as API keys, workspaces, or documents.

A threat entry declares which asset it protects via the asset field and describes risk and controls via boundary, mitigation, and verification fields.

validate_threat_model receives a tuple of threats and a set of required assets, computes the covered asset set, then finds and reports missing assets.

Boundary decision for this stage: The validation function only checks asset coverage and basic completeness; it does not evaluate the quality of threat descriptions or the effectiveness of mitigations, which require human review or more complex analysis.

Start with a concrete problem

A single-failure-domain production design passes capacity calculation

You are preparing a system design case for an interview: given peak traffic, average latency, target utilization, and number of failure domains, calculate the required number of workers.

The initial code only computes concurrency using Little’s Law and rounds up, completely ignoring the number of failure domains, so a production design with only one failure domain passes capacity calculation.

The test test_single_failure_domain_is_rejected expects a ValueError when failure_domains=1, but the current implementation returns a valid worker count, causing the test to fail.

You need to modify the required_workers function in forge/interview/system_design.py to reject designs with insufficient failure domains and ensure capacity calculation is correct.

Make a prediction

Before modifying the code, predict: if the required_workers function only computes concurrency and rounds up without checking the number of failure domains, what will happen?

• The test test_capacity_uses_littles_law will fail because the capacity calculation is wrong.
• The test test_single_failure_domain_is_rejected will fail because a single-failure-domain design is accepted.
• Both tests will pass because capacity calculation is independent of failure domains.
• Both tests will fail because the function is missing necessary parameters.

Reasoning guide: The correct answer is the second option. The current implementation only focuses on capacity calculation and does not check the number of failure domains, so a single-failure-domain design will be accepted, causing test_single_failure_domain_is_rejected to fail. The first test still passes because the capacity calculation itself is correct.

What it is

Little’s Law and failure domain constraints in capacity planning

Little’s Law states that the average number of concurrent requests in a system equals the arrival rate multiplied by the average response time, i.e., L = λ × W.

In capacity planning, we use peak RPS as the arrival rate and average latency as the response time to get concurrency, then divide by target utilization to get the required number of workers.

A failure domain is the scope of impact when a component or region fails; production systems need at least two failure domains to avoid a single point of failure.

The required_workers function must satisfy both capacity requirements and failure domain constraints: the calculated worker count cannot be less than the number of failure domains, and the number of failure domains must be at least 2.

What it is not

Capacity calculation is not simple traffic division

Capacity calculation is not just about traffic and latency; it must also consider the number of failure domains. A single-failure-domain design should be rejected even if the capacity arithmetic is correct.

Little’s Law gives the average concurrency, not the peak concurrency; actual capacity planning needs to consider peak traffic and utilization headroom.

The number of failure domains is not an optional optimization parameter but a fundamental requirement for production systems; lacking failure domain constraints leads to complete unavailability when a single point fails.

How it relates to neighboring concepts

Relationship between capacity, utilization, and failure domains

Capacity calculation is based on Little’s Law: concurrency = peak RPS × average latency, then divided by target utilization to get the required number of workers.

The failure domain constraint requires the worker count to be at least the number of failure domains, and the number of failure domains must be at least 2; this ensures the system can continue operating when a single failure domain fails.

The final worker count is the larger of the capacity calculation and the number of failure domains, satisfying both performance and availability requirements.

Boundary decision for this stage: Production system design must have at least two failure domains; single-failure-domain designs should be rejected even if capacity arithmetic is correct.

Start with a concrete problem

Interview evaluation cannot rely on a single total score

In the previous stage, you completed capacity planning; now you need to build a mock interview evaluation module.

Interviewers must score four dimensions separately: correctness, tradeoffs, evidence, and communication, and then provide improvement suggestions for the weakest dimension.

If any scoring dimension is missing, the evaluation must be rejected, otherwise it is impossible to locate the interviewee’s specific weaknesses.

Currently, the file forge/interview/mock_interview.py does not exist, so tests fail with an import error.

Make a prediction

If the evaluate function receives an InterviewScore containing only the correctness and communication dimensions, what should it do?

• Compute the average of these two dimensions and return the weakest dimension
• Raise a ValueError because the scoring dimensions are incomplete
• Automatically fill in missing dimensions with default scores
• Return None to indicate that evaluation is impossible

Reasoning guide: The correct option is to raise a ValueError. Interview evaluation must use the fixed four dimensions; any missing dimension means the evaluation is incomplete and cannot produce a score or suggestion.

What it is

Structured interview evaluation

Interview evaluation is a function that receives an InterviewScore object containing scores for four dimensions and returns the average score, the weakest dimension, and the next action.

The four dimensions are correctness, tradeoffs, evidence, and communication; each dimension requires an integer score from 1 to 5 and a non-empty evidence note.

The evaluation function first checks dimension completeness, then checks score ranges, and finally checks that notes are non-empty; only after all checks pass does it compute the weakest dimension.

What it is not

Not arbitrary scoring

Interview evaluation is not simply averaging the existing scores, nor is it letting the model judge performance on its own.

It cannot accept scores with missing dimensions, nor can it ignore evidence notes; otherwise the evaluation result is untrustworthy.

The evaluation result must be based on externally checkable rules, not subjective opinions.

How it relates to neighboring concepts

Relationship between evaluation and verification

The evaluate function is part of the verification mechanism; it ensures that interview evaluation conforms to the fixed scoring dimension contract.

The InterviewScore dataclass stores scores and notes, and evaluate uses this data to perform checks and generate results.

The test case test_incomplete_rubric_is_rejected verifies that an exception is raised when dimensions are missing, and test_complete_rubric_targets_weakest_dimension verifies that the weakest dimension is computed correctly.

Boundary decision for this stage: Interview evaluation must use the fixed four dimensions: correctness, tradeoffs, evidence, and communication; any missing dimension should be rejected.