RAG, Retrieval & Knowledge Evaluation

Start with a concrete problem

Different versions of the same document collapse to one identity in the system

In a RAG system, retrieval quality depends on every document fragment having a distinguishable and immutable identity. Currently the forge/rag package does not exist, so the tests fail on import with ModuleNotFoundError when trying to import from forge.rag.documents import ingest_document, causing all three target tests to fail collection.

We need to create the forge/rag/documents.py file and implement the ingest_document function so it accepts a source path, version string, raw text, and optional metadata, then normalizes whitespace in the text and generates a document identifier.

A common mistake is to generate document_id from only the source and normalized content while treating version as mere metadata. When the same source moves from version a to version b with unchanged content, both versions collapse to the same ID, so downstream retrieval cannot distinguish new from old evidence and may cite stale material.

Make a prediction

In ingest_document, if the document_id formula is sha256(f"{source}|{normalized}"), what happens when you call ingest_document("guide.md", "a", "body") and ingest_document("guide.md", "b", "body")?

• The two document_id values differ because the versions differ
• The two document_id values are identical because version is not part of the hash
• The two document_id values differ because Python produces different results each call
• The two document_id values are identical because source and text are exactly the same
• The first call raises ValueError

Reasoning guide: The correct answer merges the second and fourth options: because the hash input contains only source and normalized, versions a and b do not affect the hash result, so both documents collapse to the same document_id. This is exactly the root cause of the test_version_change_creates_a_new_document_identity failure in the fault lab.

What it is

Components of document identity

Document identity (document_id) is an immutable hash digest that must bind together source (source), version (version), and normalized content (normalized). Only when all three participate in the hash can any single dimension change produce a new identity.

Normalized content is produced by calling " ".join(text.split()), which compresses all consecutive whitespace (including newlines, tabs, and multiple spaces) into a single space, ensuring that semantically identical text with different formatting does not produce a different identity.

Document is a frozen dataclass (frozen=True), so its fields cannot be modified after creation, which structurally guarantees the immutability of document identity.

What it is not

What document identity is not

Document identity is not a hash of content alone. If you hash only the normalized text, then identical content from different sources collapses to the same identity, losing source provenance.

Document identity is also not a hash of metadata. The metadata dictionary may contain team names, tags, or other mutable information; including it in identity would make the identity unstable when metadata updates, violating immutability.

Version is not just a display label. Version is a core component of identity: the same source and content at different versions must have different document_id values, otherwise the system cannot distinguish a draft from a final version.

How it relates to neighboring concepts

Relationship between identity, normalization, and validation

Normalization guarantees identity stability: it ensures whitespace differences do not create new identities, which is what makes test_document_id_is_stable_after_whitespace_normalization pass.

Version binding guarantees identity distinguishability: it ensures version changes produce new identities, which is what makes test_version_change_creates_a_new_document_identity pass.

Empty content validation is a precondition guard for identity validity: before generating an identity, ingest_document must check that normalized content is non-empty, otherwise even a blank string would produce a seemingly valid hash, which is what makes test_empty_content_is_rejected pass.

Boundary decision for this stage: The boundary for document identity is: source, version, and normalized content jointly determine identity; whitespace normalization must not change identity, version changes must change identity, and empty content must be rejected rather than producing an empty identity.

Start with a concrete problem

Fixed-length chunking loses boundary context

In a retrieval-augmented generation system, long documents must be split into smaller chunks for vector retrieval, but if you split mechanically by fixed length without overlap, key facts that cross chunk boundaries disappear from retrieval results.

The file forge/rag/chunks.py does not exist yet, so the test module raises ModuleNotFoundError when importing chunk_document; you must create this file and implement structure-aware chunking logic.

The core engineering decision in this stage is that adjacent chunks must share a controlled number of words so that context on both sides of a boundary is retrievable from either chunk, while each chunk must carry a traceable document identifier and ordinal.

Make a prediction

When chunk_document splits a seven-word document with max_words=4, overlap_words=2, if the next chunk’s start position is set directly to the current chunk’s end position (i.e., start = end), what will the second chunk’s text be?

• The second chunk text will be ‘five six seven’ because the start position did not step back, losing the overlap words.
• The second chunk text will be ‘three four five six’ because the start position stepped back two words to preserve overlap.
• The second chunk text will be ‘one two three four’ because the start position did not change, causing the first chunk to be generated again.
• The second chunk text will be ‘two three four five’ because the start position stepped back only one word, causing insufficient overlap.

Reasoning guide: If start = end, there is no overlap; the second chunk starts at index 4 and becomes ‘five six seven’. The correct implementation sets start = end - overlap_words, so the second chunk starts at ‘three’ and becomes ‘three four five six’.

What it is

Controlled-overlap chunking model

Controlled-overlap chunking means that when splitting a document by word list, each new chunk’s start position steps back overlap_words words, so adjacent chunks share a segment of text at the boundary.

This mechanism ensures that even if a key fact straddles the theoretical split point between two chunks, it still appears completely in at least one chunk, so vector retrieval does not lose context due to boundary cuts.

Chunk IDs use the format documentID:ordinal, with ordinal incrementing from 0, making each chunk traceable to its source document and relative position within that document.

What it is not

Boundaries of the chunking model

Chunking is not semantic understanding; it does not judge word meanings or sentence structure, but only operates on the word list after splitting by whitespace.

Overlap is not a defect of redundant storage but an intentional retrieval safeguard; it differs from simple duplicate storage because its purpose is to repair the information break at fixed-window boundaries.

Chunking does not replace reranking or citation verification; it only splits the document into independently retrievable text units, and subsequent retrieval quality evaluation must still be performed separately.

How it relates to neighboring concepts

Dependency between chunking and document ingestion

chunk_document receives the Document object returned by ingest_document as input, relying on its document_id and text fields to generate a list of Chunk objects with provenance information.

Chunk is a frozen dataclass with four fields: chunk_id, document_id, ordinal, and text, where document_id is inherited directly from the parent document to keep the provenance chain intact.

There is a constraint between the chunking parameters max_words and overlap_words: overlap_words must be strictly less than max_words; otherwise, the step-back operation would prevent the start position from advancing, causing an infinite loop.

Boundary decision for this stage: When overlap_words >= max_words, start = end - overlap_words would make start not greater than the original start, so the loop cannot terminate; therefore, this parameter combination must be rejected with a ValueError before entering the loop.

Start with a concrete problem

Building a reproducible semantic-similarity test baseline from scratch

In a retrieval-augmented generation system, we need a reproducible method to measure semantic similarity between a query and document snippets. The system already has forge/rag/chunks.py, but forge/rag/vectors.py does not exist yet, causing test collection to fail with a ModuleNotFoundError.

This stage requires generating fixed-dimension normalized vectors from text using only deterministic local code, without relying on any external pretrained models, so that text pairs sharing lexical terms achieve a higher cosine similarity than unrelated text pairs.

The core challenge is that the bag-of-words hash mapping must actually accumulate a count for each token, and the resulting vector must be normalized; otherwise, cosine similarity degrades to zero or becomes biased by text length.

Make a prediction

When the embed function vectorizes text, if the contribution for each token in the loop is set to += 0.0 instead of += 1.0, what will happen to the assertion cosine(query, related) > cosine(query, unrelated)?

• Both similarities are 0.0 and the assertion fails
• The related text gets a higher similarity and the assertion passes
• The unrelated text gets a higher similarity and the assertion fails
• A dimension mismatch exception is raised

Reasoning guide: Because all token contributions are discarded, the vectors remain entirely zero. During normalization, the norm of a zero vector is 0, which falls back to 1.0, ultimately returning a zero vector. The cosine similarity of two zero vectors is 0.0, so the assertion 0.0 > 0.0 fails.

What it is

Deterministic embeddings and bag-of-words hash mapping

A deterministic embedding always generates the exact same vector representation for the same input text, without relying on random seeds or external network requests. This implementation uses hashlib.sha256 to map each token into a fixed-dimension bucket, forming a sparse vector based on bag-of-words features.

Cosine similarity measures the alignment of two vector directions, and its range is [-1, 1] after normalization. In a bag-of-words model, the more shared terms two texts have, the closer their components are in the corresponding buckets, resulting in a higher cosine value.

What it is not

Boundaries and limitations of deterministic embeddings

A deterministic bag-of-words embedding is not a pretrained semantic model; it cannot capture synonyms or contextual semantics. For example, ‘python’ and ‘snake’ are completely unrelated in this model because they are different token strings.

It is also not a one-off keyword-triggered file lookup, but rather an encoding of a text’s lexical features into a direction in geometric space, allowing lexical overlap to be quantitatively compared.

How it relates to neighboring concepts

Causal links between embeddings, context, and evaluation

Embedding vectors support the retrieval phase of RAG by ranking candidate document snippets via cosine similarity. The selected snippets are injected into the context for the model to reference when generating an answer.

The evaluation phase separately measures retrieval quality (whether similarity ranking is reasonable) and citation quality (whether the answer is based on the injected evidence). Deterministic embeddings ensure the reproducibility of the evaluation baseline, making every test run consistent.

Boundary decision for this stage: When two vectors have mismatched dimensions, the cosine function must raise a ValueError because vector directions in different dimensional spaces are incomparable, and silently returning a value would cause unpredictable errors in retrieval ranking.

Start with a concrete problem

Retrieval without pre-scoring source filtering leaks out-of-scope chunks

In the current code state, the file forge/rag/retrieval.py does not exist, so the test module fails at import time with ModuleNotFoundError: No module named 'forge.rag.retrieval', aborting collection before any test runs.

This stage requires you to create that file and implement the retrieve function so that, before computing cosine similarity, it removes every candidate chunk that does not satisfy the provenance metadata constraints in the filters dictionary.

If you score first and filter afterward, a chunk from the private team could be highly relevant to the query and occupy a top-k slot, leaking out-of-scope data into an engineering-only search.

Make a prediction

If retrieve first computes similarity and sorts the entire index, then applies filters afterward, and top_k=3 with only one chunk remaining after filtering, what is the length of the final returned list?

• 1, because only one chunk remains after filtering
• 3, because the top three were already selected during sorting
• 0, because filtering clears the already-sorted results
• 2, because sorting and filtering do not affect each other

Reasoning guide: Sorting first and filtering later means sorted(hits)[:top_k] already truncates the list, so filtering afterward can yield fewer than top_k results, but worse, out-of-scope chunks have already consumed top-k slots and pushed legitimate results out. The correct approach is to filter before scoring so the candidate set contains only chunks from allowed sources.

What it is

A retrieval pipeline that filters before scoring

The retrieve function takes a query string, an index list, top_k, and an optional filters dictionary, and its core pipeline has three steps: first filter the candidate set by metadata, then compute cosine similarity for the filtered candidates, and finally sort by descending score and take the top top_k results.

IndexedChunk binds a Chunk, its embedding vector, and a metadata dictionary together; the key-value pairs in metadata (such as team) are the basis for filtering, and only candidates where every filter key matches the metadata value enter the scoring stage.

SearchHit is the final returned hit, containing the original Chunk and the similarity score; sorting uses (-score, chunk_id) as the key to ensure a deterministic order when scores are equal.

What it is not

Filtering is not a post-processing step after ranking

Filtering is not a cleanup step executed after the [:top_k] truncation, because truncation happens after sorting, and if high-scoring out-of-scope chunks are mixed into the candidates, they occupy slots and push legitimate lower-scoring chunks out of the result.

Filtering is not a modification of the query vector or a penalty on scores; it is a boolean set-selection operation that removes illegal candidates from the list before similarity is computed.

retrieve does not re-embed documents or modify the index contents; it only reads index and filters and returns a new list of SearchHit objects.

How it relates to neighboring concepts

Causal order of filtering, scoring, and sorting

The filter operation determines the input set for scoring: if filtering happens before scoring, scoring only applies to legal candidates; if filtering happens after scoring, scoring applies to all candidates, and illegal candidates’ scores may exceed those of legal ones.

Sorting stability relies on chunk_id as a secondary sort key; when two candidates have the same score, the one with the smaller chunk_id comes first, ensuring consistent ordering across runs for the same index and query.

When the filtered candidate set is empty, retrieve returns an empty list rather than raising an error, because having no matching content in the allowed sources is a normal retrieval outcome.

Boundary decision for this stage: Filtering must be applied before scoring to ensure that chunks from irrelevant sources do not enter ranking; top-k is returned in descending score order, with ties broken by ascending chunk_id for stability.

Start with a concrete problem

The evidence with the highest initial similarity is not always the best answer source

In the previous stage, you implemented vector similarity retrieval that returns a set of candidate evidence from a document store. However, retrieval results are often sorted by similarity alone, and a high-similarity chunk may only mention the topic broadly without covering the specific terms the user actually cares about.

For example, given the query “python retry policy”, a chunk containing “python retry overview” has a high similarity of 0.95, but it never mentions the terms “permission” or “policy”. Another chunk containing “python retry permission failure policy” has a lower similarity of 0.75, yet it fully covers those required terms.

If you sort only by initial similarity, the first chunk ranks ahead, but the user likely needs the second chunk that contains the specific policy details. Therefore, we need a mechanism that considers both initial similarity and required-term coverage during sorting, so that more relevant evidence ranks higher.

In this stage you will create forge/rag/rerank.py and implement a rerank function that accepts a query, a list of candidate hits, and an optional set of required terms, then returns a reordered list. This function must remain deterministic: identical inputs must produce an identical order, otherwise tests cannot pass reliably.

Make a prediction

Before implementing the reranking function, predict: given two candidate evidence items where one has higher initial similarity but covers fewer required terms, and the other has lower initial similarity but covers more required terms, which should rank first after reranking?

• The evidence with higher initial similarity should always rank first because similarity is the primary retrieval metric.
• The evidence covering more required terms should rank first even if its initial similarity is lower.
• The order should be chosen randomly because both factors are important.
• The original order should be preserved without any change.

Reasoning guide: The correct answer is the second option. The purpose of reranking is to prioritize evidence that covers more required terms even if its initial similarity is lower. However, this does not mean similarity is fully ignored; the two factors are combined, and determinism must be guaranteed.

What it is

Reranking is a deterministic sort combining similarity and term coverage

The rerank function takes a query string, a list of SearchHit objects, and an optional set of required terms, then returns a new list where each element is still the original SearchHit object but the order may have changed.

The key to sorting is computing a score for each hit, where the score is a tuple containing two parts: required-term coverage (the fraction of required terms that appear in the text) and adjusted similarity (the original similarity plus query-term coverage multiplied by 0.25).

Sorting uses the sorted function with reverse=True, so hits with higher scores rank first. Because the score tuple includes chunk_id as the final comparison item, when two hits have identical scores the order remains deterministic.

What it is not

Reranking is not simple similarity sorting or random shuffling

Reranking is not merely sorting by initial similarity, because that would defeat its purpose; it must consider required-term coverage, otherwise it cannot solve the problem addressed in this stage.

Reranking is also not random shuffling, because it must remain deterministic, meaning identical inputs must produce identical outputs, otherwise tests cannot pass reliably.

Reranking does not modify the original list; it returns a new list while the original list stays unchanged, which follows functional programming conventions.

How it relates to neighboring concepts

How reranking relates to retrieval, context, and evaluation

Reranking is a downstream step in the retrieval pipeline: retrieve returns candidate hits, rerank adjusts their order, and only then is the sorted evidence injected into context for the model to use.

The quality of reranking directly affects the quality of the final answer, because the model can only see the reranked evidence; if key evidence is ranked lower, it may be truncated or ignored.

The evaluation stage checks whether reranking is effective; for example, the test test_required_term_coverage_can_promote_better_evidence verifies that evidence covering required terms ranks first.

Boundary decision for this stage: Reranking must remain deterministic, so the sort key must include a unique identifier such as chunk_id as the final comparison item to avoid unstable ordering when scores are tied.

Start with a concrete problem

An answer claim cites a chunk ID that does not exist in the retrieval results

In a RAG system, every claim in a generated answer must be bound to an actually retrieved evidence chunk; otherwise the answer loses its verifiable factual basis. The file forge/rag/citations.py does not exist yet, so the tests fail immediately with ModuleNotFoundError when importing Claim and build_grounded_answer.

The learner must create this module from scratch, defining the Claim and GroundedAnswer dataclasses and implementing the build_grounded_answer function. That function must not only build the evidence mapping but also validate that every citation ID referenced by a claim actually exists among the retrieved results.

If a citation ID is syntactically present but never appeared in the retrieval results, the system cannot trace the claim back to any source, creating a hallucination risk. Therefore unknown citations and uncited claims must both be explicitly rejected.

Make a prediction

When build_grounded_answer receives a claim citing c-9 but the hits list only contains c-1, what should the function do?

• Accept the claim because c-9 is a valid string ID
• Raise ValueError because c-9 is not in the retrieved evidence map
• Ignore the c-9 citation and continue building the answer
• Return an empty GroundedAnswer object

Reasoning guide: The correct answer is to raise ValueError. A citation ID must have a traceable evidence source within the retrieval result set, and any ID that cannot be mapped to an actual retrieved chunk is an unknown citation that must be rejected.

What it is

Citation provenance is a binding contract between claims and retrieved evidence

Citation provenance requires that every claim in an answer is explicitly bound to a SearchHit chunk from the retrieval results through its citation_ids. The build_grounded_answer function first builds an evidence map from hits, then checks each claim’s citation IDs against the keys of that map.

This binding mechanism ensures answer verifiability: any claim can be traced back to its original retrieved text through the answer.evidence dictionary, preventing the model from fabricating plausible but unsupported claims.

What it is not

Citation provenance is not merely syntax or format validation

Citation validation is not equivalent to checking whether citation_ids is a non-empty tuple or whether the string format is correct. Even if a citation ID is a well-formed string, if it never appeared in the retrieval results it is still an invalid citation.

Citation provenance is also not a factual judgment about the answer content itself; the system does not evaluate whether the claim text is true, only whether it has corresponding retrieved evidence supporting it.

How it relates to neighboring concepts

Dependencies among the evidence map, claim citations, and rejection logic

SearchHit provides the chunk_id and text of each retrieved chunk, and build_grounded_answer transforms these into an evidence dictionary. Claim references these keys through citation_ids, and the function uses set difference operations to detect unknown citations.

When citation_ids is an empty tuple the claim lacks citations and triggers a requires error; when duplicate IDs exist a duplicate error is triggered; when an ID is not in evidence an unknown error is triggered. These three layers of checking together form the complete citation validation chain.

Boundary decision for this stage: The boundary of citation validation is that the system only verifies whether citation IDs exist in the evidence map of retrieval results; it does not evaluate the content quality or factual accuracy of the retrieved chunks, nor does it perform semantic truth-value judgments on the claim text.

Start with a concrete problem

Create the RAG evaluation module from scratch and fix the citation precision denominator

In the previous stage, build_grounded_answer in forge/rag/citations.py already produces cited answers, but no module currently quantifies retrieval and citation quality. The test file tests/test_stage.py attempts to import EvaluationCase and evaluate from forge.rag.evaluation, but because forge/rag/evaluation.py does not yet exist, the test collection phase immediately raises ModuleNotFoundError.

This stage requires creating forge/rag/evaluation.py from scratch, defining the EvaluationCase dataclass to encapsulate each test case’s case_id, relevant_chunk_ids, retrieved_chunk_ids, and cited_chunk_ids, and implementing the evaluate function to compute recall_at_k, citation_precision, and failed_case_ids.

When implementing citation precision, a subtle but critical error is using the number of cases as the denominator instead of the total number of citations. This causes the system to report a perfect precision of 1.0 even when invalid citations exist, thereby masking the risk that the RAG system is fabricating evidence.

Make a prediction

In the evaluate function, if you divide the valid citation count by the total number of cases, what value will precision return for a test set containing two cases that produce three total citations (two of which are valid)?

• Returns 0.6666666666666666, which is two divided by three.
• Returns 1.0, because two valid citations divided by two cases equals one.
• Returns 0.5, because only half of the cases are fully correct.
• Raises a division by zero exception, because the denominator calculation is incorrect.

Reasoning guide: If the denominator is the total case count of 2, then 2 divided by 2 equals 1.0. This incorrectly masks the fact that one invalid citation exists. The correct denominator must be the total citation count of 3 across all cases, yielding 2/3.

What it is

The mathematical model of layered RAG evaluation

Recall@K measures retrieval stage capability by dividing the number of relevant chunks retrieved for each case by the total number of relevant chunks for that case, then taking the arithmetic mean across all cases. If a case has no relevant chunks, the max(1, len(relevant_chunk_ids)) expression avoids division by zero and records its recall as 0.

Citation precision measures the evidence fidelity of the generation stage by dividing the total number of citations pointing to truly relevant chunks across all cases by the total number of citations generated across all cases. When there are no citations at all, precision is defined as 1.0 because the system fabricated no evidence.

The failed_case_ids tuple collects every case identifier whose recall is strictly less than 1.0, enabling subsequent improvement analysis targeting retrieval failures. These three metrics together form the basic evaluation report for a RAG system.

What it is not

Unit confusion in evaluation metrics

Citation precision is not a case-level pass rate. You cannot record 1 for a case with one valid citation, record 0 for another case with one invalid citation, and then average them. That approach discards citation-level granularity.

Evaluation is not asking the model for another opinion about its own answer. Pass or fail must resolve to externally inspectable mathematical calculations, specifically the intersection between the pre-labeled relevant_chunk_ids in the test set and the system’s actual output.

Recall@K does not equal citation precision. Retrieving a relevant chunk does not guarantee the model correctly cited it during generation, so the two must be computed independently.

How it relates to neighboring concepts

Dependencies between the evaluation module and existing RAG components

The forge/rag/evaluation.py module depends on the output structure of Claim and build_grounded_answer defined in forge/rag/citations.py, but the evaluation itself is an independent offline computation process that does not modify the runtime behavior of retrieval or generation.

The EvaluationCase dataclass encapsulates the retrieval results from forge/rag/retrieval.py and the citation results from forge/rag/citations.py into computable static data, allowing evaluation to run quickly in deterministic Mock mode without actual model calls.

The failed_case_ids output by EvaluationReport provide failure nodes for subsequent Observability and Planning modules to focus on, forming a closed loop from evaluation to improvement.

Boundary decision for this stage: When cited_chunk_ids is an empty tuple, citation_count is 0, and precision must return 1.0 rather than raising an exception, because having no citations does not equal fabricating evidence.