Model Mechanics & Multimodal Input

Start with a concrete problem

Why can’t character count replace token count?

When building a multimodal input pipeline, we need to estimate how many tokens a list of messages will consume so that we can reserve budget for system instructions and output.

If we directly use character length to estimate, we would count “hello model” as 11 tokens, while an actual tokenizer might produce only 2 tokens.

This confusion leads to severely distorted budgets: messages with many short words are overestimated, while messages with long words but few characters are underestimated.

In this stage, you must implement the estimate_tokens function, which must call the supplied encoder to obtain the token sequence for each message, rather than relying on character count.

Make a prediction

Suppose you have a message list ["hello model", "short"], using whitespace_encoder and a per-message overhead of 2. If estimate_tokens incorrectly uses character length instead of the encoder, what would the result be?

• 7 (correct use of encoder)
• 20 (incorrect use of character length)
• 5 (only counting words)
• Cannot be determined

Reasoning guide: The correct answer is 20. Character length calculation is len("hello model") + len("short") + 2*2 = 11 + 5 + 4 = 20. When using the encoder correctly, whitespace_encoder splits each message by spaces, so “hello model” yields 2 tokens and “short” yields 1 token; adding the per-message overhead of 2 gives 2+2+1+2=7.

What it is

What token estimation is

Token estimation is a function that takes a list of messages, an encoder, and a per-message overhead, and returns the total token count.

The encoder is responsible for converting text into a list of token IDs; the estimation function only cares about the length of that list, not the specific ID values.

The per-message overhead simulates the extra token consumption from message wrapping formats such as role markers and separators.

What it is not

What token estimation is not

Token estimation is not character counting: character count ignores tokenization rules, for example one word may correspond to multiple tokens, or multiple words may merge into one token.

It is also not an exact model invocation cost calculation: real models have additional overhead like attention caching and batching; here we only do a linear estimate.

It does not choose the encoder: the encoder is passed in by the caller, and the estimation function must remain replaceable.

How it relates to neighboring concepts

Relationship with other concepts

The encoder is a dependency of token estimation: without an encoder, the estimation function cannot know how text is split.

The per-message overhead is an independent additive term: it does not participate in encoding but affects the total budget.

Negative overhead validation is a boundary condition: if the overhead is negative, the total token count could be underestimated, so it must be rejected.

Boundary decision for this stage: When per_message_overhead is negative, the estimated result may be less than the actual encoded token count, so the function must raise a ValueError containing the string “non-negative”.

Start with a concrete problem

Optional material squeezes out required instructions in a limited window

You are implementing the context selection function select_context for the multimodal input module. The current code treats the entire window as freely allocatable capacity, filling all items by priority from high to low without distinguishing required from optional items and without reserving space for model output.

In the test test_output_reserve_and_required_context_are_protected, the function receives system (60 tokens, required) and notes (30 tokens, optional, higher priority), with a window of 100 and output reserve of 20. The correct result should return only ["system"] because after subtracting the output reserve, the available budget is 80; the required system uses 60, leaving 20, which is insufficient for notes.

However, the current implementation first places the higher-priority notes, then system, returning ['notes', 'system'], causing the required system instruction to be displaced by optional notes and failing to protect output space.

Another test, test_impossible_required_context_is_rejected, requires that when required items alone exceed the budget, a ValueError is raised rather than silently returning a partial result.

Make a prediction

In select_context, if the window is 100, output reserve is 20, and there is a required item of 90 tokens, what should the function do?

• Return that required item, ignoring the output reserve
• Raise an exception because the required item exceeds the available budget
• Return an empty list because it cannot be satisfied
• First place other optional items, then try to place the required item

Reasoning guide: The correct approach is to raise an exception. Required items are non-droppable content such as system instructions or policies; if they exceed the budget after subtracting the output reserve, any partial result would break system constraints, so the request must be rejected immediately.

What it is

Context budget allocation model

Context budget allocation is a two-phase process: first subtract the output reserve from the total window to obtain the budget available for input content; then unconditionally include all required items, and finally fill optional items by priority from high to low until remaining capacity is insufficient.

Required items are non-droppable constraints such as system prompts and policy documents; they must all enter the context, otherwise the model’s behavior may violate rules. Optional items are retrieved reference materials, conversation history, etc., which can be selected based on priority.

What it is not

Not simple priority sorting

Context budget allocation is not simply sorting all items by priority and placing them in order, because that would allow high-priority optional items to displace low-priority required items.

It is also not using the entire window for input content; space must be reserved for model output, otherwise the model may not have enough tokens to generate a response.

How it relates to neighboring concepts

Relationship with token estimation and context concept

This stage depends on the token estimation function estimate_tokens from Stage 01 to obtain token counts for each item, but select_context itself only receives precomputed token counts.

Context budget allocation is a concrete implementation of the context concept: it determines which information enters the current model call, embodying the boundary that context is a selected set of information.

Boundary decision for this stage: When the total tokens of required items exceed the budget after subtracting the output reserve, an exception must be raised to reject the request, rather than returning a partial result or silently dropping required items.

Start with a concrete problem

Why can’t dot product be used directly as a measure of vector similarity?

In the previous stage, you implemented forge/tokens.py and forge/context_budget.py; now you need to add forge/embeddings.py to compute vector similarity.

If you directly return the dot product of two vectors, the similarity will be affected by vector length: for example, [1,0] and [10,0] have the same direction, but the dot product is 10, not the expected 1.

The test test_cosine_similarity_is_scale_invariant requires cosine_similarity([1,0],[10,0]) to return 1.0, and test_dimension_mismatch_is_rejected requires a ValueError when dimensions mismatch.

You need to implement cosine similarity: first check that dimensions match and vectors are non-zero, then compute the dot product and divide by the product of the two norms to eliminate length influence.

Make a prediction

Before implementing cosine_similarity, predict: if the function only returns the dot product of two vectors, what result will test_cosine_similarity_is_scale_invariant get?

• Returns 1.0, test passes
• Returns 10, test fails
• Raises ValueError
• Returns 0

Reasoning guide: The dot product [1,0]·[10,0] = 1*10 + 0*0 = 10, but the test expects 1.0, so the test will fail. This shows that the dot product is affected by vector length and cannot be used directly as similarity.

What it is

What cosine similarity is

Cosine similarity eliminates the influence of vector length through normalization, measuring only directional agreement: it equals the cosine of the angle between two vectors, ranging from -1 to 1.

Calculation steps: first check that the two vectors have the same dimension and are non-zero, then compute the dot product, and divide by the product of the two vector norms.

For vectors with the same direction, regardless of length, cosine similarity is 1; for orthogonal vectors, similarity is 0; for opposite directions, similarity is -1.

What it is not

What cosine similarity is not

Cosine similarity is not the dot product: the dot product is affected by vector length, while cosine similarity removes this influence by dividing by norms.

Cosine similarity is not a distance metric: it only focuses on direction, not the absolute distance between vectors.

Cosine similarity cannot handle zero vectors or dimension-mismatched vectors; these cases are undefined and must be rejected.

How it relates to neighboring concepts

Relationships with other concepts

Relationship with dot product: cosine similarity = dot product / (norm1 * norm2), so the dot product is the numerator part of cosine similarity.

Relationship with norms: norms are used for normalization, ensuring similarity is not affected by vector scaling.

Relationship with embeddings: embeddings are vector representations; cosine similarity is used to compare the directional similarity of two embeddings, but embeddings themselves are not answers.

Boundary decision for this stage: When vector dimensions mismatch or either vector is a zero vector, cosine similarity is undefined and must raise ValueError; otherwise compute the dot product and divide by the product of norms.

Start with a concrete problem

Why must invalid UTF-8 be rejected instead of silently discarded?

In the office research scenario, you collect documents from multiple sources: web pages, spreadsheets, and PDF exports. These files may contain corrupted byte sequences, such as an isolated 0xFF byte. If the parser silently discards these bytes, the document content changes without notice, causing later citations or analysis to be based on incomplete evidence.

The file forge/text_input.py does not exist yet, so the tests test_invalid_utf8_is_rejected_not_silently_dropped and test_line_endings_are_normalized fail with an import error. You need to implement a parse_text function that strictly validates UTF-8 encoding and raises a ValueError when encountering invalid bytes, rather than ignoring the error and continuing.

Different operating systems use different line endings: Windows uses CRLF (\r\n), old Mac uses CR (\r), and Unix uses LF (\n). For consistent downstream processing, parse_text must normalize all line endings to LF.

Make a prediction

What happens if parse_text decodes a document containing invalid bytes using errors='ignore'?

• Invalid bytes are discarded, the document is accepted, but content may be incomplete
• A UnicodeDecodeError is raised, and the document is rejected
• Invalid bytes are replaced with U+FFFD, and the document is accepted
• The program crashes and cannot continue

Reasoning guide: Choose the first option: errors='ignore' silently discards invalid bytes, causing the document to be accepted but with incomplete content. This is exactly the behavior that the test test_invalid_utf8_is_rejected_not_silently_dropped aims to prevent.

What it is

What text input normalization is

Text input normalization is the process of converting external byte streams into a unified internal representation, including strict UTF-8 decoding, line ending normalization, size limit checking, and preservation of source metadata.

Strict decoding means any invalid byte causes an explicit error, ensuring the input content exactly matches the original bytes with no information loss.

What it is not

What text input normalization is not

It is not a lenient cleaning process that silently discards or replaces invalid bytes, because that would compromise evidence integrity.

It is not simple string replacement; it must clearly distinguish encoding errors from line ending differences and adopt a rejection policy for encoding errors.

How it relates to neighboring concepts

Relationship with other components

The TextDocument object returned by parse_text contains source and text fields. Subsequent token counting, context budgeting, and embedding generation all depend on this normalized text.

If the text is silently modified at the entry point, all downstream processing will be based on incorrect data, so strict validation is the foundation for the reliability of the entire pipeline.

Boundary decision for this stage: When encountering invalid UTF-8 bytes, you must raise a ValueError and reject the input rather than attempt to repair or ignore it, because any byte loss may change the document’s meaning.

Start with a concrete problem

Why can’t we trust file extensions?

You are adding image support to the multimodal input module. The test test_extension_cannot_override_file_signature requires that even if the filename is fake.png, as long as the content is not a real PNG signature, validate_image must raise a ValueError containing signature.

Currently forge/image_input.py does not exist, so running python -m pytest -q tests/test_stage.py fails during collection with ModuleNotFoundError: No module named 'forge.image_input'.

You need to implement validate_image, which takes a filename, raw bytes, width, height, and an optional maximum pixel count, and returns an ImageAsset. The key decision is that the media type is determined by the file content signature, not by the extension.

Make a prediction

If validate_image only determines the media type from the filename suffix, what happens when fake.png and plain text bytes are passed?

• The function returns ImageAsset('image/png', ...) and the test fails.
• The function raises ValueError and the test passes.
• The function returns ImageAsset('image/jpeg', ...).
• The function crashes because it cannot parse text.

Reasoning guide: The correct answer is the first option. Determining the type only from the extension trusts user-controllable metadata, causing a fake .png file to be accepted as an image, and the test test_extension_cannot_override_file_signature fails.

What it is

What file signature validation is

A file signature is a fixed byte sequence at the beginning of a file, such as PNG’s b"\x89PNG\r\n\x1a\n" and JPEG’s b"\xff\xd8\xff".

validate_image checks these magic numbers using payload.startswith(PNG) or payload.startswith(JPEG) to reliably identify the format.

What it is not

What file signature validation is not

It is not a guess based on the filename suffix. The extension is user-controllable metadata and cannot be used as the basis for format determination.

It is also not full image decoding or content validation; it only checks the magic number at the file header and does not guarantee that the image data is complete or renderable.

How it relates to neighboring concepts

Relationship between signature, pixel limit, and ImageAsset

The signature check determines the media type, and the pixel limit check prevents resource abuse; together they ensure the validity of ImageAsset.

ImageAsset is an immutable dataclass containing media_type, width, and height, and can only be created after validation passes.

Boundary decision for this stage: When the file signature does not match any known format, a ValueError must be raised; when width or height is non-positive or the product exceeds max_pixels, a ValueError must also be raised.

Start with a concrete problem

Why must row locators be preserved when parsing CSV?

In the office research scenario, you need to hand a CSV table to the model, but the model cannot read the raw file directly; you must first convert the CSV into text chunks with source locators.

If parsing extracts only text content and discards row numbers, later citation or debugging cannot trace back to the specific row in the original table, breaking the evidence chain.

Currently, forge/documents.py does not exist, so the test test_csv_rows_keep_source_locators fails with an import error, which is the starting point for this stage.

Make a prediction

When implementing parse_document, which approach best satisfies the test requirements for text/csv input?

• Return the entire CSV text as one chunk with locator “document”
• Use csv.DictReader to parse row by row and generate row:i locators
• Parse CSV as JSON and raise an exception on failure
• Return only the first row and ignore the rest

Reasoning guide: The test expects two chunks with locators row:1 and row:2, so you must parse row by row and generate row-number locators.

What it is

Source locators are part of the evidence chain

A source locator records the precise position of each text chunk in the original document, such as a CSV row number or the JSON root path $.

It allows later processing stages (like retrieval or generation) to cite specific sources, enabling verifiable references and debugging.

Different document formats require different locator strategies: CSV locates by row, JSON by path, and Markdown typically by the whole document.

What it is not

Source locators are not optional metadata

A source locator is not a simple text label; it must uniquely identify the chunk’s position in the original document.

It is not a uniform identifier for the entire document; for example, all CSV rows cannot use "document" as the locator.

It is not part of the content itself but separate metadata used for tracing and citation.

How it relates to neighboring concepts

Relationship among parser, chunks, and locators

parse_document dispatches to different parsing logic based on media type, and each parsing logic is responsible for generating the corresponding chunks and locators.

Each Chunk object contains source (source file name), locator (locator), and text (text content), which together form a traceable unit.

Tests verify parsing correctness by checking the list of locators, so the locator generation rules must match the test expectations.

Boundary decision for this stage: When the media type is text/csv, you must use csv.DictReader to parse row by row and generate a row:i locator for each row; when the media type is unknown, you must raise a ValueError exception.

Start with a concrete problem

Why must a text-only model reject image input?

In the previous stage, you implemented a document parser that can extract text from images. Now you need to add a unified input entry point route_input that decides whether to accept an input based on media type and model capabilities.

The current test test_text_only_model_rejects_image_with_reason requires that when the model supports only text modality, passing image/png must return an unsupported status and the reason must contain image.

A common mistake is to route only by media type, assuming that if the parser can handle images, the model can also accept them. This leads to a text-only model receiving image input and causing silent errors.

You need to implement forge/intake.py, defining the IntakeDecision dataclass and the route_input function, checking both parser capability and model modality, and returning explainable rejections for unsupported combinations.

Make a prediction

When implementing route_input, what happens if you only check the media type and ignore model_modalities?

• Images will be routed to image_adapter even if the model supports only text, causing the test to fail.
• Images will be correctly rejected because the parser does not support images.
• Text inputs will also be rejected because the model modality check will misjudge.
• The system will crash because it cannot determine the adapter.

Reasoning guide: The correct answer is the first option. Checking only the media type will incorrectly route images to image_adapter, while the test expects a text-only model to reject images. You must also check model capabilities.

What it is

Input routing is a dual check

Input routing is a decision function that takes a media type and a set of model-supported modalities, and returns an IntakeDecision object containing status, adapter, and rejection reason.

Routing must verify two things simultaneously: whether the parser can handle the media type, and whether the model has the input capability for that modality. Only when both are satisfied should it return ready.

What it is not

Not a simple media type mapping

Input routing is not merely mapping a media type string to an adapter name. If model capabilities are ignored, unsupported inputs will be passed to the model, causing silent errors.

It is also not a check of parser capability alone. Even if the parser can extract text from an image, a text-only model cannot directly process the raw pixel data of an image.

How it relates to neighboring concepts

Relationship among parser, model, and adapter

The parser is responsible for converting raw input into a format the model can understand, such as extracting text from an image. Model capabilities define the modalities the model can receive, such as text or image.

The adapter is the concrete implementation that connects input and model, for example text_adapter handles text input. The routing function determines the modality from the media type, then checks whether that modality is in the model capability set, and finally selects the corresponding adapter.

Boundary decision for this stage: When the media type is unrecognized or the model does not support the modality, you must return an unsupported status with a clear reason, rather than returning ready or raising an exception.