Context is a budget

A follow-up edit changed one button label. The turn made eight tool calls, reread five files, and sent the same large file contents to the provider several times. The answer was correct. It also took seven minutes and cost more than the first build.

The model did not need more intelligence. It needed a better information budget.

Allocate context by purpose

total request budget
|
+-- stable instructions
+-- user and durable memory
+-- retrieved facts
+-- conversation summary
+-- recent messages
+-- current claimed input
+-- recent tool exchanges
+-- work metadata and checkpoint

Every section competes with the others. An unbounded memory document can push out the current request. Repeated tool output can erase the conversation. A giant system prompt consumes the budget before work begins.

Terminology

• Context window: The provider's maximum input and output token capacity.
• Prompt budget: The smaller limit the application chooses for one request.
• Retrieval: Selecting relevant stored material for the current turn.
• Compaction: Replacing older detailed exchanges with bounded representations.
• Stable prefix: The unchanged beginning of a request that may benefit from provider caching.
• Outline: Deterministic structure extracted from a file without returning all content.
• Checkpoint: Durable recent work metadata used for resume.

Budget sections explicitly

Character counts are crude but deterministic and cheap. Tokens vary by model, language, and content. A system can start with character budgets, then measure actual provider usage.

type ContextBudget = {
  identity: number;
  memory: number;
  recall: number;
  summary: number;
  recentHistory: number;
  workState: number;
};

const budget: ContextBudget = {
  identity: 2_000,
  memory: 6_000,
  recall: 6_000,
  summary: 4_000,
  recentHistory: 20_000,
  workState: 12_000,
};

These are application limits, not provider maxima. Leave room for tool schemas, current input, tool results, and output.

Order sections by authority and temporal importance. Stable system instructions come first. The exact claimed user input should remain intact near the end. Truncate retrieved and historical data before cutting the current request.

Memory is a set of jobs

"Memory" often bundles several different needs:

• persona and behavior rules;
• user preferences;
• durable facts;
• daily capture notes;
• conversation summaries;
• task and goal state.

Store them separately. Each has a different writer, retention policy, and prompt budget. A single ever-growing document becomes hard to edit and impossible to allocate.

Keep raw durable documents outside every request. Select only the sections needed for the turn. Revision-check writes so an automated consolidation cannot overwrite a user's edit.

Private conversations should neither enter retrieval nor feed capture. If private mode may read existing user preferences, make that rule explicit.

Retrieval before embeddings

For one user or a small workspace, Postgres full-text search plus recency weighting can be good enough. It uses existing operations and supports deletion with ordinary rows.

A practical score can combine text rank and age:

SELECT id, content,
       ts_rank(search, websearch_to_tsquery('english', $1))
       * exp(-extract(epoch FROM (now() - created_at)) / 2592000.0) AS score
FROM memory_chunks
WHERE search @@ websearch_to_tsquery('english', $1)
ORDER BY score DESC
LIMIT 8;

This example applies roughly a 30-day decay. The right half-life depends on the data. Preferences may stay relevant for years, while project status goes stale in days. Separate classes if one decay cannot fit both.

Add embeddings when measured recall quality, synonyms, or multilingual search justify another index and deletion path. They are useful, but they are not a prerequisite for durable memory.

Compact tool conversations carefully

Long tool loops create two costs. The provider rereads old output, and the request may lose cache benefits if earlier bytes keep changing.

Keep the newest few tool exchanges verbatim. Compact older call and result pairs together. Use each tool's compact input and output when available. Otherwise keep a deterministic excerpt with original length and a content hash.

Do not compact on every small growth. Rewriting the prefix too often can reduce provider prompt-cache reuse. Apply compaction only when a batch removes a meaningful amount, then preserve the new prefix until the next threshold.

Track:

• prompt tokens;
• cached prompt tokens;
• completion tokens;
• characters removed by compaction;
• cost per iteration;
• number and size of tool results.

Without those measurements, compaction can sound efficient while increasing cost.

Give follow-up work structural context

After a multi-file build, the next turn should not need to rediscover the project. Put deterministic metadata into context:

• file paths and line counts;
• function, class, heading, selector, and element ID outlines;
• current and published revisions;
• files touched since the previous turn;
• added and removed line counts;
• last verification result.

For a file over a chosen limit, a whole-file read can return the first section and the outline of the rest. Range reads remain available. The point is not to prevent reading. It is to make the cheapest correct read obvious.

Revision history also enables diff, history, and undo tools. Those tools are more precise than asking the model to infer prior state from conversation prose.

Decisions and trade-offs

Prefer deterministic compaction to model summaries inside a turn. A model summary can omit the one error line needed later and costs another call. Compact forms from tool contracts retain known fields.

Use a cheap model for asynchronous summaries and capture. These tasks are bounded and can retry independently. Do not make the main user turn pay their latency.

Keep recent results verbatim. Aggressive compaction saves tokens but hurts tool correction. The newest exchanges contain the active failure and should remain exact.

Pause at a hard budget. Silently dropping more context can make the model overwrite correct work. Save a checkpoint and ask for resume.

Do not include entire artifacts by default. Let the model call a read tool when it needs the body. The prompt can include revision, title, and size.

Failure modes

A summary can advance its cursor before the summary write commits. The omitted messages then disappear from future context. Update summary text and cursor together.

Retrieved content can contain prompt injection. Fence it, label it as untrusted data, and keep tool policy outside the prompt.

Compaction can break provider protocol by removing a tool call while retaining its result. Treat an exchange as an indivisible unit.

A cached tool result can survive a revision change. Include revision identity in range-read and verification caches.

Context sections can be truncated in the wrong order. Add tests that fill each budget and assert the current input and critical policy remain intact.

Provider usage may omit cost. Maintain local price tables for known models, version them, and mark estimates as estimates.

Context checklist

• The application sets a prompt budget below the provider maximum.
• Identity, memory, recall, summary, history, and work state have separate limits.
• Current claimed input is never silently truncated.
• Durable documents remain outside the prompt until selected.
• Retrieval has deletion, privacy, and recency rules.
• Recent tool exchanges stay verbatim.
• Older tool calls and results compact together.
• Compaction thresholds consider prompt-cache stability.
• Large files expose outlines and range reads.
• Follow-up turns receive revision and changed-file metadata.
• Hard budget stops save a bounded checkpoint.
• Tokens, cache hits, removed characters, latency, and cost are measured.

The checkpoints in Long-running work is a state machine supply bounded resume data. Once context has a price, routing becomes an economic and safety decision. The next chapter assigns models and tools by job. Continue with Route work by risk.