Model routing, cost, and quota

One model, several unused quotas

In one production case study, every sampled turn used the same model even though the provider granted separate monthly quotas per model. Follow-up app edits averaged 5.38 file reads and 2,313 returned lines. The median turn made 8.5 tool calls, and the slowest tenth took at least 411 seconds. That system was expensive in both context and time while backup capacity sat idle.

Case-study note, 2026-09-20: these figures come from a content-free 30-day production report for one deployment, provider setup, workload mix, and reporting implementation. They describe that dated single-deployment case study, not a universal baseline or model benchmark.

The decision is to route by work role, configure an ordered model chain for each role, and move to the next model when the current one reports quota exhaustion. Quality comparisons must use turn outcomes, not provider reputation.

Routing is resource allocation with a safe default.

Concept map

pending chat or nudge
  -> cheap route call
       -> chat role
       -> build role plus prefetched skills
       -> failure or malformed result -> chat role, no skills

selected role
  -> ordered model chain
       -> preferred model
       -> quota response -> next model, same transcript
       -> transient server error -> bounded retry
       -> exhausted chain -> terminal failure

usage
  -> local price catalogue
  -> per-turn cost, tokens, latency, calls, fallbacks
  -> production comparison

Intent turns can skip classification and use the chat chain. Deterministic verification should use no model at all.

Assign roles before models

Start with a small role vocabulary:

• Route classifies a pending request and selects a few relevant skills.
• Chat handles conversation, research, memory, and follow-through.
• Build handles mini-app creation and editing.
• Cheap handles titles, capture, summaries, consolidation, suggestions, triage, and bounded background jobs.

Roles describe work. Model ids remain configuration. This lets an operator change a chain without rewriting every call site.

type Role = "route" | "chat" | "build" | "cheap";

const chains: Record<Role, string[]> = {
  route: ["fast-small"],
  cheap: ["fast-small"],
  chat: ["strong-code", "general-large"],
  build: ["strong-code", "fast-code", "general-large"],
};

function modelChain(role: Role, env: Record<string, string | undefined>) {
  const override = env[`MODEL_${role.toUpperCase()}`];
  return override
    ? override.split(",").map(value => value.trim()).filter(Boolean)
    : chains[role];
}

Keep the catalogue allowlisted. Endpoint families differ, and a model available through a responses endpoint may not work with a chat-completions loop.

Make the route call disposable

The route call should receive only the pending text, the thread's known apps, and a compact skill index. Ask for bounded JSON such as \{role: "chat" | "build", skills: []\} and cap output near 300 tokens.

Its failure cannot block the user's turn. If the provider times out, returns malformed JSON, or names an unknown role, continue as chat with no prefetched skills. The main turn still has ordinary tools and can recover.

Intent runs should bypass routing. Their persisted instruction already identifies them as conversational follow-through, and adding a classifier to every reminder creates cost with little choice value.

Skill prefetch can save tool calls when routing succeeds. Put selected skill bodies into the system context and list which skills are already present. The model should call the skill tool only for material not in context.

Separate quota from transient failure

A quota response means the current model is unavailable for resource reasons. Move immediately to the next model in the role's chain. Common signals include HTTP 429 and provider-specific 402 or 403 responses with quota semantics.

Preserve the full compacted transcript, tool history, logical turn id, and session id. The fallback model continues the same turn. Record a structural provider_fallback event with source model, destination model, role, and iteration.

A timeout, HTTP 408, or server-side 5xx may be transient. Retry the same model a small number of times before replaying the turn on the next model. Treating every 500 as quota can spread a provider outage across the entire chain and duplicate tool work.

Do not fallback after an ambiguous consequential tool result. Tool calls need idempotency so replay is safe. When safety is uncertain, fail visibly instead of guessing.

If the chain exhausts, preserve the last attempted model in the terminal error. That detail matters during incident review.

Price what the provider does not

Some subscription APIs return tokens but no monetary cost. Keep a versioned local catalogue with prompt, cached-prompt, and completion rates. Compute:

fresh prompt = max(prompt tokens - cached tokens, 0)
cost = fresh prompt * prompt rate
     + cached tokens * cached rate
     + completion tokens * completion rate

Price by the model that actually produced each response, not the first model selected. Unknown models should report unpriced usage rather than silently inherit another model's rate.

Cost controls still matter under a subscription. Soft thresholds can compact older tool results. Hard thresholds can pause before another tool call while preserving completed draft work for a later resume. Track tokens, removed transcript characters, and cost by provider, model, and role.

Evaluate with outcomes

The useful question is not which model has the best benchmark score. It is which chain completes this role with fewer calls, fewer failed edits, fewer ship issues, lower latency, and acceptable cost.

Collect content-free turn totals:

• iterations and tool calls
• prompt, cached, and completion tokens
• file reads and returned lines
• fallback count
• route latency and total latency
• edit failure rate
• hard errors and advisory issues
• whether the user received the final answer

Take a baseline before changing the chain. Run one configuration for a fixed period, then compare the same report. For build work, a credible trial might swap the first two models for a week and compare calls per turn, edit failures, issues per ship, fallbacks, and latency.

Do not declare a winner from one attractive app. Production distributions expose the long turns that dominate cost.

Terminology

• Role is a stable category of work.
• Chain is an ordered list of eligible models for a role.
• Route call is the bounded pre-call that selects chat or build work.
• Prefetch inserts selected skill text before the main turn.
• Quota fallback switches models while preserving the logical turn.
• Transient retry repeats the same model after a temporary transport or server failure.
• Local pricing computes cost from a maintained catalogue.
• Turn economy measures the calls, context, time, and failures required for an outcome.

Decisions and rejected alternatives

The first decision is role-based configuration instead of one global model variable. A cheap capture task and a tool-heavy build have different needs.

The second is fail-open routing to chat. A classifier should improve allocation, never become a new availability dependency.

The third is same-turn quota fallback. Asking the user to retry wastes completed reasoning and leaves per-model quota unused.

The fourth is deterministic design and verification. Static checks, browser tests, layout audits, and lint do not need model judgment.

Rejected alternatives include random model selection, routing every tool call separately, and switching models after any error. Random selection prevents useful attribution. Per-call routing fragments context. Error-blind switching confuses outages, bad requests, and quotas.

Failure modes

• A malformed route answer blocks the main reply.
• Intent turns spend a route call without a meaningful choice.
• A quota fallback starts a new logical turn.
• Tool side effects repeat after model fallback.
• Every 5xx causes immediate chain switching.
• Local pricing uses the preferred model after fallback.
• Cached tokens are charged again at the full prompt rate.
• An unknown model receives a guessed price.
• Metrics omit role, hiding expensive background work.
• Evaluation compares averages only and misses severe tail latency.
• Quality is judged from screenshots without final-answer or error rates.

Field checklist

• Define route, chat, build, and cheap roles.
• Configure an ordered chain per role.
• Keep eligible models in an allowlisted catalogue.
• Fail route errors to chat with no prefetched skills.
• Skip routing for intent turns.
• Distinguish quota responses from transient failures.
• Preserve transcript, session, and logical turn across fallback.
• Make replayed tool calls idempotent.
• Price actual model usage locally.
• Label metrics by role and model.
• Baseline calls, reads, lines, latency, failures, and outcomes.
• Compare production distributions over a fixed trial window.

Routing data explains cost and capacity, but debugging recurring failures needs a separate record with a tighter privacy boundary. Continue with Error analytics without transcripts. The personalization inputs routed into these turns are covered in Personalization without a profile.