The improvement loop

The first live link hid a bad turn

In one mini-app production case study, a live URL appeared in about 40 seconds. That looked like success. The same turn continued for 416 seconds, made 24 tool calls, shipped four times, accumulated roughly 565,000 context tokens, then ended on a provider error without sending a final reply. Manual testing found that repeated checks had exhausted the app's space quota and left the page dead on first load.

Case-study note, 2026-09-20: these figures describe one trial run on 2026-09-19 and a separate content-free 30-day report from the same deployment. The trial numbers were manually reconstructed from one turn; the distribution came from one deployment and workload mix. They explain this dated single-deployment case, not a universal baseline.

The decision is to treat production output as the start of evaluation, not the end. Record the cost of reaching the outcome, inspect what failed after the nominal success, fix one measured source of waste, deploy, and run the same kind of trial again.

An agentic system improves when its loop can disprove its own design assumptions.

Concept map

production trial
  -> measure outcome and cost units
  -> classify hard errors and recurring issues
  -> choose largest measured pain
  -> design smallest systemic change
  -> validate locally
  -> deploy through normal path
  -> reload the actual client
  -> repeat comparable trial
  -> keep, revise, or remove the change

The loop needs a living record. Without one, every new engineer rediscovers old failures and attractive ideas return without evidence.

Define cost units before the trial

Money alone is not enough. Agent systems spend several resources, and each points to a different fix.

Turns to first usable result measure how often the user must continue the conversation before receiving value.

Tool calls measure orchestration overhead. A turn with four calls and a turn with thirty calls may have similar token cost but very different latency and failure exposure.

Generated lines measure how much custom code the model must produce and maintain. Prebuilt design assets, SDKs, pinned libraries, and generated workers can be treated as free for this unit because they do not vary per app.

Returned lines measure context load from file reads and tool results. A follow-up edit that reads 2,313 lines to change one control is paying for missing state summaries.

Latency needs a distribution. Median shows normal experience. The ninetieth percentile finds the long turns that dominate frustration.

Hard errors block a usable result. Advisory issues describe defects or risks that can ship with a live link and be fixed in the same turn.

Recurring issue classes group structural failures such as exact-match edit misses, unknown design entries, quota exhaustion, repeated reads, or target-size warnings.

Final-answer delivery is a separate outcome. A published artifact with no assistant reply is not a complete turn.

Write a trial record that can be compared

Record the date, exact request, model chain, release revision, and environment. Then capture the units above.

trial:
  request_class: shared-list
  model_role: build
  turns_to_first_link: 1
  tool_calls: 24
  generated_lines: 150
  returned_lines: 4200
  latency_seconds: 416
  hard_errors: 0
  issues_by_class:
    design: 4
    exact_match_edit: 1
    space_quota: 1
  ships: 4
  final_answer_delivered: false

The request text may remain in a controlled engineering log when policy allows, but aggregate reports should use a request class rather than transcript content. Error analytics from the previous chapter can supply recurring structural classes without copying prompts.

One trial is a case study. A turn report over a week supplies distributions. Use both. The case study explains mechanism, while the report shows prevalence.

Pick the largest measured pain

Prefer removing a turn, a hundred generated lines, or a recurring error class over adding another feature.

In the live-link failure, the first fix was not a new visual capability. Check-created sync spaces became ephemeral and uncounted. The model-facing ship result became compact. Guidance told the model to reply once published and treat warnings as optional polish. A fallback message exposed a live link when the provider failed after publishing.

Each change answered an observed failure:

• quota exhaustion came from check infrastructure consuming user quota
• repeated polishing came from warnings that looked mandatory
• context growth came from verbose tool results
• missing final delivery came from coupling the link to provider completion

This is stronger than a general instruction to "be more efficient."

Move recurring work out of generated code

Production trials often show the same lines being generated again. That is a platform gap.

If every app writes its own list reconciliation, navigation state, offline worker, and typography rules, the model spends lines on infrastructure rather than the requested behavior. Move stable patterns into a small runtime, design library, SDK, template, or deterministic generator.

Track asset budgets too. Prebuilt code is free per app but not free to download or maintain. A runtime that saves 150 generated lines while adding a megabyte of client JavaScript may be the wrong trade.

Reject large general-purpose dependencies when a narrow first-party capability solves the repeated issue. In one synchronization comparison, heavy CRDT libraries were declined because current apps needed per-key last-write-wins behavior, not collaborative rich-text editing.

Tests and production answer different questions

Local checks prove type safety, unit behavior, integration contracts, production builds, and layout at selected viewports. They should run before deployment.

Production trials reveal provider failure patterns, service-worker update behavior, real quota interaction, tool-loop stopping behavior, deployment latency, and whether the user receives the result. These cannot all be inferred from green tests.

Reload the deployed client before judging it. Auto-updating service workers may keep an old bundle in an open tab until reload or relaunch. Testing stale code can create a false regression or a false fix.

Checks should report advisory issues without blocking the agent unless the first paint is blank, non-200, throws an uncaught exception, or violates another hard correctness boundary. A gate that blocks on every design warning increases turns and may prevent the user from seeing a usable draft.

Terminology

• Production trial is a real end-to-end request against the deployed system.
• Cost unit is a count tied to work, such as turns, calls, lines, tokens, or time.
• Hard error prevents a usable result.
• Advisory issue can ship and remain visible for repair.
• Issue class is a stable structural category used across trials.
• Baseline is the measured state before a change.
• Systemic fix removes repeated work or failure from future turns.
• Living record stores current limits, trial numbers, decisions, and backlog.

Decisions and rejected alternatives

The first decision is to measure before changing. Otherwise the backlog follows whichever failure is easiest to remember.

The second is to optimize for user-visible completion, not internal activity. More calls, ships, and generated code are costs unless they improve the outcome.

The third is to ship checks as reports for advisory findings. Hard blockers stay narrow and deterministic.

The fourth is to preserve failed and surprising trials in the iteration log. Deleting embarrassing evidence guarantees repeated mistakes.

Rejected alternatives include model self-ratings, benchmark-only evaluation, and feature-count roadmaps. Self-ratings do not measure live links or dead pages. Benchmarks miss tool contracts and deployment behavior. Feature counts reward expansion even when the current path takes thirty calls.

Also reject average-only reporting. In the production case study cited above, a median of 46 seconds coexisted with a ninetieth percentile of 411 seconds. The tail is where retries, loops, and provider failures accumulate.

Failure modes

• The team records success at publish time and misses the absent final reply.
• Generated lines fall while runtime download size grows without limit.
• Advisory issues become hard gates and add turns.
• A local browser test consumes production-like user quota.
• Trials change request, model, and code at once, preventing comparison.
• The backlog ranks novelty above measured pain.
• Recurring errors are counted by raw message text instead of structural class.
• Averages hide long-tail turns.
• Service-worker staleness invalidates deployment checks.
• A fix moves cost from model output into unreadably large prompt guidance.
• The iteration log records changes but not numbers.
• A failed experiment remains because removal feels like lost work.

Field checklist

• Run a deployed end-to-end trial before choosing the next improvement.
• Record turns to result, tool calls, generated and returned lines.
• Record median and tail latency, not only an average.
• Separate hard errors, advisory issues, and final-answer delivery.
• Group recurring problems by structural issue class.
• Keep model, request class, release, and environment with each trial.
• Choose one large measured pain per iteration.
• Move repeated generated work into bounded shared assets.
• Run local validation, deploy normally, and reload the client.
• Repeat a comparable trial after the change.
• Keep changes only when measurements or clear failure removal support them.
• Update the living record with surprises and the next few candidates.

The preceding chapter, Error analytics without transcripts, supplies privacy-bounded issue evidence for this loop.

This closes the book with an operating rule. Build the smallest complete path, observe it in production, count what it costs, and remove the repeated failure from the next run. The record of those runs is what lets the system improve without pretending that more autonomy is automatically better.