Observability for agents

The health page returns 200 while users wait fifteen minutes for replies. CPU is low, Postgres is reachable, and every container is running. The failure sits one level higher: pending outbox work is aging while model calls retry.

Agent systems need normal service telemetry and a second view of agent turns. Neither replaces the other.

Two linked views

service view
host -> containers -> edge -> database -> queues -> backups

turn view
queued -> routed -> model attempt -> tool calls -> fallback or retry -> reply

join keys
time, instance_id, provider, model role, outcome class

Do not use prompts or responses as join keys. Operational telemetry should remain useful without copying user content.

Terms

A metric is a numeric time series suited to rates, totals, gauges, and distributions.

A structured log is an event with named fields. It provides detail when a metric indicates a problem.

A trace links timed spans across components. It is optional for a small cell when metrics and logs answer current questions.

A logical turn is one user request across automatic retries.

A model attempt is one call to one model. A turn can contain several attempts after transient or quota failures.

A terminal event records how one physical run or retry attempt ended. The final outcome for a logical turn is derived from its ordered attempt events.

Begin with the machine and service

Collect CPU, load, memory, swap, root disk usage, disk free bytes, and network traffic. Container metrics should show CPU, memory, restarts, and last-seen time by Compose service. Caddy should expose request rate, status groups, and duration. Postgres needs connection count and retained WAL by replication slot.

Application metrics should answer:

• Is the watcher reachable, and can it query Postgres?
• How many outbox rows are pending, processing, or failed?
• How old is the oldest pending item?
• How many poller ticks fail?
• How many model calls are in flight?
• When did the last successful off-site backup finish?

Queue age is more useful than queue length for a private system. One pending turn for twenty minutes is a serious problem even though the count is only one.

Keep metrics endpoints on the Compose network. Grafana Alloy is the reference collector. It scrapes services, reads container logs, adds instance_id, and sends data outbound to Grafana Cloud. The vendor is replaceable. Prometheus and Loki on a separate host preserve the architecture. Running Grafana, Prometheus, and Loki beside Postgres on a four-gigabyte VM wastes memory and disappears when the host dies.

A small exposition may look like:

agent_outbox_oldest_seconds{instance_id="alpha"} 37
agent_logical_turns_total{instance_id="alpha",outcome="success"} 482
agent_backup_unixtime{instance_id="alpha"} 1790000123
agent_backup_offsite{instance_id="alpha"} 1

Cap labels. Never put conversation IDs, tool arguments, prompts, URLs, error messages, or arbitrary model output in metric labels.

Measure the logical turn

Record one terminal event for every physical run or retry attempt. Give automatic retries the same logical-turn ID, and retain a run ID and attempt number on each event. This preserves evidence about failed attempts without pretending that one user request was several turns.

Reporting must group those events by logical-turn ID. Count distinct logical-turn IDs for turn rates and distributions, and take the latest attempt's outcome as the logical turn's final outcome. Keep per-attempt rows for retry and failure analysis. Sum attempt-scoped tokens and cost across physical runs only when those fields are deltas; if an event carries cumulative totals, take the final value instead. Store bounded numeric totals such as:

• queue delay and total duration
• route decision duration
• model attempts and fallbacks
• loop iterations
• tool-call count
• prompt, cached, and completion tokens
• estimated provider cost
• final model role and terminal outcome

Keep model attempts as separate counters or events so quota switching and transient retries remain visible. Report both physical runs and distinct logical turns. A provider failure rate based only on final logical-turn outcomes hides expensive recoveries.

The role matters. A cheap routing call, a conversational response, and a code-building turn have different latency and cost expectations. Report them separately with a small fixed enumeration such as route, chat, build, and cheap.

Tool metrics need stable classifications. Count schema rejection, runtime exception, structured unsuccessful result, repeated identical call, timeout, and success. Tool names are usually bounded enough for a label if the registry is fixed. Raw arguments are not.

Cost can be provider-reported or calculated locally from token usage and a versioned price table. Mark it as estimated. Cached-token pricing, provider rounding, and free tiers make local numbers unsuitable for invoice reconciliation.

Logs explain, metrics alert

Use JSON logs with timestamp, level, message, instance ID, logical turn ID, poller, provider, model role, tool name, and error class where relevant. Exclude message text, prompt text, tool arguments, authorization headers, secret values, analytics labels, and request bodies.

Fingerprint repeated structural failures from stable fields. Grouping by an entire error string creates unbounded cardinality and may retain user data. Keep failure events for a bounded period, such as 30 days, and cap reporting queries by rows and time.

An agent-turn report should show recurring fingerprints, distinct logical turns affected, physical run count, retry rate, fallback rate, token totals, cost estimates, and high-percentile duration. Turn counts and outcome rates must use distinct logical-turn IDs, while retry diagnostics retain every run event. It should not become a covert transcript index.

Alerts that lead to action

Alert on host silence from the remote monitoring service. The dead VM cannot page anyone itself. Alert on disk above a conservative threshold, very low available memory, stale backups, unhealthy containers, watcher health failure, sustained Caddy 5xx, and old pending outbox work.

Agent alerts should avoid waking an operator for one provider hiccup. Useful conditions include a sustained terminal-failure ratio, repeated identical tool failure across distinct turns, a fallback spike, or token cost well above the recent role baseline. Each alert needs a runbook link and an instance label.

Dashboards should move from host to application to turn behavior. A practical sequence is host, containers, edge, watcher queue, model and tool behavior, Postgres, then backups. Logs filtered to the selected instance belong beside the relevant panels.

Decisions and rejected alternatives

Use metrics and logs before traces. On one host, failures are usually stuck work, full disks, provider errors, or restart loops. Add tracing only when cross-process timing remains ambiguous after those signals exist.

Do not instrument browser real-user monitoring by default for a private assistant. It adds visitor data and another script while server-side queue and edge metrics answer most operating questions.

Do not log prompts for convenience. Content logging changes the privacy boundary and makes routine incident access much more sensitive.

Do not expose Grafana on the application host. A managed remote backend can detect host silence and avoids another public administration service.

Accept that container collection through the Docker socket is host-privileged. A read-only bind does not make the Docker API read-only. Keep the collector image pinned, private, and narrowly configured.

Failure modes

• Metrics without instance_id make a cell fleet indistinguishable.
• Arbitrary exception text in labels overwhelms the time-series backend.
• Counting terminal event rows as logical turns inflates usage.
• Collapsing attempt events before retry analysis hides recoveries and outages.
• A health gauge stays green while queue age rises.
• Backup scripts write a local file but never report whether off-site upload succeeded.
• Logs rotate too quickly during a retry storm.
• A dashboard exists, but no alert checks that the host stopped sending data.

Field checklist

• Host, container, edge, database, queue, and backup signals exist.
• Remote monitoring alerts when the host goes silent.
• Metrics and logs carry instance_id.
• Oldest queue age is measured, not only queue length.
• One terminal event is retained per physical run or retry attempt.
• Turn metrics count distinct logical-turn IDs and derive the final outcome from the latest attempt.
• Attempts, fallbacks, roles, tokens, duration, and estimated cost are visible.
• Tool failures use bounded outcome classes and stable fingerprints.
• Prompts, replies, arguments, secrets, and analytics labels stay out of telemetry.
• Label values come from capped enumerations.
• Alerts link to actions and tolerate isolated transient failures.
• Retention and query bounds apply to agent failure events.

Previous: Chapter 37, "Static apps at the edge".

Next chapter

Chapter 39, "Backups, restores, and drills", tests whether the state behind those healthy graphs can actually be recovered.