Secrets that never enter Git

A deployment fails before containers start. The tempting fix is to paste the production environment file into a CI secret or print it during debugging. Those shortcuts make the next deployment easier and secret recovery harder to reason about.

This book's reference contract is simple. Git knows secret names. A provider owns secret values. The target host retrieves only the values it is allowed to use.

Encrypted secrets in Git, commonly managed with SOPS, are also a valid threat-model choice. Ciphertext can make reviewable infrastructure self-contained, but the design must still distribute decryption keys, scope who and which CI jobs may decrypt, rotate recipients, prevent plaintext artifacts, and recover when keys are lost.

This book recommends an external provider because its reference cells already have a narrow runtime retrieval path, not because encrypted Git storage is universally unsafe.

Secret path

external provider
  |
  +-- shared/production
  +-- instances/alpha
           |
      narrow broker
           |
   host bootstrap token
           |
   /run/.../runtime.env, mode 0600
           |
       Docker Compose
           |
   file removed after start

This is containment, not invisibility. A Docker administrator can inspect running container environments and mounted data.

Terms

A secret provider stores and controls values such as API keys, database passwords, and signing keys.

A secret broker is a narrow service that translates the provider's access model into a runtime retrieval endpoint for one cell.

A bootstrap secret is the credential a host must possess before it can retrieve the rest.

Materialization writes provider-owned values into the format a local tool needs, usually for a short time.

A recovery adapter reads secrets from an alternate source during migration or disaster recovery while preserving the same materialization contract.

Start with a schema

List required, optional, shared-capable, and instance-only variables. Validate names and relationships without printing values. For example, Grafana remote-write settings may require a complete URL, user, and token group. A VAPID configuration may require public key, private key, and subject together.

Reject unknown names. Otherwise a typo can look like a successful retrieval while the application silently uses a default. Reject duplicate keys, multiline values, NUL bytes, and syntax that the destination format cannot represent safely.

Keep public deployment configuration elsewhere. Hostnames, cell IDs, image digests, and runner labels do not need encryption. Mixing them with secrets makes review and rotation needlessly difficult.

Why a broker may be necessary

Cloudflare Secrets Store can bind values directly to Workers. A Hetzner VM has no Cloudflare workload identity that consumes those values. The reference design therefore deploys one Cloudflare Worker broker per cell. It binds shared and instance secret chunks, accepts one authenticated materialization request, and returns a merged environment document with instance values taking precedence.

This vendor choice is explicit. AWS Secrets Manager with an instance role, HashiCorp Vault with an agent, SOPS plus a host key, or another provider can satisfy the same host-side interface. The broker exists because the selected provider lacks a direct VM consumption path. Do not invent one if the provider already supplies workload identity.

Keep the broker narrow:

• Accept only POST on one versioned route.
• Bind it to one environment and one instance.
• Compare bearer tokens without leaking timing details.
• Return Cache-Control: no-store.
• Never log response bodies or secret values.
• Give the broker no deployment or general provider-management powers.

The host can call a stable adapter contract:

agent-secret-fetch materialize \
  --environment production \
  --instance alpha \
  --output /run/agent/alpha/runtime.env

The adapter writes no values to standard output, creates the file with mode 0600, and returns nonzero on failure.

Runtime retrieval

The deploy user owns the broker URL and bootstrap token files. The token file must be owner-readable only. Fetch into a memory-backed runtime directory when the operating system supports it. Set a restrictive umask, validate the result, start or recreate containers, then remove the file.

Deletion does not erase values from running container configuration. It narrows accidental exposure through shell history, build artifacts, working directories, backups, and later commands. The deployment user already controls Docker and remains inside the production trust boundary.

Do not echo a command containing a token. Do not enable shell tracing around secret operations. Redact canary values in tests and assert that they never appear in standard output, standard error, release manifests, image layers, workflow summaries, or bundle files.

Rotation is a normal operation

Define which service consumes each secret. Rotation then follows a bounded sequence:

1. Write the new value to the provider.
2. Materialize and validate a fresh runtime file.
3. Recreate only affected services.
4. Verify authentication and application health.
5. Revoke the old value when overlap is supported.
6. Remove the runtime file and record the rotation time, not the value.

Session-signing key rotation may invalidate active sessions unless the application accepts a short list of current and previous keys. Database password rotation may require changing both server and clients in a coordinated window. LLM API keys usually support overlap. Treat these as separate runbooks.

Test cell isolation during rotation. A host for alpha must not retrieve beta's namespace. Shared production credentials should be shared only when provider policy and blast-radius expectations allow it.

Recovery adapters

A file adapter is useful for local development, migration from an old environment file, and restoration when the external provider is unavailable. It should implement the same fetch contract and pass the same schema validation.

Do not make the file adapter the silent permanent fallback. If the broker fails and deployment automatically reads an old local file, a supposedly rotated credential may return without notice. Recovery use should require an explicit adapter path and source file.

Keep an offline or separately protected recovery package that documents which keys must be recreated. Avoid storing the production environment beside backups unless the recovery policy intentionally protects both under different controls.

Failure containment

A broker outage should stop a deployment before it touches running containers. The current application can continue with values already present in its containers. This is preferable to partially recreating services with empty defaults.

A malformed secret response should fail schema validation. A missing optional provider feature should disable that feature, not substitute a mystery value. A compromised alpha broker must not read beta secrets. Per-cell brokers and instance namespaces enforce that boundary.

Rate-limit and monitor broker failures, but do not log request authorization headers. Secret retrieval telemetry should contain instance, result, latency, and error class only.

The hardest limit remains the host. A compromised root or Docker-privileged user can read the bootstrap token and application secrets. A broker reduces distribution and cross-cell reach. It cannot rescue a fully compromised cell.

Decisions and rejected options

Choose encrypted secrets in Git only when the threat model and operating model support it. SOPS can be a sound choice when recipients are narrowly scoped, CI decrypts only in protected jobs, plaintext never reaches artifacts or logs, and key rotation and recovery are rehearsed.

Repository history intentionally retains old ciphertext, so removing a recipient or rotating a data key does not erase prior encrypted versions. For the reference system, keep the external-provider path because it gives each cell a narrow runtime identity and avoids distributing repository decryption keys to deployment CI.

Do not put secrets in OpenTofu variables or cloud-init. Both can persist them in state, provider consoles, logs, and instance metadata.

Do not let production fetch secrets during every application request. Retrieval belongs to deployment and controlled rotation. This keeps a provider outage from becoming an application outage.

Do not create one all-powerful broker for the fleet. Per-cell brokers cost more configuration but contain a token leak to one instance namespace.

Field checklist

• Git stores no plaintext secret values; any encrypted values have documented recipients, CI decryption, rotation, and recovery.
• Required, optional, grouped, shared, and instance-only keys are defined.
• The host can read only shared production plus its own instance namespace.
• Broker endpoints are narrow, authenticated, uncached, and value-silent.
• The bootstrap token is owner-only and absent from cloud-init and state.
• Runtime files use restrictive permissions and are removed after use.
• Canary tests search logs, artifacts, images, and manifests for leaks.
• Rotation runbooks identify affected services and overlap behavior.
• Recovery adapters require explicit selection and use the same validation.
• Broker failure occurs before changes to running containers.
• Operators understand that Docker administration remains a secret boundary.

Previous: Chapter 35, "Cells, not a cluster".

Next chapter

Chapter 37, "Static apps at the edge", separates public generated apps from the private cell without giving them server execution.