Build once, deploy by digest

The same commit is built twice. CI produces one image, but the production host runs docker build again and gets another. A package registry changed between builds, a base tag moved, or the host had a stale cache. The source SHA matches while the bytes do not.

Build once removes that ambiguity. CI creates the artifact. Production runs that exact artifact.

Release flow at a glance

commit
  -> validate source
  -> build backend and web images
  -> publish registry digests
  -> package deployment scripts and public config
  -> target one cell
  -> fetch secrets on that cell
  -> pull digests, back up, migrate, replace, verify
  -> record current digests

Vocabulary

An image tag is a mutable registry name such as app:main or app:abc123.

An image digest is a content address such as sha256:.... Pulling that reference identifies exact bytes.

A deployment bundle contains host-side Compose and operational scripts. It contains no application source or secrets.

A schema revision is a hash of the database and shared client schema used to check component compatibility.

An attestation records build provenance. An SBOM lists software components in an image.

Separate validation, publication, and deployment

The validation job runs on a clean hosted runner. Install from the lockfile, typecheck, test, build the UI, render production Compose with placeholders, and build both runtime targets. Pull requests stop here. They never execute on the production runner.

Publication runs only for trusted refs. BuildKit creates a backend image and a web image. The backend can run the agent worker, sync process, or migrations under different commands. The web image contains static assets and Caddy. Publishing them separately permits a UI-only release without restarting data services.

The deployment job receives output references like these:

registry.example/agent-backend@sha256:4a1f...
registry.example/agent-web@sha256:9c30...

The tag remains useful for browsing the registry. The digest is the deployment input.

Use the registry close to the source control and CI system unless there is a reason not to. GitHub Container Registry is the reference choice because GitHub Actions already builds the project and can issue narrow package permissions. Amazon ECR, Google Artifact Registry, or a private registry preserve the same design.

Images need compatibility labels

Independent components still share contracts. A web client may depend on a schema shape served by the backend. Add the source revision, build time, version, and schema revision as OCI labels during the build. Before a web-only rollout, inspect the active backend and candidate web labels. Reject a mismatch before replacing anything.

Do not place credentials in build arguments or image layers. Build arguments appear in provenance and caches more often than people expect. Runtime secrets belong to the target cell and arrive after the image exists.

A CI build can pass public metadata without passing credentials:

with:
  push: true
  tags: registry.example/app:${{ github.sha }}
  build-args: |
    SOURCE_REVISION=${{ github.sha }}
    SCHEMA_REVISION=${{ steps.schema.outputs.hash }}
  provenance: mode=max
  sbom: true

Pin third-party workflow actions to immutable revisions. A floating action tag reintroduces mutable code into the trusted build.

Treat deployment as a small transaction

Acquire a per-cell lock first. Two deployments must not migrate or replace services concurrently.

Preflight should validate the public instance file, materialize and validate secrets, authenticate to the registry, pull requested digests, inspect image labels, and write a candidate release manifest. Any preflight failure leaves running containers unchanged.

A full rollout then:

1. Creates instance-scoped Postgres and applicable Redis backups.
2. Starts and checks Postgres.
3. Runs migrations once with the candidate backend image.
4. Replaces backend, sync, cache, and web services.
5. Checks container health and the public login contract.
6. Atomically promotes the candidate manifest.

A web-only rollout compares schema labels, runs compose up -d --no-deps web, checks Caddy and a hashed asset, then updates only the web digest in release state. Record container IDs in tests to prove no backend or database service restarted.

Store release state with an atomic rename:

write_candidate > "$root/releases/current.env.next"
mv "$root/releases/current.env.next" "$root/releases/current.env"

Keep several previous manifests. They are tiny and turn rollback into a deterministic digest selection.

Rollback has two meanings

Application rollback replaces containers with prior image digests. It works when the database remains compatible. If a health check fails before a migration, the deployer can do this automatically.

Data rollback restores a pre-deploy database dump. Migrations are usually forward-only, and silently restoring a database would discard writes made after deployment. Print the backup path and recovery command. Require an operator to make that data-loss decision.

Design migrations for an overlap window. Add columns before requiring them, deploy readers that tolerate both shapes, backfill, then remove obsolete fields in a later release. This makes image rollback useful for more than a few seconds.

Decisions and declined alternatives

Build on hosted CI, not on the target host. The host then needs Docker, Compose, and the release bundle, but no compiler, package manager, Git checkout, or dependency cache.

Publish backend and web separately. A single image would simplify the registry list but force Caddy and the UI to share the backend lifecycle.

Use digests, not commit tags alone. Commit tags can be moved or republished. A digest names content.

Keep deployment on a repository-scoped self-hosted runner that connects outbound. Direct SSH from CI is workable, but it requires another network credential and an inbound path. The runner is still production privileged, so only trusted jobs may target it.

Do not let an operator force a web-only release when changed paths include schema, dependencies, Compose, backend code, or unknown files. A classifier may narrow a release, but uncertainty must widen it to full.

What fails in practice

• A workflow builds successfully but records the tag instead of the digest.
• A manual deploy then pulls newer bytes.
• A bundle contains an example secret file that later becomes real.
• A registry login failure happens after the backup and migration because preflight ordering was wrong.
• A deployment health check sees an old container still serving and promotes a failed candidate.
• Two workflow runs share no concurrency key and race.
• A previous web image starts but cannot speak to the active schema.

Test these paths with fake container commands and at least one real disposable stack. Shell syntax tests do not prove transaction ordering.

Release checklist

• Pull requests run validation only on hosted runners.
• Trusted CI publishes backend and web images once.
• Production references include @sha256:.
• Images carry source and schema labels, provenance, and an SBOM.
• The deployment bundle has checksums and no source or secrets.
• Preflight completes before state changes.
• Per-cell locking prevents concurrent deploys.
• Full deploys back up before migration.
• Web-only deploys prove other containers did not restart.
• Release state updates atomically and keeps prior manifests.
• Image rollback and database restore are documented separately.

Previous: Chapter 33, "Containers, Caddy, and clear ports".

Next chapter

Chapter 35, "Cells, not a cluster", shows how this release model supports several isolated installations without turning them into replicas.