Secrets on a host

The secrethive command runs on the host where the containers run. It writes a compose project's secrets as one file per key, for applications that read /run/secrets, and it gets the values from the server at deploy time with an identity of the host's own. It is one command that runs and ends: no agent, no daemon, nothing that listens.

It ships as a single binary for linux-x64 and linux-arm64, and as a container image.

Two files in the compose repository, no values in either

The host profile, secrethive.host.yml at the root: what is true of the whole host.

version: 1
host: web-1
appdata: /srv/appdata
bundle_key: /etc/secrethive/bundle.key          # this host's private key, outside the repository
providers: [api, bundle]
server: https://vault.example.org
server_key: /etc/secrethive/server.key          # this host's key for the server, outside the repository
server_certificate: secrethive-server.crt       # only for a server with a certificate it made for itself

`providers`	Where values come from
`[bundle]`	Only the sealed bundle next to the manifest
`[api, bundle]`	The server when it answers, the bundle when it does not
`[api]`	The server or nothing

With server_certificate, exactly that certificate is trusted and nothing else. There is no switch that turns verification off.

The manifest, <stack>/secrethive.yml: which secret files exist where, and which secret fills each.

version: 1
stack: shop
vault: shop-prod
targets:
  app:
    path: ${appdata}/shop/secrets
    owner: "1001:1001"            # the user the container runs as
    restart: [shop]
    keys:
      ConnectionStrings__Default: sql-connection
      Api__Token:                 api-token
      Api__User:                  api-token#user          # one field of a secret
      Mail__Password:             mail/smtp-password      # another vault
      Site__Pfx:                  site#pfx                # one part of a certificate

A reference is name, vault/name or vault/name/3 for a pinned version, optionally followed by #field. For a certificate the part is certificate, chain, key, pfx or pfx-password.

Containers that take secrets as variables

Most third-party images do not read /run/secrets; they take a password as an environment variable. For those a target can be one file instead of one file per key:

targets:
  compose:
    path: ${appdata}/compose-secrets/shop
    owner: "0:0"                # read by docker compose itself; never mounted into a container
    format: env
    restart: []
    keys:
      DB_PASSWORD:   db-password
      SMTP_PASSWORD: mail/smtp-password

render then writes secrets.env in that folder, a line per variable in single quotes, which Compose takes as they are. The stack's docker-compose.yml names it for the files it includes, beside .env:

include:
  - path: shop.yml
    env_file:
      - .env
      - /srv/appdata/compose-secrets/shop/secrets.env

A plain docker compose up -d keeps working, by hand, at boot and from a deploy job, with nothing to remember; and with the rendered file missing, Compose refuses to start anything rather than start with empty secrets. .env keeps the settings and can go into the repository.

A changed value reaches a container when Compose makes it anew (docker compose up -d after render); a restart keeps the variables the container was made with. A value with a line break cannot be a line of an env file; one with a backslash at its end or before a quote would not come back as Compose reads it. Both are refused before anything is written.

bundle import takes the variables of such a target out of the stack's existing .env: only values that are unmistakable (plain, or quoted with nothing in them that Compose would rewrite); any other is refused by name and left where it is. To prove a move changed nothing, compare the hash of docker compose config before and after.

The commands

secrethive render <stack> --host <profile> [--dry-run] [--target <name>]
secrethive check  <stack> --host <profile>
secrethive status <stack> --host <profile>
secrethive run    <stack> --host <profile> -- <command>

secrethive bundle pull   <stack> --host <profile>
secrethive bundle push   <stack> --host <profile> [--dry-run] [--server-key <file>]
secrethive bundle import <stack> --host <profile>
secrethive bundle set    <stack> <reference> --host <profile>
secrethive bundle prune  <stack> --host <profile>
secrethive host keygen --host <profile>

Command	What it does
`render`	Makes each target directory hold exactly what the manifest declares. A file holds the bytes of the value and nothing else, is owned by the container's user, mode `0400` in a `0500` directory. Files are completed beside the target and renamed in, so a container never reads half a secret. An unchanged file is not rewritten; a file the manifest no longer declares is removed (anything there that is not a regular file is reported and left). Prints which containers to restart
`check`	Refuses a deploy when a reference does not resolve, a value is empty or has whitespace at an edge, or a running container's user is not the owner of the directory it reads
`status`	Compares what is on disk with what the server (or the bundle) holds, and each file's age with its container's start time: there is no reload, so a file newer than the container means the container still has the old value. Also says when the bundle is behind the server
`run`	Starts a command with the manifest's `env` references in its environment, for applications that do not read files
`bundle pull`	Brings the bundle in step with the server: compares versions, reads only what changed, seals it for this host
`bundle push`	The other direction, once: makes on the server what the bundle holds and the server lacks

Every command prints names only. No value is printed, logged or taken as an argument.

Exit	Meaning
0	Nothing to do, or in step
10	`render`, `bundle pull` or `bundle push` changed something; `render` names the containers to restart
20	`status` found a difference
1	Refused; nothing was written

bundle set, bundle import and bundle prune exit 0 when they did their work; run exits with the command's own code.

A render is all or nothing: every value is resolved and checked before the first file is touched.

The bundle: the same secrets when the server is not there

<stack>/secrethive.bundle holds the stack's values sealed for one host. Each value is encrypted with a key of its own, and that key with the host's public key, so sealing needs nothing secret and only the host's private key opens it. It is safe to commit: a diff shows which names changed and nothing else.

With [api, bundle], a run reads the server when it answers and the bundle when it does not, decided once before anything is read. The server saying no is not "does not answer". A revoked or expired key, a missing role, a secret that is not there, an address outside the allowed networks: each is reported, the run is refused, and the bundle is not used to get around it. Otherwise revoking a host's key would change nothing until its bundle was deleted.

The fallback is as old as the last bundle pull. status says when it is behind; the command pulls only when it is asked to. A deploy asks; a stack nobody deploys for months is kept fresh by a daily timer that runs bundle pull for every stack that has a manifest. That is safe to run unattended: it writes no secret file and restarts nothing, exit 10 means something was pulled, and a pull the server refuses (a key that ended, a role taken away) is the one result worth an alert.

A deploy

secrethive bundle pull shop --host web-1      # keep the fallback fresh; commit the bundle when it changed
secrethive check shop --host web-1
secrethive render shop --host web-1           # exit 10 names what to restart
docker compose up -d

As a container

hive() {
  docker run --rm --user 0 --network host \
    -v /srv/compose:/srv/compose -v /srv/appdata:/srv/appdata \
    -v /etc/secrethive:/etc/secrethive:ro \
    -w /srv/compose <registry>/secrethive-cli:0.14.0 "$@"
}
hive render shop --host web-1

--user 0 because the files are handed to other users, which only root can do. The paths are the same inside the container as outside it on purpose, so the profile means the same in both. check and status look at running containers and need the docker command, which the image does not carry: run those two with the binary. They also read which user a running container's process belongs to, so inside a container of their own they need the host's process list (--pid host); without it they refuse and say so.

Setting a host up

On the server: a service identity for the host, Secrets User in the vaults its stacks read, and a key limited to the host's address. See Roles, keys and networks.
On the host: the key in a file only root reads, outside the repository; secrethive host keygen for the bundle key pair (commit the public half).
The profile and a manifest per stack in the compose repository.

Moving a stack that already has its secrets on the host

bundle import seals the files that are on disk today into the bundle. bundle push then makes them on the server:

On the server	What push does
Not there	Makes it from the bundle: the value, and every `#field` the manifest reads as a sensitive field. A certificate is made from the parts the manifest reads (`#pfx`, `#pfx-password`, `#certificate`, …); the server checks that they are one certificate and refuses otherwise
There, and equal	Nothing; marks the bundle with the server's version
There, and different	Refuses the whole run. It never overwrites: which of the two is right is for a person to settle

Everything is looked at before anything is written, and --dry-run shows the plan. The key that writes is named for that run with --server-key and belongs to whoever does the move, as Secrets Officer; the host's own key stays read-only. The profile only has to name the server for this, with providers still [bundle], so stacks can be pushed one by one before the host is switched: providers is per host, and switching makes every stack on it read the server.

After the switch, bundle pull finds nothing to do and status reports in step, without a single secret file having been rewritten. From then on a value is changed on the server, followed by bundle pull and render.

Limits

A value read through the API is text. What is not UTF-8 text has to be a certificate part, which travels as bytes.
Through a public name behind a CDN, values pass that CDN in memory. A host that can reach the server directly should use its own address with server_certificate.