Stack Lifecycle

Machine translation of the French original.

Creating

1. Create 10-stacks/<nom>/ in the vault, with README.md and decisions.md.
2. Write /opt/docker/<nom>/docker-compose.yml to the host, according to the conventions.
3. Validate: docker compose -f /opt/docker/<nom>/docker-compose.yml config.
4. Start: docker compose -f /opt/docker/<nom>/docker-compose.yml up -d.
5. If the service is exposed on the web → Publish a new service.
6. On vm-docker and vm-compute-01, register the stack in /opt/docker/scripts/stack-waves.conf — otherwise it disappears on the first restart (see below).
7. Commit the vault and push to both remote repositories.

Modifying

Any configuration change is accompanied, immediately, by an entry in 10-stacks/<nom>/decisions.md: what changed, why, and how to revert.

This is not bureaucracy. It is the only channel that all working sessions review: a spoken message or a note in memory each has a silent failure mode, decisions.md does not.

Boot-up

On vm-docker and vm-compute-01, the services declare restart: on-failure:5, never always.

docker-staggered.service is the only boot-up authority. It reads /opt/docker/scripts/stack-waves.conf, mounts projects in waves and waits between them.

sequenceDiagram
    autonumber
    participant S as systemd
    participant D as dockerd
    participant G as docker-staggered
    participant C as conteneurs

    S->>D: docker.service
    D-->>S: prêt
    S->>G: After=docker.service
    loop vagues 0 à 5, 15 s d'écart
        G->>C: compose up -d (projets de la vague)
    end
    Note over S,G: à l'arrêt, l'ordre s'inverse :<br/>staggered-stop tourne PENDANT que dockerd est encore debout

Why this mechanism exists:

• on vm-compute-01, the bottleneck is the lock xtables — each iptables rule takes 20 to 28 seconds, and dockerd blocks for more than ten minutes;
• on vm-docker, it's the I/O wait — 80 containers on a virtual disk at ~1.4ms per random read brings the load up to 85.

stateDiagram-v2
    [*] --> running: compose up -d
    running --> crash: le processus meurt<br/>(exit ≠ 0)
    crash --> running: on-failure:5 relance
    crash --> abandonne: 5 échecs d'affilée

    running --> arrete: staggered-stop<br/>(exit 0, arrêt PROPRE)
    arrete --> running: docker-staggered au boot,<br/>SI le projet est dans stack-waves.conf
    arrete --> oublie: sinon, personne ne le rallume

    note right of oublie
        exited · ExitCode=0 · RestartCount=0
        aucune erreur, aucune alerte
        et le vhost Caddy répond en 502
    end note

on-failure:5 only covers the left branch. The right branch has no automatic retries: it's the wave file, and nothing else.

A stack missing from stack-waves.conf does not recover, and nothing signals that

This is the direct consequence of on-failure:5: a cleanly stopped container (code 0) is not restarted by Docker. Or, this is exactly what staggered-stop does before an host restart – a clean stop.

If the project is not in stack-waves.conf, it will not be turned on at boot. The container remains exited with ExitCode=0, RestartCount=0, with no errors in the logs, no alerts: it did not crash, it was shut down and never restarted.

This happened on 2026-08-30 on homelab-docs, which was put into service the previous evening and was missing from the file: it was cleanly stopped at 01:45 during the nightly patching restart, and remained missing until we searched for it. The Caddy vhost, however, was still responding – with a 502 error.

The check, after any deployment:

grep -c '<nom-du-projet>' /opt/docker/scripts/stack-waves.conf   # doit rendre 1

Verify that a container adheres to the declared policy

docker inspect <conteneur> \
  --format '{{.HostConfig.RestartPolicy.Name}}:{{.HostConfig.RestartPolicy.MaximumRetryCount}}'

This should be compared to the restart: in the Compose file. A discrepancy means that a up -d --force-recreate has re-enabled the old policy.

Diagnosing

Order of investigation, from least to most expensive:

1. docker ps – Is the container running, and if so, when did it start?
2. RestartCount – Is it restarting in a loop?
3. docker logs --since 1h – What does it say about itself?
4. Health check – but see the warning below.
5. Network, volumes, memory.

healthy does not mean "the service is working"

The health check is what the image has chosen to test, nothing more. Real-world examples encountered here: a closed port while the container is healthy, a secondary service failing without affecting the probe, a interval that lasts 15 minutes and makes the health check useless during an incident.

A corollary for monitoring: failing on unhealthy, never on starting.

A probe can measure something other than the service

A single-process service that writes its state to disk synchronously freezes its event loop during writing – 20 to 30 seconds for a graph with 45,000 nodes. The probe expires, but the container has never restarted. The answer lies on the supervision side (longer delay, threshold for consecutive failures), not on the service side.

Decommissioning

1. Stop and remove: docker compose -f /opt/docker/<nom>/docker-compose.yml down.
2. Remove the Caddy block from the Caddyfile, from both Caddies if the service was public.
3. Remove the domain name from the certificate issuance list.
4. Remove the entry stack-waves.conf from the orchestrated hosts.
5. Archive the vault directory instead of deleting it, and close decisions.md with a dated entry.

A published name at the edge with a valid certificate is a door that is waiting

Leaving the Caddy block in place after stopping the container does "nothing": when someone starts a container with the same service name, it becomes public instantly, without going through a review. Blocks were removed for this very reason during the public surface audit on 2026-07-22.

Dernière vérification de cette page : 2026-08-30