What's running, and why¶
Machine translation of the French original.
The park contains six Docker hosts, approximately three hundred containers, an hypervisor, and a team of AI agents (as of 2026-10-06). Listing the containers would not be very informative; this page proceeds differently and surveys the park by capacity. For each, it provides what is in place and what led to the installation. Almost always, it's an incident or a missing component that we have identified, rarely a desire for a tool.
What this page hides
It names the tools, but does not provide version numbers, addresses, ports, or detection rules, and does not highlight any open vulnerabilities. An architecture is described; a list of exposed versions is a list of CVEs to try. The details are found in the private vault.
At a glance¶
| Capability | What it carries | To go further |
|---|---|---|
| Detection and Response | SIEM, behavioral detection, inventory, remote investigation, HTTP blocking | Detection · Response |
| Secrets | Two vaults: machines and humans | Read a secret |
| Backups | One host-encrypted repository, offsite copy | Continuity |
| Monitoring and Alerts | Metrics, end-to-end probes, disk health, a central point for notifications | Wire up a notification |
| Identity and Access | SSO, self-hosted mesh network | Security |
| Web Exposure | Two-stage reverse proxy, one DNS record per name | Reverse proxy chain |
| Tickets | Detection loop → ticket → fix → evidence | — |
| AI Agents | A team of agents in a sandbox | Agents |
| Language Models | Local inference on GPU, single gateway | Agents |
| Automation | Workflows | — |
| Code | Self-hosted forge, shared mirror | Continuous integration |
| Hosts | Hypervisor, autonomous nightly updates | Park of hosts |
| Knowledge | One source vault, internal wiki, this site | Where to write what |
Detection and Response¶
Wazuh acts as a SIEM: file integrity, known vulnerabilities, correlation with MITRE ATT&CK. Falco monitors process behavior at the kernel level, and osquery maintains the host inventory, which it feeds into Wazuh. Velociraptor is used to ask a question of the entire park during an incident. CrowdSec blocks at the HTTP level.
Why. Wazuh responds to the need for self-hosting auditing in June 2026, rather than just auditing services. Velociraptor arose from a file system failure on 2026-08-24: it took nine hours because each machine had to be interrogated individually. CrowdSec, on the other hand, starts from a simple observation: dozens of public names, and no intrusion prevention system in front of them.
Falco follows a rule specific to the environment: each new host starts in isolation, for 24 hours, to measure its noise. Then, exceptions are defined one by one, and only then is the notification enabled. An alert that is never read eventually becomes ignored.
Secrets¶
Machine secrets (API keys, service passwords, tokens) are in OpenBao, on a machine that only handles them. They are stored as encrypted files SOPS within containers. Human secrets (web accounts, emergency codes) are in Vaultwarden.
Why. The separation date is 2026-09-27. A secret read by a machine and a secret typed by a person have different lifecycles, different rotation rates, and different readers. Furthermore, the machine vault is kept offline: each host retains its encrypted copy, so that a powered-off vault delays rotations without preventing the environment from restarting.
Backups¶
restic, driven by Backrest, backs up six hosts, each in its own encrypted repository, to the NAS. One copy is offsite, out of reach of the backed-up hosts. The NAS also creates its own volume snapshots.
Why. The host-based backup runbook remained in the "planned" state for weeks, while the data accumulated. Backup levels and timed restoration are described in Resilience.
Monitoring and Alerts¶
Prometheus collects metrics from hosts and maintains versioned alert rules. Gatus probes services end-to-end and provides a public status page. Scrutiny monitors the SMART status of physical disks. A morning dashboard gathers the status of the hosts and the latest incidents. alerts-hub is the single point of entry for notifications: a single call, and the message goes to the appropriate channel's messaging system.
Why. On 2026-08-14, the environment had no alert rules. Forty-one stops due to memory shortages were discovered the following day, on a screenshot. For public probes, the decision of 2026-09-16 is to measure the entire chain (DNS, proxy, application), because a service can be healthy behind a broken path.
Identity and Access¶
Authentik provides SSO. For any new service, the rule is OIDC first. The forward auth (a wall in front of the application) is a last resort, and we never combine the two. The private network between hosts goes through a Headscale-managed tailnet, self-hosted.
Why. Forward auth does not pass any identity to the application: no account, no role, no true disconnection. Combined with OIDC, it intercepts the connection return and causes a loop. Headscale replaced the hosted service on 2026-08-19: node keys that don't expire silently, versioned access lists, and OIDC without a subscription.
Web Exposure¶
A request originating from the internet traverses two layers of Caddy. The first handles the edge, while the second serves internal services. Certificates are issued through DNS challenges. Each public name has its own DNS record: there are no wildcards in the zone.
Why. The rule of 2026-09-10 is simple: a name is only exposed if someone has explicitly decided to expose it for that specific name. A wildcard by default exposes everything that is forgotten. The details are in Reverse proxy chain.
Tickets¶
What is detected automatically opens and closes its own ticket, automatically, with a hysteresis to avoid interfering with the probes. What is found during a working session opens a ticket manually, and this ticket is only closed with a written resolution: how to verify it. A triage agent distributes the queue.
Why. In late August 2026, 529 tickets still had the status "open", and nothing was recalculating this status. A state that is declared in a system capable of calculating it always ends up lying.
AI Agents¶
A team of AI agents, internally named STA Inc, operates at a rate of one container per agent, on a dedicated sandbox VM. Each agent only receives its own tokens; the secrets of the park are not on this VM. The agents work in Vikunja. Among them, Clio, the documenter, measures the coverage of the documentation and fills in the gaps.
Why. On 2026-10-02, it was discovered that an agent launched from the development VM had access to all secrets and all hosts. The dedicated VM closes this scope of action. The agents are organized in a tree, so that no one can send a task back to itself in a loop.
Language Models¶
Ollama serves models locally, on the host's GPU. LiteLLM is the single gateway for the entire estate: every call, whether local or to an external provider, passes through it, with a key per user, a budget, and a fallback.
Why. The single gateway makes costs and usage transparent from one place, and allows you to switch providers without affecting users. It became an independent stack on 2026-09-28.
Automation¶
n8n handles workflows: approval chains, notifications, document processing. It only communicates with Docker through a filtering socket proxy, never directly to the socket.
A rule based on experience accompanies n8n: a daily task that does not rely on a scheduled n8n trigger, which can fail. It uses a system timer that checks the status and retries.
Code¶
Gitea, self-hosted, is the source for every repository. GitHub is used for sharing. A synchronizer runs every five minutes in both directions and never forces anything.
Why. A one-way mirror forcefully overwrites: a merge made on GitHub would disappear on the next pass (decision of 2026-09-27). An application with a user interface has its own repository; the locally built image has its versioned recipe alongside its configuration.
Hosts¶
VMs run on a Proxmox hypervisor. The Linux hosts apply their updates at night, without intervention, and each night produces a report. The hypervisor stack is manually maintained, because its installation scripts restart the management services.
Why. Until September 2026, the VMs ran on a desktop workstation that handled over eighty containers, using a slow virtual disk, while the other host was saturated. The migration to the hypervisor, in September 2026, moved this workload to fast local storage.
Knowledge¶
The vault, which consists of Markdown notes stored in Git and edited in Obsidian, is the source of all knowledge within the park: historical decisions, runbooks, and approximately 1,700 incident FAQs. An internal wiki serves as a readable mirror of this vault. This site is a read-only and public version.
Why. An FAQ arises from an incident and recounts an investigation, including false leads. It should be stored in the vault. This site only retains the information that remains true after the investigation is closed. Details on how to document this information can be found in Where to write what.
And for daily use¶
The park also serves as: a photo library (Immich, whose image analysis runs on the GPU, approximately ten times faster than on a CPU, processing over forty thousand photos), a home automation system (Home Assistant, Node-RED, MQTT), and a media library. These are the services for which everything else exists.
Dernière vérification de cette page : 2026-10-06