Continuity – Surviving When Everything Else Has Failed¶

Machine translation of the French original.

Prevention, detection, and response can all fail. What remains, then, is the ability to recover. The question on this page is not "Has the backup succeeded?" but rather "Can I recover, and in how much time?" – two questions that have different answers.

The technical terms refer to the glossary.

Four Levels of Backup¶

flowchart LR
    classDef src fill:#3f3f46,stroke:#3f3f46,color:#fff
    classDef bk fill:#1e40af,stroke:#1e40af,color:#fff
    classDef off fill:#0f766e,stroke:#0f766e,color:#fff

    subgraph hotes["Hôtes du parc"]
        h["Hôtes de conteneurs · machines virtuelles<br/>hôte GPU · serveur hors site"]:::src
    end

    cli["Client de sauvegarde par hôte<br/>restic · chiffré AVANT l'envoi"]:::bk
    rest["Serveur de dépôt, sur le NAS<br/>ajout seul : aucune suppression<br/>depuis les hôtes"]:::bk
    snap["Instantanés du NAS"]:::bk
    rep["Réplication hors site"]:::off
    o1["Fournisseur hors site n° 1"]:::off
    o2["Fournisseur hors site n° 2"]:::off
    git["Configuration et décisions<br/>git, deux dépôts distants"]:::off

    h --> cli --> rest
    rest --> snap
    snap --> rep
    rep --> o1
    rep --> o2
    h --> git

Level	What it protects against
1 · Encrypted host-based backup	Loss of a host, deletion, application corruption
2 · NAS snapshots	Human error in storage, corruption of the repository itself
3 · Two off-site copies, with two different providers	Loss of the entire site – disaster, theft, ransomware that affects the NAS
4 · Git on two remote repositories	Loss of configuration, composition files, decisions

Encryption is done by the client, before sending: neither of the backup servers holds the keys to the data they store. Each host has its own client, its own repository, and its own recovery plan; integrity checks of the repositories are scheduled.

Immutability – The Ransomware Question¶

A modern ransomware doesn't just encrypt data; it searches for backups, along with the access it finds on the compromised machine. The key question is therefore:

Can someone who takes over a host also destroy its backups?

The property in question is called immutability: a backup that cannot be modified or deleted from the original host.

The backup server operates in a write-only mode. A client can create new snapshots; it cannot delete or overwrite them. The deletion of old snapshots is, in fact, performed through other access methods.
Off-site, immutability is enforced by the provider itself – either through object locking, or snapshots that replication access cannot reach.

How to verify: by trying

Immutability is not something that can be read from a configuration. It must be verified: by attempting to delete a snapshot, and waiting for a refusal. A probe repeats this action regularly. A positive response – the deletion has succeeded – is a failure of the test, not a success.

Why 'freshness' is not enough

Freshness monitoring answers the question "Has the backup run?". A ransomware creates fresh backups – of encrypted content, at a specific time, without any errors. Only immutability and a dated restore exercise answer the question "Can I recover?".

True Restoration – The Time-Based Exercise¶

A backup that has never been restored is just an assumption. The restore exercise turns it into a measure:

choose a representative stack – a database, files, or cache;
create a new, disposable virtual machine that has never seen the stack;
restore from the backups alone, without copying anything from the original machine;
time the process until the application API responds normally;
determine the age of the recovered data;
destroy the machine.

Le premier exercice a donné un RTO de l'ordre de trois minutes, et un RPO d'environ une journée — près de deux dans le pire cas, selon le moment de la panne par rapport à la dernière sauvegarde. Ce sont des chiffres mesurés, pas des objectifs recopiés d'un plan.

Chaque exercice est consigné dans un registre. Une sonde passe au rouge si le dernier More than 90 days: A proven restoration from a year ago says nothing about today's safeguards.

The measured RTO is a floor, not that of a disaster

The environment variables of a stack are never written in plain text: they are encrypted in the repository and decrypted in memory at startup. When restored from a backup, they reappear as a link to a file that no longer exists.

Restoring a stack therefore requires two additional things compared to a backup: the secret vault, and the key that decrypts it. The exercise was carried out with the key available. Its value applies to an ordinary incident; it does not apply to a situation where the key would have been lost along with the rest. This leads to the next section.

The master key – the only scenario of total loss¶

All the secrets within the stacks are encrypted using a single key. Losing your private key means losing all the secrets – even in the Git history, where the encrypted versions remain permanently and become unreadable forever.

And the dependency is circular: the secret vault itself is a stack within the park. Recovering the key from a vault that cannot be started without the key is a closed loop.

The solution is an offline copy: on paper, kept outside the park, with no network connection.

A copy that has never been reviewed is no better than having no copy at all. However, re-reading a private key is equivalent to exposing it. The verification kit addresses both:

a test secret, encrypted for the master key and versioned;
a script that reads the key from standard input – never as an argument, never on the screen, never in a file – attempts to decrypt the test secret, and returns pass or fail;
proven in both directions: a false key returns fail, the true key returns pass, including when entered manually from paper.

The entire exercise is performed on a system booted from a USB drive, network disconnected, directly from the sheet itself. It is attested, dated, and the attestation expires: a probe reminds it when it is missing.

What these choices have in common¶

Testing the property, not the configuration. Immutability is verified by attempting to delete; restoration is verified by restoring.
Encrypting as close to the data as possible. The client encrypts; the servers retain bytes that they cannot read.
Identifying circular dependencies before the disaster. The key in the safe, the safe in the park, the park behind the key: this is the kind of loop that you only discover when trying to start from scratch.

To read next¶

Compliance and evidence – how these measures become evidence.
Response – what precedes the restoration.
Reading a secret from the safe.

Dernière vérification de cette page : 2026-09-11