Skip to content

Configuring a Notification

Machine translation of the French original.

Anything that needs to alert someone should post a single JSON to the dispatcher. This routes it to two destinations: the team messaging and the mobile messaging. Each sender is unaware of the other — it's the dispatcher that holds the routing information, and this allows you to add or remove a destination without modifying any sender.

flowchart LR
    e["Émetteur<br/>script · cron · service · workflow"]
    h["Répartiteur<br/>service systemd, 24/7"]
    z["Messagerie d'équipe"]
    t["Messagerie mobile"]

    e -->|"POST JSON"| h
    h -->|"synchrone<br/>décide le code de retour"| z
    h -.->|"file asynchrone<br/>reprise sur limitation de débit"| t
    e -.->|"repli, seulement si le répartiteur ne répond pas"| z

The Fallback

curl -s -X POST http://<répartiteur>:8099/alerts-notify \
  -H 'Content-Type: application/json' \
  -d '{"channel":"general","title":"mon titre","message":"mon message"}'

That's it. Three fields, with only one being mandatory in practice: message.

Field Role
channel Chooses the destination channel. If missing or unknown, defaults to general, is logged, and flagged in the response
title Becomes the subject of the channel on the team messaging side — see trap #2
message The body
format: "raw" Forces a code block, for a command output

What Not to Do

  • Not the public URL of the workflow engine. It works — it's a facade that relays — but it adds two components to the path of an alert. A new sender targets the dispatcher directly.
  • Not the team messaging directly, except for the fallback described below. A sender targeting it directly never reaches the mobile messaging, so never a phone, so never anyone outside the office.
  • Not a hand-written webhook. The shared fallback rule addresses four traps that are undetectable by either, and all four are silent.

The Fallback

The dispatcher returns 200 once the team messaging has received the notification, 502 otherwise. This 502 is the signal: it triggers the sender's fallback, sending it directly to the team messaging.

The fallback exists for one case only: the dispatcher doesn't respond. It doesn't reach the mobile messaging — this can only be reached via the dispatcher.

There are two mechanisms, and the choice is clear:

The sender… Mechanism
rotates from the note deposit import of the shared module
is deployed outside the deposit or resides in a container copy of the shared module

The reason for the second case is not convenience: a import from a network share would put that share on the alert path. Missing file server ⇒ the script dies before it attempts to send, and we also lose the path that would have worked.

An untested fallback is one that is assumed to work

It only works when the distributor fails – that is, when no one is looking. Testing means breaking the main path, on the relevant host, and verifying that the message still arrives.

And testing that it fails is also important: clear all the key sources, not just one. A negative test that passes for a bad reason is worse than no test at all.

Registering the sender – non-negotiable

A sender that is not on the manifest is never upgraded when the shared module evolves. It will continue to function, with the version it was installed with, until a patch that it has not received actually takes effect.

Two files, in the same session as the installation:

  • the manifest of the copies to keep up to date;
  • the inventory of the senders, regenerated by scanning all hosts.

The inventory is the only view that finds what no one has declared. This is how four divergent copies of the same watchdog were discovered, one of which was on the manifest.

The inventory tells you where it's going, not how it's sent

It classifies by pattern matching. It doesn't know if the file is running. A measurement taken on 2026-09 identified five "emitters" from a category, of which two were dead code – their target path did not exist on any machine. This false positive leads to action: "These emitters are outside the dispatcher, align them." Aligning inert code produces a complete-looking map that describes a script that doesn't run.

Before acting on a line in the inventory, look for the caller: cron, timer, service, or another script.

The Four Traps

1. An HTTP 200 status code does not guarantee a successful response

Several APIs encapsulate their errors in a body success: false under a 200 status code. A form check curl -o /dev/null -w '%{http_code}' then responds to "the web server is up," never to "the call succeeded." The shared block reads the body and rejects a non-JSON 200.

This allowed, in 2026-09, to determine that a historical destination had been dead for three weeks: it responded with 200 to everything, with an error code "this API does not exist" in the body.

2. The subject is a thread identifier

On the team messaging side title, the subject becomes the thread name. A timestamped subject creates a new thread with each send – 48 threads per day for an emitter that runs every half hour.

The subject carries the emitter's identity; what varies goes in message. A subject that varies on an entity 🔴 CRITICAL — <conteneur> is legitimate: it creates a thread per container, which is what we want.

3. A guard on a dead token neutralizes the fallback

The pattern, found on seven files from four hosts in 2026-09:

poster_sur_le_repartiteur "$1" && return 0
[ -n "${JETON_D_UNE_DESTINATION_MORTE:-}" ] || return 0   # ← ici
repli_direct "$1"

The guard tests the token for a decommissioned destination to allow a fallback, while the fallback is aimed at a destination that is still active. Since the token no longer exists, the function returns 0 without attempting anything: an apparent success, a lost alert, precisely the day the switch fails.

Damage measured on one of these probes: 18 alerts swallowed, including the detection and resolution of a faulty host, which was never announced.

It wasn't an oversight

The fix was applied to two out of six files, and never propagated. A fix that is not pushed to all locations where the pattern exists is a fix that is believed to be in place.

4. A guard per time window survives its purpose

Some probes HOUR_START / HOUR_END were in place because two machines were offline at night. They are no longer offline. The guard, however, remained — and prevented false alerts during migration.

Focus on the cause, not the time. A failure is diagnosed; it is not explained in advance by the moment it occurs.

Arming a new alert

An alert connected to a system that is already running first sees everything.

  1. Measure the volume that the rule would produce, over a representative period, before connecting the notification.
  2. Calibrate the threshold based on this history — otherwise, you are measuring the instrument's cadence, not the state of the fleet.
  3. Set a arming date: what existed before is not an event.
  4. Set a hysteresis and a duration. A notification that repeats identically says nothing more than the first.

Verifying

# 1. le répartiteur répond
curl -s http://<répartiteur>:8099/health

# 2. le chemin nominal — lire les destinations dans la RÉPONSE, pas le code HTTP
curl -s -X POST http://<répartiteur>:8099/alerts-notify \
  -H 'Content-Type: application/json' \
  -d '{"channel":"general","title":"probe","message":"test"}'

# 3. le repli — celui qu'on oublie, et le seul qui compte la nuit

/health This also highlights the depth of the queue for mobile messaging. This queue exists because this destination imposes rate limits: waiting for the incoming request cost 2,295 messages lost in fifteen days on a properly written restart, disabled by a constant that was too low.

What the queue costs, and what needs to be monitored

A failure on this destination is no longer visible to the sender: it only appears in the log and in the queue depth. Without monitoring this depth, we replaced a noisy loss with a silent loss.

What should not go through the router

  • Off-band probes, which exist to communicate when the network is down. They must reach a phone without traversing the monitored infrastructure.
  • Conversational bridges – allowing communication with an assistant from a messaging system is not a notification, it is bidirectional transport.
  • Application mail intended for the end user.

Dernière vérification de cette page : 2026-09-09