Home / Several agents and adversarial checking

Three failure states, and accepting a risk properly

An automation that processes a queue should distinguish success, recoverable failure that will be retried, and definitive failure that is abandoned, and only the third should reach a person, otherwise every recoverable failure wears out the value of the alert on the day it is right; accepting a risk with a compensating control differs from ignoring it, a snapshot taken before execution allowing the result to be sorted into three paths, silent normal variation, a significant change that only alerts, catastrophic loss that restores automatically.

A queue of automated tasks produces three possible outcomes for each item processed: a success, a recoverable failure that will be retried later, and a definitive failure that will stay failed whatever the system does afterwards. Lumping the last two together into a single failure category breaks the value of any alert built on top of it.

Why confusing the two failures wears out the alert

If every recoverable failure triggers the same alert as a definitive failure, a team receives a notification for an incident that will probably resolve itself on the next attempt. Repeated often enough, this alert loses its value: the people receiving it learn to dismiss it without reading it, because most of the time it signals nothing that demands immediate action. On the day it finally carries a definitive failure, the expected reaction no longer comes, precisely because the alert has been right too often for the wrong reasons. A one-off network timeout illustrates a typical recoverable failure, it often disappears on the next attempt; missing data at the source, which will not reappear on its own, illustrates a typical definitive failure.

result = process(task)

if result == success:
    report nothing

elif result == recoverable_failure:
    requeue the task, do not alert

elif result == definitive_failure:
    alert a person

Accepting a risk is not the same as ignoring it

A compensating control accepts the risk of an action rather than preventing it in advance: it lets the action happen, then checks afterwards whether it had the expected effect. A snapshot taken before execution makes this check possible by giving a point of comparison, the result obtained is then sorted into three paths, a normal variation that stays silent, a significant change that alerts without acting on its own, and a catastrophic loss that triggers an automatic restore. A compensating control that has never been triggered on test data does not yet deserve the trust placed in it. A previous lesson shows that multiplying agents does not make a result more reliable; cleanly distinguishing these three failure states is another way of keeping control over what actually gets escalated, rather than letting every alert fire at the same level.