Home / Checking what it gives back

A green result proves nothing until it has itself been proven

A check that has never been made to fail by a case it is actually supposed to catch has proven nothing, in the strict sense that proving requires a precise command to have run and its output to have been read: a broken detector returns zero problems, and that reads exactly like a success.

A verification script that reports zero problems can mean two opposite things: the system it checks is genuinely clean, or the check itself no longer works. Nothing in a green result lets you tell the two apart until that check has failed at least once against a case it is required to catch.

The trap of zero

Two measured cases make this trap concrete. A broken escaping pattern once returned zero flagged violations where thousands should have appeared. Another broken pattern, the opposite way round, flagged violations that did not exist, and a reversed order of operations produced continuous false alarms in a real case: the figure attached to this lesson gives the exact count for both failures. Both came from the same root cause, a check that had never been tested against itself. A scheduled task's exit code sets the same trap: a script can report success for weeks while the work it is meant to do silently fails for a reason outside its own error handling.

The positive witness, proven by execution

The fix comes down to one rule. A check must look, in the same pass, for at least one thing it must find, a case known to be bad: this is the positive witness. The demonstration takes only a few lines. A script that searches for the banned word secret in a file, with a regular expression that forgot to ignore case, runs on a file that contains that word twice, written with an initial capital letter.

node verif.js fixture.txt
violations trouvees : 0

Zero. The checked file nonetheless contains two occurrences of the banned word. Nothing in this result tells a genuinely clean file apart from a broken check, until someone has verified that this check can detect a known case. Adding case insensitivity, exactly the witness this check must now find, changes the verdict.

node verif2.js fixture.txt
violations trouvees : 2

Produce the witness before declaring the check active

The order of operations matters more than it looks. Wire up the writing of the witness on success first, run the task once for real so that this witness genuinely exists, and only then declare the check active. Reversing this order, declaring a witness check active before anything has written that witness, trains people to ignore the entire monitoring system: an ignored system is worth less than no system at all.

One last habit closes the loop. Rename the witness, never delete it, confirm the check reacts every time, then restore it. Also list out the conditions a completeness check covers instead of summarising them behind a phrase like everything is fine: that phrase can stay true for one condition and false for the other two without anyone noticing.

This principle carries on into test the checker before trusting its verdict, which applies the same requirement to a script that measures a number rather than a simple presence.