Home / Checking what it gives back

Test the checker before trusting its verdict

A check that returns a plausible number is not the same thing as a check that returns a correct number, and a faulty check rarely fails loudly, it returns a result that looks right.

A script written to measure something, a count of problems, of characters matching a pattern, of badly named files, runs and displays a number that looks plausible. A plausible number and a correct number are not the same thing, and the gap between the two almost never announces itself.

Five bugs, five convincing wrong numbers

During a single audit, five home written measurements turned out to be wrong, each for a different reason. A directory check returned false on a valid symbolic link, as if the linked folder did not exist. A byte level comparison miscounted accented characters made of several bytes. A grouping step ignored case and merged entries that should have stayed distinct. A byte order mark injected at the start of a file threw off every later read, and a regular expression poorly matched to line endings skipped content. None of the five errors announced itself: each produced a reasonable looking number, and each was wrong.

What a regular expression dot actually crosses

The line ending case is measured rather than assumed. In JavaScript, by default, a regular expression dot crosses no line break, neither the plain newline used on Unix systems nor the pair of characters Windows uses. The script pointe.js compares the two, on one file of each style.

node pointe.js unix.txt
point simple, sans drapeau s : false
point simple, drapeau s      : true
deux points, drapeau s       : false

node pointe.js windows.txt
point simple, sans drapeau s : false
point simple, drapeau s      : false
deux points, drapeau s       : true

The Windows specific trap only appears once the flag that lets the dot cross line breaks is turned on: a Windows line ending counts as two characters, a carriage return and a line feed, and a single dot only covers one of the two. On a Unix file, that same flag is enough, since there is only one character to cross. A self test that includes content straddling a Windows line ending, with the flag active, catches the error that the same input with a Unix line ending cannot reveal.

The self test, before trusting

The fix costs a few minutes and must run before the first real output of a measurement script. Give it a fabricated input it absolutely must flag, and a clean input it absolutely must not flag. Have these two cases cover the real quirks of your ground: different line endings, accented characters, symbolic links, mixed case. A script unable to correctly classify its own test cases with a known answer deserves no trust on real data, however plausible its number looks.

Count the false alarms too

A dead link checker once flagged a batch of problems for which the figure attached to this lesson gives the exact count and the share that was actually well founded. Most of the flagged items came from a naming inconsistency the checker had not anticipated, and two distinct false positive bugs were hiding under that noise: a byte order mark mistaken for missing metadata, and a quoted example inside a code block mistaken for a genuine style violation. Both bugs were only found by testing the detector against known cases, never by scrutinising its output more closely. A false accusation costs as much trust as a missed defect: measure a detector's false positive rate, not just what it catches.

Having the program itself write its output file avoids the byte order mark: on PowerShell 5.1, the redirection operator and Out-File insert one, invisibly; Set-Content adds none but writes in the local encoding, breaking accented characters. Remedy: explicit UTF-8 writing without a marker, via WriteAllText and UTF8Encoding($false). This principle ties back to a green result proves nothing until it has itself been proven: a check that has not proven itself on known cases deserves no credit on real cases.