Home / Checking what it gives back
A form check never sees a false value
An automated check almost always verifies form and very rarely truth: a valid file, a correct encoding or a keyword on the page can all sit beside a false number. Validating a screenshot of the page that generates the document likewise does not prove what the person actually opens.
An automated check that scans a deliverable before publication almost always verifies one thing: form. Is the file valid, does the encoding look right, does an expected keyword appear somewhere on the page. It very rarely verifies truth: does the displayed number match the correct value, is the claim actually accurate. This gap stays invisible until the day a false value sails through a check showing all green.
What a form check never sees
A rule bans publishing a disputed figure. The source constant holding it follows the rule perfectly: it is never printed directly anywhere. A second constant, derived from the first in the same file, is nonetheless displayed on a large number of the site's pages. Several distinct automated checks stay green, because none of them compares the value actually displayed against the banned value: they check syntax, encoding, structure, never meaning. The figure that follows quantifies this leak.
The same flaw takes a simpler form with a keyword based check: a rule requiring a document to mention a given term is trivially satisfied by writing anything false next to that term, since it only looks for the word, not its context. This mechanism carries on from a green result proves nothing until it has itself been proven: a check that has never been made to fail by the exact case it should catch proves nothing about that case.
Validating what the person actually opens
A visual fix was confirmed on a screenshot of the HTML page that generates the document, then announced as shipped. The file the person opened was a PDF, produced from a copy of the HTML that had not yet received the fix: the defect stayed visible in the distributed PDF, which had never been the thing checked.
The same trap appears around caches: rereading a piece of data through the path that just wrote it proves only that path's consistency with itself, and a cached copy can return the old value. Break it with a fresh address parameter, a private window or a server without a cache; a simple refresh often leaves a relay untouched.
When the measurement itself miscounts
A form check can also get the measure wrong. Under Git Bash on Windows, grep does not count carriage returns reliably. Measure with the language that applies the threshold, and keep a witness that must return a non-zero count:
printf 'a\r\nb\r\n' > crlf.txt
grep -c $'\r' crlf.txt
node -e "console.log(require('fs').readFileSync('crlf.txt','utf8').split('\r').length-1)"0
2Taken under Git Bash (GNU grep 3.0), the output shows grep at 0 on a file that holds two carriage returns, and node at 2: the witness proves that the node counter detects and that the grep one misses. On Windows, grep strips carriage returns from a file it judges to be text before searching; the -U option, which reads it as binary, returns 2. Without it, the zero count reads as a clean file. Outside Git Bash, replay the witness before concluding.
The check that would have caught the leak
The same mechanism, with a built file: the banned constant is never printed directly, and a check that looks for that printing finds nothing, rightly so:
$ printf 'const CHIFFRE_INTERDIT = 999;\nconst AFFICHAGE = CHIFFRE_INTERDIT.toString() + ",00 EUR";\nconsole.log(AFFICHAGE);\n' > /tmp/source-v2m3l7.js $ grep -c "console.log(CHIFFRE_INTERDIT" /tmp/source-v2m3l7.js 0
But four lines further down, the derived constant picks up this value and does get printed, at runtime:
$ node /tmp/source-v2m3l7.js 999,00 EUR
This check returns a result that is honestly green, and yet false for the use made of it: it checks one precise line of the source file, not what the person actually sees on screen. Identify what the person will actually open, a PDF, a live page, an application screen, then check exactly that thing, with the tool that genuinely opens it, after breaking the cache. A form check is never a correctness check, and it must never remain the last gate before publication.
What a form check looks at, and what it misses
| What a form check verifies | What it lets through silently |
|---|---|
| The file is valid HTML or JSON | Whether a displayed number matches a banned value |
| A required keyword appears somewhere on the page | Whether the sentence around it is true |
| The source constant never shows the banned value | A constant derived from it, further down in the file |
| The check runs on the source file | Whether that same value has survived into the shipped artefact |
A leak measured with precision
A rule bans displaying a specific figure on the site. An automated check scans the page's source file and searches for that figure written out, character for character. This same source file also contains three values, written separately in three places in the code, whose sum exactly matches that figure.
Write, in one sentence, what this result establishes, and in one sentence what it does not establish.
What this establishes: The check establishes that the banned figure does not appear written out in the source file it examined.
What this does not establish: It does not establish that this figure is absent from the site's pages as actually displayed to visitors, since it never compares the source value to the value rendered on screen.
The three most common miscalibrations
- Too broad This check establishes that the banned figure appears nowhere on the site, source and displayed pages included.
- Too narrow This check establishes nothing at all, not even the absence of the figure written out in the source file.
- Beside the point This check shows that the team knew the rule banning this figure before writing it into the code.
- A form check verifies syntax, encoding or the presence of a keyword, never whether the claim around it is true.
- A banned value can leak through a derived constant while the original constant stays perfectly compliant with the rule that bans it.
- A screenshot of the source proves nothing about the final file the person actually opens, PDF, export or cached page.
- Rereading a piece of data through the path that just wrote it only proves that path's consistency with itself, not the value genuinely served to the visitor; under Git Bash, grep can also miscount, hence a positive witness measured with the language that applies the threshold.
- Breaking a cache demands an active step, a fresh parameter, a private window or a cache free server: a simple refresh is not always enough.
Take a deliverable you have just produced, note what its form check cannot see, then check that separately on the final version actually opened by the person concerned.