Home / Checking what it gives back
A form check never sees a false value
An automated check measures form, encoding, the presence of a keyword, never truth, and validating a screenshot or an intermediate step never proves what the person will actually open.
An automated check that scans a deliverable before publication almost always verifies one thing: form. Is the file valid, does the encoding look right, does an expected keyword appear somewhere on the page. It very rarely verifies truth: does the displayed number match the correct value, is the claim actually accurate. This gap stays invisible until the day a false value sails through a check showing all green.
What a form check never sees
A rule bans publishing a disputed figure. The source constant holding it follows the rule perfectly: it is never printed directly anywhere. A second constant, derived from the first in the same file, is nonetheless displayed on a large number of the site's pages. Several distinct automated checks stay green, because none of them compares the value actually displayed against the banned value: they check syntax, encoding, structure, never meaning. The figure following this paragraph gives the exact numbers for this leak: the number of pages affected, the number of checks that stayed green despite it, and the gap, in lines of code, between the banned constant and its derivative.
The same flaw takes a simpler form with a keyword based check: a rule requiring a document to mention a given term is trivially satisfied by writing anything false next to that term, since it only looks for the word, not its context. Anything a deliverable adds to satisfy a form check deserves to be checked against the source of truth, not merely found present on the page. This mechanism carries on from a green result proves nothing until it has itself been proven: a check that has never been made to fail by the exact case it should catch proves nothing about that case.
Validating what the person actually opens
A visual fix was reviewed and confirmed correct on a screenshot of the HTML page that generates the document, then announced as shipped. The file the person concerned actually opened was a PDF, produced from that HTML by a separate conversion step. That step carried its own defect: it regenerated the PDF from a copy of the HTML that had not yet received the fix, so the problem, though properly fixed in the live page, remained visible in the distributed PDF, because the PDF had never been the thing checked.
The same trap lurks around caches. Rereading a piece of data through the same path that just wrote it proves nothing more than that path's consistency with itself: a cached copy can return the old value even though the write fully succeeded. Breaking that cache demands an active step: adding a fresh parameter to the address, opening a private browsing window, or querying a server directly configured to ignore its own cache. A simple browser refresh often leaves an intermediate relay untouched.
The check that would have caught the leak
Here is the same mechanism, replayable with a file built for the example: the banned constant is indeed written in the source code, never printed directly, and a check that looks for this direct printing finds nothing, rightly so:
$ printf 'const CHIFFRE_INTERDIT = 999;\nconst AFFICHAGE = CHIFFRE_INTERDIT.toString() + ",00 EUR";\nconsole.log(AFFICHAGE);\n' > /tmp/source-v2m3l7.js $ grep -c "console.log(CHIFFRE_INTERDIT" /tmp/source-v2m3l7.js 0
But four lines further down, the derived constant picks up this value and does get printed, at runtime:
$ node /tmp/source-v2m3l7.js 999,00 EUR
This check returns a result that is honestly green, and yet false for the use made of it: it checks one precise line of the source file, not what the person actually sees on screen. The lesson comes down to one habit: identify what the person will actually open, a PDF, a live page, an application screen, then check exactly that thing, with the tool that genuinely opens it, after breaking the cache. A form check is never a correctness check, and it must never remain the last gate before publication.
What a form check looks at, and what it misses
| What a form check verifies | What it lets through silently |
|---|---|
| The file is valid HTML or JSON | Whether a displayed number matches a banned value |
| A required keyword appears somewhere on the page | Whether the sentence around it is true |
| The source constant never shows the banned value | A constant derived from it, further down in the file |
| The check runs on the source file | Whether that same value has survived into the shipped artefact |
A leak measured with precision
A rule bans displaying a specific figure on the site. An automated check scans the page's source file and searches for that figure written out, character for character. This same source file also contains three values, written separately in three places in the code, whose sum exactly matches that figure.
Write, in one sentence, what this result establishes, and in one sentence what it does not establish.
What this establishes: The check establishes that the banned figure does not appear written out in the source file it examined.
What this does not establish: It does not establish that this figure is absent from the site's pages as actually displayed to visitors, since it never compares the source value to the value rendered on screen.
The three most common miscalibrations
- Too broad This check establishes that the banned figure appears nowhere on the site, source and displayed pages included.
- Too narrow This check establishes nothing at all, not even the absence of the figure written out in the source file.
- Beside the point This check shows that the team knew the rule banning this figure before writing it into the code.
- A form check verifies syntax, encoding or the presence of a keyword, never whether the claim around it is true.
- A banned value can leak through a derived constant while the original constant stays perfectly compliant with the rule that bans it.
- A screenshot of the source proves nothing about the final file the person actually opens, PDF, export or cached page.
- Rereading a piece of data through the path that just wrote it only proves that path's consistency with itself, not the value genuinely served to the visitor.
- Breaking a cache demands an active step, a fresh parameter, a private window or a cache free server: a simple refresh is not always enough.
Take a deliverable you have just produced, note what its form check cannot see, then check that separately on the final version actually opened by the person concerned.