Home / Checking what it gives back
A very rigorous review can still see nothing
The more disciplined a review method is, the more it gives the illusion of having covered everything, because it can only judge what already exists, never what is entirely missing.
An adversarial review, however disciplined, always answers a closed question: does this piece of code do what it claims to do. It rereads files, functions, screens that already exist in the deliverable it is given. A screen never built, a function never written, a flow never wired up appear nowhere in this scope, so no review, however thorough, can find them there.
A check can only judge what exists
This is not a weakness of method, it is a structural limit of the act itself: you only reread what has been written. The proof lies in a single command, which builds itself the file it queries before querying it, rather than depending on a live mapping in the project that could be edited between the writing of this lesson and its reading:
$ printf 'v2m3l1\nv2m3l2\nv2m3l3\nv2m3l4\nv2m3l5\nv2m3l6\nv2m3l7\nv2m3l8\nv2m3l9\n' > planning.txt
$ grep -c "v2m3l10" planning.txt
0
$ echo "code de sortie : $?"
code de sortie : 1The command returns zero occurrences on its standard output, because no string matches, and it does signal something else besides: its exit code is 1, the convention by which grep indicates the absence of a match. The exit code is part of the output, just as much as the printed text, and ignoring it means losing part of the information already returned. This signal still does not say a lesson is missing: whether the module's tenth lesson was dropped along the way or was never planned at all, the command would return exactly the same zero and the same code 1 either way. A check confined to what the file contains cannot tell these two situations apart.
Rigour makes the illusion worse, not better
The result is counter intuitive: the more disciplined a review method is about what it looks at, the more its positive verdict inspires confidence, and the more misleading that confidence becomes about what it did not look at at all. As with a review pointed at the latest diff, a green result obtained through a serious method reads as total coverage, while it says nothing about what is absent from the scope examined.
What closes the blind spot
No amount of extra discipline applied to the same act closes this gap, since the act by construction only bears on what exists. What closes it is a different question, asked outside the code being reviewed: a complete user journey rather than a list of cases to check, handed to real use, even very brief, carried out early by the person concerned. Such use is not more rigorous than a disciplined review, it examines a different scope, one that includes total absence, where a code review can by construction only examine what has been written.
What rigour changes, and what it never changes
Often missed
A relaxed check lets through defects that are nonetheless present in the code.
Missed too, unsurprisingly
A relaxed check obviously does not see what was never written.
What rigour genuinely changes
A disciplined check rereads what exists with the intended care, without this saying how many defects still escape that same scope.
Missed all the same
The same check, however rigorous, can say nothing about a screen, a function or a flow that was never built: it is not in its scope.
A planning mapping describes a ten lesson module, numbered one to ten, for a writing team. Several adversarial agents each receive the files for the nine lessons already written, with instructions to look for defects in them. Each agent submits a defect report limited to the files it was given.
Write, in one sentence, what this result establishes, and in one sentence what it does not establish.
What this establishes: These reviews judge the lesson files submitted to them, without saying anything about a file absent from what they examined.
What this does not establish: The silence of these reviews does not establish that the missing tenth lesson is minor or without consequence, nor that a further review pass would catch it.
The three most common miscalibrations
- Too broad This silence establishes that the mapping is complete and that no lesson is missing from this module.
- Too narrow This silence establishes nothing at all about the work done by these agents, since no defect was found.
- Beside the point This silence shows that the number of agents deployed was insufficient for this work.
- An adversarial review, however disciplined, only judges what already exists in the scope it is given, never what is entirely absent from it.
- A check confined to existing files returns the same silence for something forgotten and something that was never planned at all: it cannot tell the two cases apart.
- The more rigorous a review method is about its scope, the more its green result inspires a confidence that far exceeds what it actually covered.
- A brief real use is not more rigorous than a disciplined review: it examines a different scope, one that includes total absence.
Open a product or deliverable you have just had carefully reviewed, and try to reach its final goal without following any instructions or documentation. Note every point where you get stuck not knowing what to do. The number of stopping points is your verifiable result, and each one is a candidate for something the code review could never see.