Home / Checking what it gives back
Where you look determines what you find
A review pointed at the latest change tells you where you looked, never where the defects actually are: without having examined the rest with the same care, no conclusion about the relative danger of recent code versus old code holds up.
An adversarial review almost always rereads the latest diff first, the latest commit, the block you just changed. This narrow scope is a reasonable choice, but it produces a misleading effect if you forget it: what this review finds depends first on where it looked, not only on where the defects actually are.
A count without a denominator compares nothing
Claiming that recent code is more dangerous than old code would require two numbers: the share of the repository reviewed that was recent, and the defect rate found in each of the two shares, old and recent, for equal reviewing effort. Without these two numbers, a count of defects found in the latest diff only measures the scope chosen, never the relative danger of the two zones. This is an ordinary selection bias, recognisable as soon as it is named: counting only the cases that were noticed, instead of the whole population, then generalising from that chosen subset, produces the same error whatever name you give it.
The bias can be checked without depending on any live file from the course, which gets rewritten with every correction and would return a different output on every rereading. The command below builds the two files it counts itself, before counting them: a reader who reruns it on their own machine gets exactly the same output, today as in six months. Counting how many times a word appears in a single file, then in two neighbouring files, gives two correct counts and two different counts, without either one saying how many times that word would appear in a wider set that was never queried. The command here counts occurrences, not lines: grep -c counts lines containing the pattern, not the number of times it appears; grep -o prints each match on its own line, which wc -l can then count without confusing a line that contains the word once with a line that contains it three times.
$ printf 'Ce fichier decrit un défaut de comptage.\nUn deuxieme défaut apparait plus bas.\nUn troisieme défaut clot la liste.\n' > fichier-b.txt
$ grep -o "défaut" fichier-b.txt | wc -l
3
$ printf 'Ce fichier voisin decrit un premier défaut.\nPuis un second défaut.\n' > fichier-a.txt
$ grep -o "défaut" fichier-a.txt fichier-b.txt | wc -l
5The scope queried determines the number found. Three, or five, depends only on the number of files the command opened, not on some deeper property of the text fabricated for the demonstration.
What still holds up despite the bias
Reviewing the latest change first still makes sense, not because it would necessarily concentrate more defects, but because it is the only code no one else has examined yet. An old line has already survived several reviews and several runs. A line written ten minutes ago has only survived its author's own review, and that immediate review replays the same intention that produced the error, if there is one: it cannot see it, as shown by the lesson on a green result that has not yet been proven. It is a matter of urgency, not a statistical pattern.
A guard or check just added to fix a specific defect inherits the same urgency: it has just been born, no one has made it fail yet, and a green result it returns today proves no more than a green result returned by code that has never been tested.
What a pointed review reveals, and what it cannot say
Real defects, counted exactly within this precise scope.
The density of defects in the rest of the repository, not reviewed with the same effort, so no comparison between recent code and old code.
An adversarial agent is instructed to review a repository's latest commit looking for defects. It spends twenty minutes on this and reports back three defects, each located in the lines that commit modified. Each defect comes with a line number and a two sentence explanation.
Write, in one sentence, what this result establishes, and in one sentence what it does not establish.
What this establishes: The latest commit contains at least three defects, found by this specific review.
What this does not establish: The number of defects found in the latest commit does not allow its defect density to be compared with that of the rest of the repository, which was not reviewed with the same effort.
The three most common miscalibrations
- Too broad Recent code in this repository generally concentrates more defects than old code.
- Too narrow The three defects found do not really count, since only one commit was examined.
- Beside the point This result shows that the commit's author works with less rigour than the rest of the team.
- A defect count limited to a single scope measures that scope, never the relative danger of the rest, for lack of a denominator common to both zones.
- Looking harder in one zone makes you find more there: a recognisable selection bias, which needs no bad faith at all to produce reasoning that is fluent and wrong.
- Reviewing the latest change first stays defensible because it is the only code no one else has examined yet, not because defects would statistically concentrate there.
- The review an author makes of their own code at the moment of writing it replays the same intention that produced the error: it therefore cannot reveal it.
- An outside look, a person or an agent who did not write the code, tests what a review by the author can never test.
Ask an agent in a fresh session, who did not write your latest diff and does not know your intent, to review it with the sole instruction to look for a reason it might be wrong. Then have this same agent, with the same instruction, review a file that has stayed stable for several months. Note which of the two passes returns a defect: this tells you where to look next, not where the defects necessarily are most numerous.