Checking what it gives back
- 01A green result proves nothing until it has itself been proven7 min
A check that has never been made to fail by a case it is actually supposed to catch has proven nothing, in the strict sense that proving requires a precise command to have run and its output to have been read: a broken detector returns zero problems, and that reads exactly like a success.
- 02Test the checker before trusting its verdict7 min
A check that returns a plausible number is not the same thing as a check that returns a correct number, and a faulty check rarely fails loudly, it returns a result that looks right.
- 03The remedy against invention can invent too7 min
A verification device is a deliverable like any other, produced by the same kind of mechanism as what it checks: replaying that mechanism a second time does not make it independent, only a genuinely different method proves it.
- 04Where you look determines what you find4 min
A review pointed at the latest change tells you where you looked, never where the defects actually are: without having examined the rest with the same care, no conclusion about the relative danger of recent code versus old code holds up.
- 05A very rigorous review can still see nothing4 min
The more disciplined a review method is, the more it gives the illusion of having covered everything, because it can only judge what already exists, never what is entirely missing.
- 06Proven, or simply not yet refuted5 min
Proven means a precise command has run and its output has been read; not yet refuted only means that nothing has caught a problem so far, a far weaker claim.
- 07A form check never sees a false value8 min
An automated check measures form, encoding, the presence of a keyword, never truth, and validating a screenshot or an intermediate step never proves what the person will actually open.
- 08Reopen every source before believing it8 min
A research agent reports what an index shows, not what a live page contains today, and presents a dead link with the same confidence as a live one: any source meant to support a conclusion gets reopened before being cited.
- 09Count the whole population, and name the measure for what it captures7 min
Measuring the few cases noticed by chance rather than the whole population produces a conclusion that is confident and yet wrong, and that same conclusion stays wrong if the name given to the result says something other than what the instrument actually captured.