Home / Checking what it gives back

Count the whole population, and name the measure for what it captures

Measuring the few cases noticed by chance rather than the whole population produces a conclusion that is confident and yet wrong, and that same conclusion stays wrong if the name given to the result says something other than what the instrument actually captured.

Two distinct reflexes protect against the same error, concluding before having seen the whole set of facts: counting the entire population before recommending a change, and writing down what a measure actually captures before naming it. Skipping either one produces a conclusion that is full of confidence and yet can be entirely wrong while looking rigorous.

Count before recommending

Spotting a few cases that break a convention and drawing a general recommendation from them, dropping the convention entirely, is an appealing line of reasoning and almost always premature. Measuring the complete population often tells a different story: a large majority of the elements already followed the convention, consistently and deliberately: the four cases noticed at the outset represented only a fraction of the eleven elements actually non compliant across the whole population, a narrower defect unrelated to the convention itself. A convention that is massively respected is a decision someone made on purpose, not an accident waiting to be fixed. And the fact that a team agrees to move forward on this diagnosis only validates the authorisation to act, never the accuracy of the diagnosis itself: one can agree in good faith to correct a bad count.

The figure following this paragraph illustrates the same discipline on a second example, a folder of nine files built for the demonstration, where a glance at whichever file comes first says nothing about the real count:

$ mkdir -p /tmp/demo-v2m3l9
$ for i in 1 2 3 4 5 6 7; do echo '{"sources":["x"]}' > /tmp/demo-v2m3l9/lecon$i.json; done
$ for i in 8 9; do echo '{"sources":[]}' > /tmp/demo-v2m3l9/lecon$i.json; done
$ node -e "const fs=require('fs');const d='/tmp/demo-v2m3l9';const f=fs.readdirSync(d).filter(x=>x.endsWith('.json'));let a=0;f.forEach(x=>{const j=JSON.parse(fs.readFileSync(d+'/'+x));if(j.sources&&j.sources.length)a++});console.log(a+' sur '+f.length+' avec une source declaree');"
7 sur 9 avec une source declaree

Two files out of the nine have an empty source field, and nothing in their name or position gave it away: you had to open them one by one to find out, exactly as with the forty three elements in the first example.

Name a measure for what it captures

A chart titled working day can, in reality, measure something else: the gap between the first and the last message sent during the day, silently excluding every hour spent away from the keyboard in between. The figure is not wrong, it is mislabelled, which is worse. A wrong figure eventually gets noticed and corrected. A false label spreads quietly and contaminates every conclusion built on it, with no way for an outside reader to see it just by looking at the title. The same trap catches any indirect signal presented as if it measured something else directly: a higher frequency of capital letters in a series of messages was, for instance, named a signal of rising frustration, when the instrument literally captures nothing but a frequency of capital letters. The figure following this paragraph revisits this example and sets the supposed label against what the instrument actually captures. Where you look determines what you find rests on the same missing denominator as the habit of counting before concluding.

Before titling any chart, any dashboard tile, any column, write in one simple sentence what the instrument literally captures, then check that the title says exactly that, nothing more.