Verification and proof

Every check needs a positive witness

Most courses teach you to write tests. Fewer teach you this: a broken verification check does not usually announce itself with an error. It quietly reports zero problems, and zero problems reads exactly like success. If nobody built in a way to tell the difference between "genuinely clean" and "the check itself is broken," you will not notice, and neither will anyone reading the report.

Here is the fix, stated as a rule: a verification script must search, in the same pass, for at least one thing it must find. This is called a positive witness, a known-bad case planted or present in the data that the check is guaranteed to catch if it is working. Without a positive witness, a check that finds nothing is indistinguishable from a check that is silently miscounting, misreading its input, or looking in the wrong place entirely. Two measured cases make this concrete: a broken escape pattern once showed zero flagged violations where thousands should have appeared, and a separately broken pattern flagged 136 violations that did not exist. Both came from the exact same root cause: a check that had never itself been tested.

The same logic extends past code checks, into anything that runs on a schedule without a human watching it. A scheduler's own exit code is not enough proof that a job did its job: a script can report success over and over while the work it was supposed to do silently fails for a reason outside its own error handling, such as a dependency the script assumed was there and was not. The fix is to cross two signals: the scheduler's own state (is the job enabled, what was its last exit code, when did it last run) and an application-level witness that the job writes itself on success: a timestamp file, a specific data field, or the newest dated report landing in a folder. A job with no such witness should be reported as unverifiable, never assumed healthy by default just because nothing complained.

Order matters more than it seems. Wire up the write-on-success first, actually run the job for real once so the witness gets produced, and only then declare the check active. Doing this backwards, declaring a witness check before anything writes the witness, is not a small slip: in one real case it created six permanent false alarms, alarms that trained people to ignore the whole monitoring system, which is worse than having no system at all.

One more habit closes this out: a completeness check that summarizes several conditions behind a single sentence hides exactly how many of them are actually being tested. "All checks passed" sitting on top of a rule that quietly covers three separate conditions can pass while only one of the three is genuinely being verified. Enumerate the conditions a check covers, do not summarize them, and finally, drill the whole mechanism on purpose: rename a witness (never delete it) and confirm the check reacts every single time, then restore it. A detection mechanism that has never been drilled is unverified code wearing a green checkmark, and a stale everything-is-fine mechanism is worse than none at all.

Test the verifier before you trust the verdict

You just wrote a script to measure something: how many issues exist, how many characters match a pattern, how many files are correctly named. The script runs, prints a number, and the number looks plausible. Here is the uncomfortable fact this lesson is built on: a plausible number and a correct number are not the same thing, and the gap between them almost never announces itself.

On a single audit, five separate self-written measurements turned out to be wrong, each for a completely different, subtle reason: a directory check that returns false on a symbolic link, as if the linked folder did not exist. A byte-level comparison that miscounts characters made of several bytes, common with accented letters. A shell redirect that silently injects an invisible byte-order marker at the start of a file, throwing off anything that reads it afterward. A regular expression whose dot does not match across Windows-style line endings, silently skipping content that spans a line break. And a grouping operation that ignores letter case, quietly merging things that should have stayed separate.

None of these five announced themselves. Each one produced a number that looked entirely reasonable, and each one was wrong. That is the core lesson: a broken measurement script rarely fails loudly, it fails by handing you a confident, wrong answer that you have no reason to doubt unless you specifically go looking.

The fix is a habit that costs a few minutes and should run before you trust any measurement script's first real output: give the script a self-test before you trust its verdict. Feed it a fabricated input that it absolutely must flag, and a clean input that it absolutely must not flag, and make sure both test cases cover the realistic edge cases of your actual terrain: different line endings, accented or multi-byte characters, symbolic links, and mixed case. If the script cannot correctly classify its own known-answer test cases, its verdict on real data is worthless, no matter how plausible that verdict looks.

A real example of this playing out: a broken-links checker once flagged 23 issues, and only 2 of them were genuinely real problems. Most of the rest were a naming-convention mismatch the checker did not account for, and two separate false-positive bugs were hiding underneath that noise, a byte-order marker misread as a missing metadata block, and a quoted example inside a code block misread as a real style violation. Both of those two specific bugs were found only by testing the detector itself against known cases, not by staring harder at its output. This points at a broader truth worth remembering on its own: a false accusation costs as much trust as a missed defect does, so measure your detector's false-positive rate, not only how much it catches. And on Windows specifically, be aware that redirecting a script's output to a file through a shell operator can inject encoding artifacts that break whatever reads that file next; having the program write its own output file directly avoids that entire class of problem.

Fix the class of defect, not the instance

When you fix a bug, the natural stopping point feels like the moment the reported case works again. That instinct is what makes a fix too narrow. The habit worth building is different: after every fix, ask what family the defect belongs to, then sweep every other member of that family in the same pass, before you move on to something else.

Three same-day fixes made this concrete. A hardcoded path was secured, while a sibling path carrying the identical defect was left untouched. One of two duplicated alert latches was reset, while the other stayed stuck. A success witness was added to a monitored process, but not to the separate watcher that reads its output. Each of these fixes, taken alone, was correct. Each was also too narrow, and all three followed the exact same shape: fix the instance in front of you, miss the twin sitting one file or one module away.

None of these three were caught by the person who wrote the fix. All three were caught by adversarial review, meaning a second pass whose whole job is to look for exactly this pattern. That is the real distinction between a review that adds value and a review that just re-checks the same work: a simple re-check asks whether this fix works. A useful adversarial review asks where else this exact defect still lives.

• Instance fix: repairs the specific case that was reported.
• Class fix: repairs every sibling that shares the same underlying pattern.
• Adversarial review: a second pass whose mandate is to hunt for surviving members of the family, not to re-verify the one fix that already works.

The practical takeaway: the moment a fix lands, name out loud what class of bug it belongs to (a hardcoded value, a duplicated state, a missing witness), then search for every other place that pattern could recur, in the same sitting.

A correction is finished when it converges with a source you did not produce

A dramatic improvement feels like proof. A number drops 79 percent right after you fix an identified cause, your own internal check turns green, and it is tempting to call the correction finished. That feeling is exactly backwards. A large drop is a signal of progress, never proof of correctness, and the moment it looks most spectacular is the moment you are most tempted to assume the cause was singular, when multiple causes often stack on the same number.

The only thing that actually closes the loop is agreement with a source you did not produce yourself, an independent one. In one case, a figure fell 79 percent after a fix, looked finished, and was not: comparing it against an independent accounting file revealed a second, unrelated cause that the first correction's dramatic drop had simply masked. The correction was only truly finished once the number converged with that outside source, not when the internal check turned green.

The same discipline applies to any number destined for a human or a client: reread it from the authoritative, live source, meaning production, never a local development clone. A clone drifts even when it started as an exact copy, and one reported figure was off by an order of magnitude for exactly that reason, a stale local database standing in for the real one.

There is a second, symmetrical trap: a correct fix can create its own regression elsewhere. Qualifying a delay in more precise language lengthened every string just enough to push twice as many pages past a length limit. The fix was correct on its own terms and still broke something else. Build the remeasurement of the same metric into the tool itself, and weigh any flagged regression by what it actually costs, whether the affected text still carries its meaning, rather than by the raw count of items over a threshold.

• Convergence: agreement between your corrected number and an independent source you did not produce.
• Masked cause: a second, unrelated defect hidden by the size of the first correction's drop.
• Drift: the gap that opens between a development clone and the live production source over time.
• Coupled regression: a side effect a correct fix introduces somewhere else in the same system.

The practical takeaway: never call a correction finished on the strength of the drop alone. Check it against a source you did not build, reread client facing numbers from production, and remeasure the same metric again after the fix lands.

Mark completion on the result, never on the attempt

An automation that stamps a task done, removes it from a queue, counts it as a success, or exits with status zero the instant it is attempted, rather than when it actually succeeds, can lose work forever without anyone noticing. The danger is not the failure itself. It is that the system reports success anyway, so nothing ever tells you to look.

A real sync script did exactly this: it accumulated 239 consecutive failures and zero successes over several weeks, while still marking every session as synced, dequeuing it, and exiting with code zero every single time. All three layers of monitoring built to watch it reported everything fine, because every one of them was reading the attempt, not the result.

The fix is a change in what gets recorded and when. On failure, the work should stay queued, with a persisted retry counter capped at a small number of attempts, and it should only be abandoned through an explicit logged line, never through a silent drop that leaves no trace. The success counter and the exit code must reflect genuine successes only. And a success witness should measure what the task actually produced, a file that was written, a timestamp that moved forward, not merely that the process returned without crashing.

• Attempted: the task ran, whatever the outcome.
• Succeeded: the task produced the specific, observable result it exists to produce.
• Success witness: a check tied to that produced result (a file, a timestamp), not to the process's own exit code.
• Explicit abandonment: a logged line stating a retry limit was hit, as opposed to a silent drop that leaves nothing behind.

The practical takeaway: before trusting any automation's report, check what its success signal is actually wired to. If it is wired to the attempt, it can run for weeks doing nothing useful while telling you everything is fine.

A commit message is a story, not a state

A message describing what was done is not a measurement of what actually happened, and it never turns into one just because time passes. This applies to more than commit messages: a CI status field and an agent's own report about its work share the exact same failure mode.

A commit titled as closing a security issue was believed. The actual code still leaked both secrets on five separate call paths, because the commit had only touched nine comment lines. The title told a story about intent. The code told a different one about what actually happened, and only the code was true. The same day, several recommendations pulled straight from a sub agent's own inventory report were already stale or wrong, for the identical reason: a report is a narrative written by the thing being evaluated, not an independent measurement of it.

Continuous integration has its own version of the same trap. A polling loop that reads a job's intermediate status field, rather than waiting for its terminal conclusion field, can read an empty or transitional value and announce success before the job has actually finished. The fix is mechanical: wait for a non-empty terminal field, targeted at the exact commit you expect, then go check the real effect on the target system, not the tool's own announcement of victory.

Before declaring any defect fixed, open the file that carries it and cite the exact line. When the subject has an observable counterpart somewhere else, an environment variable actually loaded by a build, an API response, a generated file, check that counterpart directly instead of trusting the narrative that describes it.

• Story: a message, title, or report describing an intended or claimed outcome.
• State: the actual, observable condition of the code, the job, or the system right now.
• Terminal field: a status value that will not change again, as opposed to an intermediate one that might.
• Observable counterpart: a directly checkable fact (a file, a variable, a response) that stands in for a claim.

The practical takeaway: read the commit, the field, or the report as a claim to verify, never as a fact already established. Then go open the actual file, wait for the actual terminal status, and check the actual observable counterpart.

Reopen every URL an agent hands you

A search agent reports what an index currently shows it, not what the live page actually contains today. It will present a dead link and a genuinely live one with exactly the same tone of confidence, because from its point of view both are simply a match it found.

The scale of the problem is worth knowing in numbers. On one identity research sweep, eleven agents returned 103 findings. Reopening the 94 that carried a URL showed 33 dead or redirected, and 9 pointing to a namesake rather than the actual subject. Close to half the batch was unusable, despite every single item being presented as fact with the same confident tone as the ones that held up.

The fix is a separate verification pass. Any URL destined to support a conclusion, or to be cited, gets reopened by a different agent than the one that originally found it, and that second agent is explicitly instructed to try to disprove the finding: does the URL actually respond, does the content match what was claimed, is this genuinely the right subject rather than a namesake. On identity or attribution research specifically, state the disambiguating criterion out loud (a specific employer, a specific date, a specific location) and default to undetermined rather than confirmed whenever a page fails to meet it.

Report both numbers when you finish a sweep like this: the raw count of findings and the count that actually survived reopening. A zero gap between the two, on a batch of any real size, is not evidence of a perfect sweep. It is evidence that whoever checked did not really try.

• Raw finding: whatever a search agent reports as a match, unverified.
• Surviving finding: a URL that was reopened and confirmed to actually support the claim.
• Disambiguating criterion: the specific fact (employer, date, location) that separates the real subject from a namesake.
• Undetermined: the honest default when a page does not meet that criterion, as opposed to falsely marking it confirmed.

The practical takeaway: never cite or act on a URL an agent handed you without reopening it yourself, or having a different agent try to disprove it. Report the raw count and the surviving count side by side, every time.

A form check cannot see a false value

An automated check that scans a deliverable before it ships almost always tests form: is the file valid, does the encoding look right, is a required keyword present. It very rarely tests truth: is the number correct, is the claim actually accurate. That gap is invisible until the day a false value slips through a check that reports everything green.

Here is a real shape of the failure. A rule forbids publishing one disputed number. The source constant that holds that number respects the rule perfectly: it is never printed directly. But four lines later, a second constant is derived from it (a rounded copy, a formatted copy, whatever) and that derived value gets printed across a homepage, a footer, and a hero banner, repeated on 66 pages. Three separate automated checks all stayed green, because none of them compared the printed value against the forbidden one. They checked syntax, encoding, and structure. None of them checked meaning.

A close cousin of this bug is the keyword check: a rule that says "the document must mention X." That check is trivial to satisfy by writing something false about X, since the check only looks for the keyword, never for whether what surrounds it is correct. Whatever gets added to satisfy a form check has to be verified at the source of truth, not just verified as present on the page.

The fix is not a smarter form check, it is a different check entirely: forbid the derivatives, not just the original constant, and run the check against the FINAL built artifact (the actual HTML that ships, not the source file), reading its list of forbidden values from the same data file that flagged the number as disputed in the first place. Never describe a form check as a correctness check to anyone who has to trust the result: a deliverable can pass every single scan available and still be commercially or legally wrong.

Mass replace acts on strings, not on meaning

A mass replace (a scripted search that swaps every occurrence of one string for another across many files) treats every match as identical. It has no idea that the same sequence of characters can mean two completely different things depending on where it sits. The more common a pattern is, the more likely it means two things somewhere in your project, and a batch tool will happily rewrite both.

One real case: a batch replace updated a business figure everywhere it appeared. It also rewrote the exact same numeric string inside a legal clause, where that number meant a claim filing deadline, not the response time promise the replace was meant to fix. Nothing about the string looked different. Only the surrounding sentence gave away that it meant something else.

The opposite failure is just as common and much quieter: a scripted substitution can find nothing to replace and still report success. If the target attribute or tag it was looking for is missing from a particular file, most tools treat "pattern not found" as a normal, non-error outcome. In one case, this left a visual element invisible on the page while the exit code (the number a program returns to say whether it succeeded, zero usually meaning success) looked perfectly clean.

Two habits close both gaps. Before running a mass replace, copy the affected files somewhere safe and read the diff (the list of exact changes) file by file, paying special attention to files nobody was thinking about when the rule was written, and simply exclude legal or contractual text from automated passes. After running it, verify the change actually happened by counting how many times the NEW pattern now appears, never by trusting the exit code alone, and prefer tools that fail loudly when the thing they were supposed to anchor onto is missing.

Ship the generator and its destination check together

Generating a reference and deleting an object are mirror images of the same problem. A generator can produce a reference that points at nothing. A deletion can leave references pointing at something that no longer exists. Both look complete from the outside. Both need a dedicated check, and that check is worth as much as the thing it watches.

Take a redirect (a rule that sends a visitor from an old web address to a new one). A generator produced 237 redirect rules with nothing verifying that the destinations actually existed. The output file looked complete: 237 rules, correctly formatted, ready to deploy. But a redirect pointing at a dead page is worse for search ranking than no redirect at all, since it actively sends visitors and search engines into a wall. A separate destination checker, one that visits every target, flags redirect chains, and flags duplicated sources, found zero problems on a later pass. That zero only meant something because the check existed. Without it, "the generator ran and produced 237 lines" would have looked exactly the same whether the destinations were real or not.

This generalizes well beyond redirects: sitemaps, RSS feeds, hreflang tags (markers that tell search engines which language version of a page to show), structured data, and image catalogs all share the same shape. Whenever you ship something whose whole job is to point at something else, ship its destination-witness alongside it, in the same commit, not as an afterthought.

The mirror case is deletion. After 56 videos were deleted from a channel, its playlists still held 48 dead entries, because the platform does not automatically remove a deleted item from every list that referenced it. Those dead entries also quietly skewed an unrelated sort by view count. After any bulk deletion, ask explicitly what REFERENCED the deleted objects (indexes, sitemaps, internal links, join tables in a database, caches) as a separate question from whether the objects themselves are gone. Deleting the object is the easy half of the job.

A guard that exempts itself is not a guard

A rule that only works if someone remembers it, invokes it, or that turns itself off when a certain word appears, is not really a rule. It is a hope. This lesson covers three degrees of the same problem, from a rule that is merely documented, to a guard that actively disarms itself, to a guard that is technically working but too quiet to matter.

The first degree is the most common: a rule that lives only in documentation or memory. A design rule guaranteeing a minimum touch target size (how big a button or link needs to be so a finger can hit it reliably) was documented correctly and understood by everyone, yet manually applied on only 2 of 155 possible spots in the live product. A separate security check was temporarily disabled "by discipline, we will remember to turn it back on," and stayed off until a completely unrelated review caught it. Robustness comes in a clear order: make the rule the default (a global style, a function's default value, so following it takes zero effort), or mechanize it into a blocking guard that is actually tested by attempting the forbidden action and confirming it gets blocked, or, at the very least, measure where the rule is not being applied so the gap is visible.

The second degree is worse, because it looks like a real guard: a mechanized check that exempts itself under a condition. A delivery gate scanned messages for two forbidden patterns, but disabled itself whenever the message contained certain keywords, meant to allow people to discuss the rule without tripping it. A message that literally announced "I am respecting the rule" while containing both forbidden patterns passed silently, precisely because it contained the exempting keywords. This is not a coincidence: a message that talks about a rule is exactly the kind of message most likely to also contain the pattern the rule forbids. Exempting by vocabulary works against the guard by design. Exempt by structure instead (a code block, an explicit delimiter that is hard to produce by accident), and before shipping any guard, actively try to write the one sentence that would disarm it.

The third degree is subtler still. A guard can be technically present, fully mechanized, and still lose. A permanent instruction, loaded into every single session, produced zero visible output for months, and quietly lost against a competing instruction that announced itself loudly at every startup. Being loaded is not the same as being noticed. A standing rule needs to manifest at the moment it matters, not merely sit correctly in a file somewhere.

Validate what the user opens, not an intermediate step

A validation only proves the state of whatever you actually validated, nothing more. Every step between that check and the moment a real person opens the real thing can quietly break something that was correct a moment earlier. The habit worth building is simple to state and easy to skip under time pressure: identify exactly what the user will open, then validate that exact thing, with the tool that actually opens it.

A concrete failure: a photo crop was fixed, then validated by taking a screenshot of the HTML source that generated the page, and announced as delivered. But the file the user actually opened was a PDF, generated from that HTML by a separate step, and that step introduced its own defect. The original problem was fixed in the source and still visible in the PDF, because the PDF was never the thing that got checked.

The same trap shows up around APIs and caches. After writing a change through an API, reading the same API back is not proof the change took effect: the read path can be served from a stale cache (a stored copy of a response, kept around so the system does not have to redo the work every time), which can return the old value even though the write succeeded. The only real proof is checking the externally observable output, the thing an outside visitor actually sees.

Caches cause a second, quieter version of the same problem after a rebuild. If validation does not deliberately break the cache first (adding a fresh query parameter to the URL, forcing a hard refresh, or checking against a server configured to skip its cache), it can validate the previous version without anyone noticing. And if a resource is rebuilt but served under the exact same file name, every visitor's browser can keep the stale version for the entire lifetime of its own cache, even though the underlying file on the server is already fixed. A genuinely new version needs a genuinely new file name, not just new content behind the old one.

Proven, or merely not yet disproven

Every automated check answers one narrow question. A type check proves the code compiles. A test suite proves the code paths it exercises behave as written. A headless preview (a screenshot or a simulated run of an app with no human touching a real screen) proves the components mount without crashing. None of that proves the product actually works for a real person holding a real device.

This is the gap between two very different states, and courses that teach testing usually blur them under one word: tested. The honest split is: proven means you ran a specific command and can cite its output. Not yet disproven means nothing caught a problem so far, which is a much weaker claim: a green preview or an absence of console errors only means the checks you ran did not happen to reveal a defect, not that none exists.

A real batch of deliveries makes this concrete. Seven batches shipped with every automated signal green: type checks passing, full coverage across every language version, a headless preview reviewed and clean. The person who actually installed the app on a real device found problems within minutes, none of which any check had caught: general sluggishness that only shows up under a real touchscreen and real memory pressure, a setting configured for the wrong physical hardware, and a contact button that dialed a landline number no messaging app could ever reach. Every one of those defects was invisible by construction to a headless run, because a headless run has no finger, no screen, and no phone line.

The fix is not to add more automated checks, since this class of defect resists automation by nature. The fix is honesty in how you report status. Before calling anything done, write down explicitly what was proven, with the command and the cited output line, and what remains merely not yet disproven, such as a clean preview or the absence of console errors. Then name upfront what can only be settled on the real device or by the real user, so nobody mistakes silence for a verdict.

Count the whole population, and name the measure by what it captures

Two small habits protect against a surprising amount of bad analysis: count everything before you recommend a change, and write down exactly what your measurement captures before you name it. Skip either one and a confident, well-meaning conclusion can be flatly wrong while looking rigorous.

The first trap is sample bias: noticing a few cases and generalising from them without checking the rest. In one real case, someone spotted four files that broke a naming convention and drafted a sweeping recommendation to abandon that convention entirely, since it clearly was not being respected. Measuring the full population first told a different story: 32 of the 43 relevant items already followed a coherent, deliberate convention. The real defect was narrower and unrelated to the convention itself. A convention already followed by a large majority is a decision someone made on purpose, not an accident waiting to be corrected. And someone simply agreeing to move forward with a plan validates the authorization to act, it never validates that the underlying diagnosis was right.

The second trap is a mislabeled measurement: a number that is technically correct but named after the wrong thing. A chart titled "workday" was actually measuring the gap between the first and last message sent to an assistant tool in a day, silently excluding every hour spent away from the keyboard. In a different case, a note read someone's use of capital letters as a sign of rising frustration, when the real driver was just typing faster under time pressure. A wrong number gets noticed and corrected eventually. A wrong label is worse: it quietly contaminates every conclusion built on top of it, and an outside reader has no way to catch it just by looking at the chart.

Both habits reduce to one discipline. Before recommending a change to a convention, count the whole population, not the cases you happened to notice. Before titling a chart, a dashboard tile, or a column, write in one plain sentence what the underlying instrument literally captures, then check that the title says exactly that, and nothing more.

Agents silently drop your accented characters

Agents routinely strip locale-specific accented characters, the marks like an acute accent or a cedilla that many languages (French, Spanish, German among others) need to spell words correctly, even when explicitly told to keep them. The result stays perfectly valid, readable text: nothing crashes, nothing looks broken, and the missing marks are invisible to a casual read. This makes the failure genuinely dangerous: it passes every check that only looks for something obviously wrong.

The most common way people try to catch this is a simple check: does the document contain accented characters at all. That check has a hidden expiry date. It stays green forever the moment a document switches language, because it is still searching for one language's specific set of marks in text that has moved to another language, or to plain code. A useful check has to match the document's actual current language, get updated the moment that language changes, and require a real numeric floor (a minimum density of marks per paragraph) rather than being satisfied by "at least one occurrence" somewhere in the whole file.

A second trap sits right next to the first: a locale-correction pass must never touch code. Identifiers, attribute values, and tag names are frequently ordinary words in the target language, and by their form alone they are indistinguishable from prose. A script that blindly adds accents wherever a word looks like it is missing one can silently rewrite a variable name or an attribute value, breaking the program while looking like a harmless spelling fix. Corrections belong only in content files: the text a reader actually sees, never the code that renders it.

The most reliable detector does not try to auto-correct anything. Instead it flags two kinds of anomaly and hands the location to a human or a narrowly scoped sub-agent: a density anomaly (a line full of ordinary function words in one language with zero of that language's accented characters, which is statistically almost impossible in real text) and an internal inconsistency (the same word spelled two different ways inside one document). To verify that a correction pass actually worked and did nothing else, run a byte-for-byte diff that strips the accented characters from both the original and the corrected version and requires them to match exactly. Any reformulation, any added or removed word, shows up immediately, at its exact location in the file.