Home / Several agents and adversarial checking

Capping a judge loop before launching it

Set the cap before the first pass, in the batch's criteria: a maximum number of judge passes and of agents. The stopping criterion belongs to the judge, whom you do not control, and as long as it finds something the loop continues and its cost climbs. When the cap is reached, fix the sourced major findings and close.

A typical verification loop corrects a deliverable, has it judged, corrects again, and repeats until the judge answers OK. Its stopping criterion belongs to the judge, whom you do not control: as long as the judge finds something, the loop continues and its cost climbs, without any single turn looking excessive.

Set the cap before starting

The cap is written into the batch's criteria, before the first pass: a maximum number of judge passes and a maximum number of agents. When the cap is reached, you fix the major findings that rest on a source, list the rest as known minors, and close. An extra pass remains possible, but it is requested with its measured cost, not decided by default. The following demonstration builds its own findings files and applies this cap:

mkdir -p judges-demo && cd judges-demo
printf 'major: cited source missing\nminor: title too long\n' > pass1.txt
printf 'major: the rule added in pass 1 contradicts point 3\n' > pass2.txt
printf 'major: the rule added in pass 2 is ambiguous\n' > pass3.txt
cat > boucle.sh << 'EOF'
#!/bin/sh
PLAFOND=2
passe=1
while :; do
  majeurs=$(grep -c '^major' "pass$passe.txt")
  echo "pass $passe : $majeurs major line(s)"
  if [ "$majeurs" -eq 0 ]; then echo "stop: no major finding"; break; fi
  if [ "$passe" -ge "$PLAFOND" ]; then echo "stop: cap of $PLAFOND passes reached, the rest is listed as known minors"; break; fi
  passe=$((passe+1))
done
EOF
sh boucle.sh
printf 'findings: 0\nfiles read: 0\n' > verdict.txt
lus=$(sed -n 's/^files read: //p' verdict.txt)
if [ "$lus" -gt 0 ]; then echo "verdict accepted"; else echo "empty verdict with no proof of reading: redo it"; fi

pass 1 : 1 major line(s)
pass 2 : 1 major line(s)
stop: cap of 2 passes reached, the rest is listed as known minors
empty verdict with no proof of reading: redo it

The loop stops at the second pass while major findings remain: the cap decides, and the remaining finding is recorded instead of being chased. The last test applies the last idea of this lesson, explained below.

The arbiter who adds rules

When an arbiter settles each finding by adding a rule to the document, the next round often finds its major defect in the rule just written. That is what passes 2 and 3 of the test file show: each concerns the rule added in the previous pass. Before adding, check whether an existing rule covers the case, then whether removing the mechanism is enough; add only as a last resort, with the shortest fix. If the third major finding in a row concerns the same mechanism, go back to the previous state.

An empty, fast verdict is redone

A judge that returns "approved, zero findings" almost at once on a large batch may have read nothing: a badly passed file path is enough to produce that result. Accept a verdict with no findings only if it carries proof of what was read, such as the number of files and quotations. The demonstration's last test refuses the verdict whose files-read counter is zero. A refused verdict is relaunched with distinct lenses, and the fixes are themselves judged, since a fix can create its own defect. The lesson on judge panels describes these lenses.