Home / Several agents and adversarial checking

Succeeded, failed, interrupted, no trace

When a cutoff stops a queue of tasks, a log kept outside the process, written before and after each task, separates four states: succeeded, failed, interrupted midway, and no trace of an attempt; confusing them leads you to rerun what may already have acted, or to declare failed what nobody has examined.

Take a typical case. A chain of checks processes a queue of tasks one after another. A usage cap hits in the middle of the queue and stops the supervisor process along with the agents it was driving. On restart, the supervisor marks as failed every task with no result, including those that no agent had started.

Four states, because a cutoff creates two

A system that only knows succeeded and failed files under failed two cases that the cutoff produces. The interrupted task has started: it may have acted in part, and it gets inspected before being rerun. The task with no trace is rerun, once it has been checked that no other process picked it up. Writing the "running" line before starting is what separates them: without it, an interrupted task looks like a task never touched.

The absence of a line establishes one bounded thing, as the lesson on what a rigorous check does not see reminds us: this log carries no trace of an attempt, and says nothing about another process that may have taken the task.

The demonstration

The two scripts are created in a throwaway folder. The first writes six tasks into queue.json, then simulates a cutoff during the fourth. The second sets this starting list against the log: it is the list that reveals the tasks with no trace.

mkdir -p demo-queue && cd demo-queue
cat > process-queue.js << 'EOF'
// Builds the starting queue, then processes it, logging before and after each task.
const fs = require('fs');
const queue = ['task-1', 'task-2', 'task-3', 'task-4', 'task-5', 'task-6'];
fs.writeFileSync('queue.json', JSON.stringify(queue));
fs.writeFileSync('log.jsonl', '');
const results = { 'task-1': 'succeeded', 'task-2': 'failed', 'task-3': 'succeeded' };
for (const id of queue) {
  fs.appendFileSync('log.jsonl', JSON.stringify({ id, state: 'running' }) + '\n');
  if (id === 'task-4') process.exit(1); // simulated cutoff during the fourth task
  fs.appendFileSync('log.jsonl', JSON.stringify({ id, state: results[id] }) + '\n');
}
EOF
cat > read-log.js << 'EOF'
// Rereads the starting queue and the log, and sorts each task into four states.
const fs = require('fs');
const queue = JSON.parse(fs.readFileSync('queue.json', 'utf8'));
const last = {};
for (const line of fs.readFileSync('log.jsonl', 'utf8').split('\n')) {
  if (line) { const e = JSON.parse(line); last[e.id] = e.state; }
}
const classes = { succeeded: [], failed: [], interrupted: [], 'no trace': [] };
for (const id of queue) {
  const state = last[id];
  if (state === undefined) classes['no trace'].push(id);
  else if (state === 'running') classes.interrupted.push(id);
  else classes[state].push(id);
}
for (const [name, ids] of Object.entries(classes)) console.log(name + ': ' + (ids.join(', ') || '-'));
EOF
node process-queue.js
echo "exit code: $?"
cat log.jsonl
node read-log.js

exit code: 1
{"id":"task-1","state":"running"}
{"id":"task-1","state":"succeeded"}
{"id":"task-2","state":"running"}
{"id":"task-2","state":"failed"}
{"id":"task-3","state":"running"}
{"id":"task-3","state":"succeeded"}
{"id":"task-4","state":"running"}
succeeded: task-1, task-3
failed: task-2
interrupted: task-4
no trace: task-5, task-6

Exit code 1 signals the abnormal stop. Task 4 carries "running" with no result: interrupted, not failed.

A trace that outlives the process

A tally kept in the supervisor's memory disappears with it. The log therefore lives outside the process: a file as here, a database table or an external message queue would also do. appendFileSync finishes writing the line before moving on: if the process is killed, the lines already written stay in the file. A power cut is another case, because the system may hold the last lines in memory before putting them on disk; an explicit sync, fs.fsyncSync in Node, forces that write.

The lesson on the three failure states separates, among the tasks that returned a result, the recoverable failure that will be retried and the final failure that goes up to a person; this one deals with the tasks the cutoff left without a result.