Sub-agent multiplexing
Agents spawn helpers, pass them state, and gather results through a worker pool and durable background tasks. This was never the gap, so nothing was rebuilt.
15 modules · worker pool · durable tasks
Autonomous Infrastructure
Agents that recover on their own,
Agents that recover on their own,
behind switches you control.
Snippbot agents now share one memory of what is true, notice when a tool call fails, work out why, and can retry safely, with every autonomous action off until you turn it on. Built in nine phases, audited in thirteen, and every finding is counted in public.
Four pillars, judged against the code
We started from four requirements for agents that can run without a human watching every step, and audited each one against the code that exists, not a feature list. Two were already mature. The other two were the real work.
Shared blackboard
One durable register of facts, intents, events and counters that every agent can read. Each entry belongs to a scope, every write is screened for secrets, and content written by others comes back marked untrusted.
7 scope kinds · 10 verbs · leases · DLP
Hot-plug tool matrix
Packages, MCP servers and skills load at runtime, and every tool call returns its output, errors and exit code through one wrapper. That uniform shape is what made fault handling possible.
90 core tools · 0 duplicates
Fault-adaptive recovery
A failed tool call is classified, checked against its preconditions, and either retried within a budget or handed to a person with the evidence attached. Nothing acts until you switch it on.
10 failure classes · 4 gates · kill switch
Fault-adaptive recovery
What happens when a tool call fails
A failure is never retried blindly. Each step can stop the loop, and anything the system can't fix safely goes to a person instead of being guessed at.
1
Observe
The tool's output, errors and exit code are captured exactly as returned.
2
Classify
A closed classifier names one failure class, reading error output before normal output, with bounded cost on huge logs.
3
Verify
Precondition checks confirm a retry could actually work. "Unverifiable" counts as no.
4
Guard
A fixed table maps each class to the only allowed changes. A write guard protects code, git and databases, whatever the letter case.
5
Budget
Each attempt is reserved against attempt, per-run and daily dollar limits before it runs. An unpriced tool is refused.
6
Retry or escalate
A safe retry runs within a 60-second deadline. Anything else becomes a hand-off with the evidence, kept out of transcripts.
permission_deniedmissing_dependencybad_argumentsnetworkrate_limitoutput_contractmissing_pathauth → escalateport_conflict → escalateunknown → escalate Classes join the closed set only with live evidence. missing_path was added after it appeared in 31% of 30,154 real failure records, and port_conflict after 105 real occurrences. Choosing a different port would silently break every client, so it escalates rather than mutating.
Shared blackboard
One memory of what is true, with an owner on every line
Before this, what one agent learned stayed in its own run. Now agents share facts ("this API key rotated"), claim work ("I'm rebuilding the index"), and record events, all in one local database that never leaves your machine.
- Screened on every write. Every field an agent supplies passes data-loss prevention: secrets are redacted, private keys are refused outright, and values are capped at 16 KiB.
- Untrusted by default across scopes. If a read returns anything an agent didn't write itself, it's scanned for prompt injection and wrapped as external content before the model sees it.
- Respects your modes. Approval Mode pauses blackboard writes for your OK, and Plan Mode refuses them.
- Leases that really expire. A crashed agent's claim is reclaimed automatically. Retention cleanup is preview-only by default, and no agent can apply a sweep across scopes it doesn't own.
Safety first
Every autonomous action starts switched off
Acting is opt-in twice: the feature switch and a funded budget. Out of the box the budget is zero attempts and $0, and only ten read-only tools can ever be retried.
Observe failures
SNIPPBOT_RESILIENCE_OBSERVEClassifies failed tool calls and records them for review. It never retries, changes or blocks anything, and stays off the daemon's event loop.
Act on failures
SNIPPBOT_RESILIENCE_ACTLets the supervisor retry. It also needs an attempts limit and dollar ceilings above zero, and the tool must be on the read-only allowlist.
Kill switch
~/.snippbot/RESILIENCE_ACT_OFFCreate this file and acting stops before the next attempt. It's checked before every single attempt, and a file that can't be read counts as tripped.
Daily cleanup report
SNIPPBOT_BLACKBOARD_SWEEP_REPORTA daily dry run that reports what retention would remove. It never deletes; applying a sweep is a deliberate manual step.
Job hygiene
Long-running work that cleans up after itself
Background jobs leave folders behind. Retention removes old ones, and treats "never delete something still needed" as a harder rule than "free the disk".
- Never removes undelivered results. If a job's completion callback was refused, its folder holds the only copy, so it's kept.
- Checks the process, not just the clock. A live process group is never "crashed". A reused process ID is caught by comparing the process's start time with when the job ended.
- Run by hand, not on a schedule. Retention is a command you run; nothing deletes in the background.
Audited in public
Built in nine phases. Audited in thirteen.
Each phase was audited again after it was built, by re-measuring the code rather than trusting the earlier write-up. A finding only closes with a code reference or a measurement. One of our own findings turned out to be wrong, so it's listed as withdrawn, not fixed.
159findings recorded
131closed with evidence
32 / 32fault-injection checks
15,900+core tests on every change
| Phase | What it covers | What the audit caught | Findings | Closed · open · info |
|---|---|---|---|---|
| Build | Original audit and build of every phase (Nov 3)Four-pillar review, first implementation | One of our own High findings was wrong and is listed as withdrawn | 54 | 45 · 2 · 6 · 1 withdrawn |
| 0 | Test environmentTests run the code under review | Tests were quietly exercising a different checkout's code | 5 | 4 · 1 · 0 |
| P0 | Blackboard registerScopes, screening, tool | Writes bypassed Approval and Plan Modes; size limits were missing | 9 | 7 · 0 · 2 |
| P1 | Sweeper and leasesRetention, lease reclaim, CLI | Any agent could purge every scope | 6 | 5 · 1 · 0 |
| P2 | Job-folder retentionCleaning up background jobs | A refused callback counted as delivered; a running job looked crashed | 7 | 5 · 1 · 1 |
| P3 | Classifier and mutation tableFailure classes, write guard | The write guard could be bypassed with letter case | 10 | 8 · 1 · 1 |
| P4 | Supervisor loopCap, deadline, hand-off | A tight retry cap dropped the hand-off; secrets reached the event bus | 10 | 10 · 0 · 0 |
| P5 | Fault-injection harness23 cases + 9 checks | The harness could delete a home folder and still report a pass | 11 | 11 · 0 · 0 |
| P6 | Tuning on real failures30,154 real records mined | Found a production integration that had been silently down for two weeks; since fixed | 9 | 7 · 0 · 2 |
| P7 | Live observationHook in the tool path | Register writes stalled the daemon about 8 s per failure; moved off the event loop | 8 | 7 · 0 · 1 |
| P8 | The acting halfRetries, budget, kill switch | The daily spend ledger now survives a restart | 11 | 8 · 0 · 3 |
| P8-4 | Escalation surfaceRegister intent, tool note, chat card | Raw failure text, keys included, could reach the card; now screened | 10 | 8 · 2 · 0 |
| Cleanup | Dead code removedFour unreachable modules | A release smoke check had been failing every base install | 5 | 3 · 1 · 1 |
| Merge | Integration and CIRebase, Linux CI | A time-zone test left Linux's clock wrong for the next test; proven in a container, fixed | 4 | 3 · 1 · 0 |
| All findings | 159 | 131 closed · 10 open · 17 info · 1 withdrawn | ||
closedopeninformational
Where it stands
What we're not claiming yet
The shared blackboard is live for every agent. The recovery system is built, tested and wired into the tool path, but it's off by default and limited by design. Automatic retries have run end to end only on an isolated test daemon with a scripted model. Until a real model has driven a retry in production, we won't call this verified.
Real-model acting turn
The first production retry, driven by a real model on an allowlisted tool, with its cost recorded against the budget.
Escalations in your chat
Hand-offs are written to the register and attached to the tool result today. The chat card needs a link from a project run to its conversation before it can reach you.
Wider reach
Acting covers project execution with non-native models. Chat turns observe only. Extending it is a deliberate decision, not a default.
Audit record current as of Nov 5, 2026