Workstream W4 · CTC Research Lab

Multi-Agent Safety Evaluation

Safety properties established for a single agent do not compose: a fleet of individually-safe agents can still collude, cascade one agent’s error into system-wide failure, or carry a self-replicating prompt injection from one agent into the next. W4 turns that premise into a measurement programme – the object under test is the fleet, and the risk lives on the inter-agent channels, where output-oriented benchmarks are structurally blind.

Research agendataxonomy operationalised · metrics & testbed designed · falsifiable targets, no risk numbers claimed

↓ Working paper (PDF) CTC Research Lab PI write-up ↗
++
k(k−1)
Directed channels in a k-agent fleet – the interaction surface where combinatorial risk lives.
Premise
~41.7%
Share of violations output-only audits miss, because they travel on inter-agent channels, not in final outputs.
Literature
0
Risk numbers claimed in this agenda – inventing them would be the exact dishonesty this work exists to prevent.
By design
++
Context

Safe agents don’t compose
into safe fleets.

Single-agent evaluation scores the nodes of a system. Wire k agents together and up to k(k−1) directed channels open between them – and the failure modes that matter are properties of those channels, not of any node in isolation.

This is not a speculative claim. Injecting adversarial traits into one agent drives dangerous conduct to propagate through an otherwise-safe group, bypassing standard input filters because the danger emerges in multi-turn dialogue rather than the initial prompt. Prompt injections can self-replicate agent-to-agent, spreading with epidemiological dynamics. And a full-stack privacy audit finds that inter-agent message channels leak far more than final outputs – output-only audits miss a large share of violations because the violation travels on the channel, not in the answer. The measurement problem is therefore concrete: to see multi-agent risk at all, an evaluator has to observe the edges.

The object under test is the fleet – risk lives on the edges A1 A2 A3 A4 A5 Nodes = agents single-agent benchmarks score only these – a safe pair is not guaranteed Edges = channels k agents → up to k(k−1) directed channels – risk is combinatorial, benchmarks blind
Figure 1 – the failure modes this agenda measures are properties of the interaction, not any single model: invisible when agents are scored alone.
++
Task Force 1

A taxonomy operationalised,
not invented.

TF1 does not add another taxonomy. It operationalises the existing ones – Hammond et al., MAST, ESRH – into the interaction-emergent subset the metrics target, so each mode becomes a concrete, testable scenario bound to a metric.

Nine interaction-emergent failure modes, three families Coordination Miscoordination conflicting sub-plans · wasted work Error cascade one agent’s error becomes ground truth Deadlock / livelock mutual waiting · motion, no progress Adversarial dynamics Collusion unsafe plan reinforced · can be covert Injection propagation a prompt injection self-replicates Emergent goal drift fleet objective diverges from intended Delegation Unsafe sub-goal a handoff loses a safety constraint Capability leakage privilege escalates across handoffs Oversight evasion work routed around the human gate
Figure 2 – maps onto Hammond et al. (3 modes + 7 factors), MAST (14 modes) and ESRH. The contribution: each mode becomes a scenario bound to a metric (TF2) and an instrumented run (TF3).
++
Task Force 2

Numbers from behaviour,
not self-report.

Each failure mode needs a number that means something – computed from what the agents actually did in an instrumented run, reported with variance across seeds, and subject to the discrimination rule: a metric on which every run scores identically measures nothing and does not ship.

Cascade depth downstream agents on a seeded error E B C D depth = hops before caught Injection reach agents changed by one poisoned input P 2 3 4 5 self-replicating spread – read from the graph, not outputs Delegation safety does a constraint survive every handoff? L S1 S2 ✕ constraint dropped at S1→S2 emergent risk = (fleet failure rate) − (aggregate single-agent baseline)
Figure 3 – each metric is the emergence-gap instrument for its mode. A low score on a narrow scenario set is evidence against those specific failures – never a safety certificate.
++
Task Force 3

The observer is
the contribution.

Taxonomy and metrics become measurement in a testbed that runs a real multi-agent scenario inside a hardened sandbox – with an observer layer recording the full message graph and every tool call, so the metrics are computed after the fact from what actually happened.

Pinned scenario semver + hash · synthesis = W2 Multi-agent run sandbox = W1 · + observer Per-mode metrics cascade · reach · delegation Risk profile re-derivable artifact Reuses standing infrastructure – sandbox = W1 · scenario synthesis = W2 · single-residency = W3. The new component is the observer.
Figure 4 – the pipeline from pinned scenario to audit artifact, entirely on the sovereign single-GPU setup.
Why an observer, not an output audit

Multi-agent violations travel on inter-agent channels – output-only audits miss a large fraction of them. The observer records the complete message graph, every tool call, and taint/provenance metadata, so injection reach traces to its source. Metrics are computed from what agents actually did – never from a model’s self-report.

++
Falsifiable targets

Five success criteria, G1–G5.

Following the discipline of the companion papers, a “shipped” result is defined in advance – marked predicted until met – so the agenda cannot quietly redefine success later.

G1 Emergence gap measured For each failure mode, report the fleet-versus-aggregate gap with variance across seeds. A scenario ships only if its metric discriminates – not identical across runs. Falsified if: the gap is indistinguishable from zero across scenarios – single-agent evaluation would then suffice, and that outcome would itself be a publishable result. Predicted · load-bearing
G2 Full observability Every reported number is computable from the recorded message graph and tool calls, never a self-report; provenance/taint tracing is validated end-to-end on a seeded injection. Falsified if: any reported number cannot be recomputed from the recording, or taint tracing fails to locate the seeded source. Predicted
G3 Reproducibility The risk profile is re-derivable months later from the pinned scenario (semantic version + content hash) – an audit artifact, not a vibe. Falsified if: an independent re-run from the same pinned scenario does not reproduce the profile. Predicted
G4 Sovereign runnability The whole harness runs on the single-GPU, air-gapped setup of the companion workstreams, so an independent team can reproduce it without frontier compute. Falsified if: any required component exceeds the single-GPU, air-gapped envelope. Predicted
G5 Honest scope Results are reported as evidence against the specific failures provoked, never as a safety certificate; scenario coverage and its gaps are stated with every result. Falsified if: a result is presented without its coverage statement – or read as a licence to deploy. Standing rule

G1 is the load-bearing claim: if the emergence gap cannot be shown to differ from zero, the agenda’s own premise is falsified – and that outcome would itself be a useful, publishable result. The criteria run on the same infrastructure the program already operates: W1’s hardened sandbox, W2’s scenario synthesis, W3’s single-residency discipline.

++
Applied

A go / no-go number
before deployment.

A research agenda still has to answer “and then what?” – the concrete deployment the three task forces serve.

Fleet risk measurement, end to end
  • Situation. An organisation wants to ship a fleet of interacting agents – research and coding agents that delegate to each other and share tools. Single-agent benchmarks say each one is fine, but there is no principled way to know the fleet won’t collude, cascade an error, or carry a prompt injection from one agent into the next.
  • Task. A defensible, reproducible measure of multi-agent-specific risk before deployment – a concrete go / no-go bar rather than a subjective judgment that the system “seems safe”.
  • Action. Pin the fleet’s scenario (semver + hash); run it in the hardened sandbox with the observer recording the full message graph and tool calls; compute cascade depth, injection reach, and delegation safety with variance across seeds; report each as evidence against its named failure mode, with coverage stated.
  • Result. A re-derivable risk profile that quantifies the emergence gap for this fleet on this scenario set – computed from observed behaviour, re-runnable by an auditor months later, produced entirely on sovereign single-GPU hardware. Not a safety certificate, but the first defensible number where there was previously only a judgment call.
++
Limitations

Four bounds, stated up front.

Limitations & threats to validity
  • Scenario coverage is the ceiling on the claim. Every metric is evidence only against the failures its scenario set was built to provoke; an unmeasured interaction pattern is an unmeasured risk. The honest reading of any result is “safe against these probes” – not “safe”.
  • The observer changes what it measures. Recording the full message graph is intrusive, and a sufficiently capable agent could in principle detect instrumentation and behave differently under observation. The mitigation – a passive, out-of-band observer – is real but partial; the threat is itself a research question.
  • The sovereign constraint bounds fleet scale. A single 32 GB GPU under single-residency limits concurrently-served agents; very large fleets must be studied through time-multiplexing or scaled-down proxies, and emergence-gap measurements may not extrapolate cleanly to production fleet sizes.
  • The agenda measures – it does not mitigate. A risk profile is not a safe fleet, and a low measured gap on a narrow set must not be read as a licence to deploy. The deliverable is a defensible measurement and a go/no-go input, not a guarantee.

Status & reproducibility. This is a research agenda: the taxonomy is operationalised, the metrics and testbed are designed, and the criteria of G1–G5 are the falsifiable targets to be met – no risk numbers are claimed. Scenarios are pinned by semantic version and content hash; every metric is computed from the observer’s recording; runs execute in the W1 hardened sandbox on the single-GPU, single-residency setup of W3, and the risk profile is released as a re-derivable audit artifact.