Anomaly detection

AgentStat can flag three kinds of trouble: an agent that claims to work but produces nothing, one burning tokens far above its normal rate, and one flapping between states. Detection is off by default.

<AgentStat
  agents={agents}
  anomalyDetection
  onAnomaly={(agentId, anomaly) => console.warn(agentId, anomaly.message)}
/>

What it detects

Each anomaly carries the numbers that triggered it, for example stalled 12s while active or runaway output 100/s (4.4× the usual peak of ~23/s).

What doesn't count:

• Bursty but healthy output, output resuming after a pause, and start-up ramps aren't spikes.
• Output returning after a stall isn't a spike, unless it stays at spikeRatio × the agent's normal rate or more for 3 seconds.
• Naturally slow agents (under 0.5 tokens/s) aren't "stalled".
• Quiet time while waiting or complete, or while thinking before output starts, doesn't count toward a stall in the next status.

Severity

Markers are amber triangles for warnings and red diamonds for critical, each labelled with the agent's name, so severity never depends on color alone.

Episodes and callbacks

An anomaly is an episode: it opens once and ends 5 seconds after its condition was last seen, so a retry loop stays one episode. It can escalate from warning to critical but doesn't flicker back.

• onAnomaly(id, anomaly) fires when an episode opens, and once more if it escalates.
• onAnomalyEnd(id, anomaly) fires once when it ends, also when detection is switched off or the agent is removed.
• ref.current?.getAnomalies(id) returns the anomalies active now.
• Screen readers hear new and resolved anomalies through a polite live region; critical ones go to an alert region.

An open spike keeps the baseline it opened against, so a long runaway can't become its own normal. It stays open while the rate is above 1.2× that baseline. After 10 minutes (spikeMaxEpisodeMs) the new level is accepted and the alert ends.

Defaults

Override any of these with anomalyConfig; it's merged over DEFAULT_ANOMALY_CONFIG:

<AgentStat
  agents={agents}
  anomalyDetection
  anomalyConfig={{ stallDurationMs: 20_000, spikeRatio: 2 }}
/>

How it's measured

The library repository has a labelled evaluation, npm run eval:detectors. It replays scenarios through the real store and counts true alerts, false alerts and missed incidents across five sets of scenarios:

• Original: written with the detectors.
• Adversarial: written by a reviewer to find blind spots.
• Holdout: written by a reviewer who hadn't seen the tuning.
• Events: written with the event API.
• Holdout 2: written by a reviewer who saw neither the detectors nor the other sets.

Results for 0.6.0:

The two holdout sets are the honest numbers: about half of their alerts are false. In the first, the false alarms are stalls during long tool calls and orchestrator waits that were reported as active instead of waiting (see the limitations below). In the second, 12 of the 14 false alarms are spikes: on many short or retried runs, on output returning after a silence, and on an orchestrator; the other 2 are stalls on a reasoning model's long thinks. Holdout 2 was first measured at 28.8% precision and 6.6 false alarms per agent-hour; fixes made after seeing it brought it to the numbers above, so it's no longer fully unseen.

Limitations

The detectors only see token rate and status, so keep the status current, including complete when an agent is idle.

• Report waiting. A tool call, an orchestrator waiting on workers, or a user who stopped typing all look like "no output". With events, tool calls set waiting for you; otherwise report waiting (or complete while idle). Waiting reported as active alerts after 10 s, as thinking after 30 s.
• A wait that never ends is only flagged after waitingStallMs. AgentStat can't tell that the workers an orchestrator waits on are gone.
• Requests that differ a lot in speed (one at 80 tokens/s, the next at 150) can briefly read as a runaway after a pause.
• Tool-call loops that only alternate working states aren't flagged as thrash: that alternation is also how healthy agents work.
• Rates computed per small chunk (alternating 0 and high values) can hide runaways. Use events, or send a rate averaged over about a second.
• If you only send rates and never a status, only a stream that goes completely silent counts as a stall. Sending a status even once turns full stall detection on.

Without the chart

useAgentTelemetry runs the same detection without a canvas. For your own pipeline, the detector is a plain function:

import { detectAnomalies } from '@dan-build/agentstat';

const now = performance.now();
const rates = [{ t: now - 12_000, v: 24 }, { t: now - 11_000, v: 0 }, { t: now, v: 0 }];
const statuses = [{ t: now - 20_000, status: 'active' as const }];

const found = detectAnomalies(rates, statuses, 'active', now);
// [{ kind: 'stall', severity: 'warning', message: 'stalled 11s while active', … }]

Importing only detectAnomalies adds about 1 KB.