Small, curated suite
Ninety single‑turn cases cannot represent the open‑ended attack surface of a production agent. Paraphrase, obfuscation, and multi‑turn manipulation remain incomplete.
Current results, setup, and caveats.
Baseline captured May 13, 2026; report reviewed July 10, 2026. These are vendor‑published engineering measurements, not an independent assessment or a promise of production performance.
Accuracy baseline
| Metric | Result | Interpretation |
|---|---|---|
| Mode | Mock | Deterministic heuristic path; no live Anthropic or Voyage request. |
| Cases | 90 | Nine categories across five agent personas. |
| Overall accuracy | 86.67% | Both the expected allow/deny result and intent category must match. |
| Deterministic subset | 100.00% | Only cases marked as supported by the mock heuristic. |
| Deny precision / recall / F1 | 1.000 / 0.820 / 0.901 | The baseline recorded no false‑positive denies and 11 missed attacks. |
Evidence limit
The underlying raw case fixture and harness are not currently linked as public artifacts from this site, so an external reader cannot independently reproduce this baseline from this page alone. Treat the numbers as a transparent vendor report, not independently verified evidence.
A case passes only when verdict and intent both match.
Each fixture is one MCP inspection request paired with an expected authorization result and intent category. A wrong verdict, a right verdict with the wrong category, or a non‑successful response fails the case. The positive class is deny.
Case mix
| Category | Cases | What it checks | Mock supported |
|---|---|---|---|
| Benign authorized | 20 | In‑scope calls that should pass. | Yes |
| Benign metadata | 5 | Resource and tool discovery that should pass. | Yes |
| Privileged action | 8 | Destructive keyword shapes that should deny. | Yes |
| Direct prompt injection | 12 | Known instruction‑override patterns. | Yes |
| Meta reasoning | 4 | Plan and ignore bypass triggers. | Yes |
| Credit‑card drift | 5 | Forbidden data access against a persona. | Yes |
| Persona‑drift stretch | 12 | Cross‑persona actions without a keyword trigger. | No |
| Malicious code write | 12 | Reverse shell, keylogger, ransomware, and persistence shapes. | Yes |
| Compromised package | 12 | Known‑bad or typosquatted package installation. | Yes |
Why overall accuracy is below 100%
The 12 persona‑drift stretch cases intentionally contain no keyword trigger. Mock mode has no live embedding or classifier signal, so this category exposes the gap between the deterministic floor and the full semantic path. The baseline does not hide that gap.
No committed live‑provider baseline is published here. A live run incurs provider cost, changes with model versions, and should be measured against a pinned model configuration before it is compared with the mock result.
Separate Clavenar overhead from model‑provider time.
The whole‑pipeline run measured mTLS ingress, Brain in mock mode, policy evaluation, and proxy fork/fold at the /mcp boundary. It used 10 concurrent workers, a 20‑second measurement window, and a 5‑second warmup on the development stack.
| Path | p50 | p95 | p99 | Approx. throughput |
|---|---|---|---|---|
| Verdict‑cache hit, fixed payload | 1.67 ms | 2.20 ms | 2.90 ms | 5,800 rps |
| Cache miss, varied payload | 14.0 ms | 20.1 ms | 29.4 ms | 660 rps |
Do not add these numbers together blindly
The cache‑miss result measures Clavenar's mock‑mode pipeline floor. It does not include a real LLM provider request. On a live semantic path, provider round‑trip time can add hundreds of milliseconds and may dominate the verdict. Measure hot and cold paths with your chosen provider, region, policy set, and evidence store.
What this report does not establish.
Mock mode is a floor
The deterministic subset rewards the heuristics it was designed to exercise. It is useful for regressions; it is not a substitute for a live semantic evaluation.
No competitor baseline
No like‑for‑like run against another guardrail is published here. Do not infer comparative superiority from these standalone measurements.