In healthcare, "good enough" is not a feature — it is a liability. A 90%-accurate agent that flags care gaps, codes diagnoses, or matches patients to trials is not a success. It is one wrong answer away from member harm, a failed audit, or a safety event. The question is never "does the model respond?" It is "can we prove the response before it reaches a member?"
This blueprint describes a reference architecture for building and rigorously testing a portfolio of healthcare AI agents — the kind that sit on top of a payer's analytics platform. The agents are deliberately simple. The engineering that matters is the harness around them: the layer that turns "the model said so" into "we can prove it, or we escalate to a human."
The Problem: You Cannot Unit-Test a Probability
Traditional testing assumes deterministic pass/fail. You assert that expected equals actual and move on. A large language model is probabilistic: the same prompt can produce different words every time, and "correct" is rarely a single string. Point an exact-match assertion at an LLM and you get a red build for a valid paraphrase, and a green build for a confident hallucination.
The shift is from testing to evaluation: scoring outputs against criteria, over a dataset, with tolerances and gates. But evaluation has a trap of its own — most LLM evaluation leans on another model as a judge, and an uncalibrated judge is an unmeasured instrument.
The Core Idea: Rules as the Oracle
The pattern that anchors everything is rules-as-oracle. A deterministic engine computes the correct answer. The LLM agent is a reader and proposer whose output must reconcile to that engine — or escalate to a human. The agent never gets to be the source of truth; it gets to be a fast, fallible first pass over a source of truth we already trust.
The quality gate is the line the agent cannot cross without a human.
This inverts the usual framing. Instead of asking a model to decide and then wondering whether to trust it, we let a provably-correct engine decide, and we measure how often the model agrees, where it diverges, and whether it fails safely when it is unsure.
Four Agents, Four Oracles, Four New Things to Prove
The reference portfolio has four agents. Each is chosen to add a testing dimension the previous ones did not have — so the portfolio, read left to right, is a widening map of how agentic systems fail.
| Agent | What it does | Deterministic oracle | The new thing it forces you to prove |
|---|---|---|---|
| Care-gap assessment | Flags open quality-measure gaps (HEDIS-style) for a member | A rules engine over the clinical record | Grounding — every finding cites a real record |
| Member outreach (chained) | Drafts compliant outreach for the open gaps | A compliance policy as pure functions over the message text | Generated-language compliance + a clean agent-to-agent handoff |
| Risk-adjustment coding | Proposes HCC risk codes from documented conditions | An HCC grouper (code crosswalk + hierarchy + risk score) | No upcoding — never code what the record doesn't support |
| Trial matching | Decides patient eligibility against structured criteria | A three-way evaluator: eligible / ineligible / insufficient-data | Exclusion safety + abstention when data is missing |
A few of these deserve a closer look, because they are where the testing gets interesting.
The Testing Approach: Two Planes and a Gate
Testing an AI application means testing two planes at once.
The deterministic plane is the ordinary software around the model — data loading, schema, APIs, the rules engines themselves. This is normal automated testing: exact assertions, reconciliation, referential integrity. In this reference build, the rules oracle is even implemented twice — once in Python, once in SQL — and the two are reconciled row by row, because the oracle is the thing everything else is measured against and it had better be right.
The probabilistic plane is the model's behavior — its findings, codes, decisions, citations. This is scored with golden datasets and metric gates, not exact strings. Both planes run in CI. A change to a prompt, a model, or a rule re-runs the whole suite, and a failed gate blocks promotion exactly the way a failed unit test or a security scan blocks a deploy.
The Pipeline
The whole discipline lives in one CI/CD pipeline of five sequential stages, each gating the next:
- test — unit and integration tests for the deterministic plane, published as a JUnit report.
- evals — a job per agent runs its oracle → agent → scorers → gates; any red gate fails the build and keeps the gate report as an artifact.
- selftest — the fault-injection matrix that deliberately breaks each agent and asserts the gates catch it (the test of the harness itself).
- security — SAST over prompts, tool allow-lists, and output contracts, plus Promptfoo and garak DAST fired at a running, mock-backed endpoint.
- deploy — a manual, gate-blocked promotion that only becomes clickable once every stage before it is green.
Every Metric Maps to a Named Failure Mode
The discipline that keeps an eval suite honest: write down how the system actually fails, in plain language, before choosing metrics — and make every metric trace to one failure mode.
| Failure mode | The gate that catches it |
|---|---|
| Hallucinated / fabricated evidence | citation validity, coding support, eligibility support |
| Wrong answer vs ground truth | oracle agreement, code agreement, match agreement |
| Missed a required output | gap recall, code recall, match recall |
| Broken output format | schema validity |
| Unauthorized action (excessive agency) | tool-authorization / action allow-list |
| Prompt injection | injection block-rate (zero tolerance) |
| Handoff information loss | handoff coverage |
| PHI leakage | PHI containment |
| Answering when it should abstain | abstention correctness |
| Enrolling an excluded patient | exclusion correctness |
A Suite That Cannot Fail Proves Nothing
The most important test in the whole harness is the test of the harness. For each agent, a fault-injection matrix deliberately breaks the model — makes it drop a finding, cite a record that doesn't exist, code an unsupported diagnosis, obey an instruction hidden in the data — and asserts that the gates catch it. If a broken build ever passes green, that meta-test fails.
The Agents Never Act
Every agent's action space is an allow-list of read-and-notify verbs. Closing a care gap, attesting a code to the payer, enrolling a patient in a trial — none of these are reachable by the agent, by construction. Clean output routes to a human queue; anything suspicious is blocked and escalated. Fail-closed is the default.
Evaluation: Where a Deterministic Oracle Stops, a Judge Begins
Deterministic ground truth is a luxury, and this architecture spends it wherever it can. But it does not reach everything. It can tell you a care-gap status is correct; it cannot tell you the free-text rationale is coherent, or that an outreach message is clear and respectful rather than merely compliant.
That is the one place a large language model earns a seat as a judge — as an additive layer, never a replacement for the deterministic gates. The split is clean:
- The deterministic gates answer: is it correct, compliant, safe? These block releases.
- An LLM-as-judge answers: is the compliant output actually good? This is informational, and it never blocks until its threshold has been calibrated against human labels.
Non-Determinism Needs Bounds, Not a Single Run
Against a real model, a single evaluation run is noise. The discipline is baseline-and-bounds: run the suite several times on fixed inputs, record the mean and variance per metric, and treat a change as real only when it moves outside that band. Without it, you will chase regressions that are nothing but sampling. And cost belongs on the scorecard next to quality — an agent that is correct but burns a fortune per task is not shippable, and that trade-off is invisible unless you measure it.
Observability Closes the Loop
Every step of every agent — retrieve, guardrail, generate, validate, route — is a traced span, and the eval scores attach to those traces. When an attack is blocked or a claim is rejected, you can see exactly where. Tracing is the cheapest step in the whole program and the one that makes every other step diagnosable; instrument it first.
Security Is a Test Surface Too: SAST and DAST for Agents
Application security has long split into static analysis of the artifact (SAST) and dynamic attack of the running system (DAST). Agentic systems have direct equivalents — the "artifact" simply now includes prompts, tool wiring, and model configuration.
| Classic AppSec | Agentic equivalent | What it inspects |
|---|---|---|
| SAST | Static analysis of prompts, tool allow-lists, output contracts, secrets | Is untrusted input fenced and declared as data, not instructions? Is the action allow-list least-privilege? |
| DAST | Automated red-teaming of the live agent | Fire an injection corpus at a running, mock-backed endpoint and assert containment |
| RASP / WAF | Runtime guardrails in front of the model | Input scanning + output validation, inline |
| STRIDE threat model | OWASP Top 10 for LLM Applications, agentic threat frameworks | The catalog each check maps against |
Two categories here are genuinely new versus classic AppSec, and both are exercised by this architecture: tool-authorization testing (can a compromised agent invoke an action it shouldn't?) and chain/handoff testing (can one agent's output inject the next?). Prompt injection arrives the way it actually does in production — not typed at the prompt, but hidden inside the data the agent ingests, whether that is a clinical note or a referral message.
What a Real Model Actually Did
That is not a bug in the harness; that is the harness doing its job — catching a real model's real weakness before it reached a member, and pointing precisely at the fix.
The Principle
Give every metric a failure mode to catch.
Let a deterministic oracle judge where ground truth exists,
and a calibrated model-judge only where it doesn't.
Gate the release on the metrics that can hurt someone — and never relax them to make a model pass.
In lower-stakes domains you can ship an agent and watch the dashboards. In healthcare, the member notices before the dashboard does. The harness is not overhead on top of the agent — the harness is the product. The agent is just the part that's easy to build.
The Bottom Line
Do not ask whether the model is good. Ask whether you can prove it, on this input, right now, against something you already trust — and whether it fails safely when it cannot. Build the oracle first. Make the agent reconcile to it. Gate the release on the answers that can hurt someone.
Prove it, don't hope. Build the oracle first, make the agent reconcile to it, and gate the release on the metrics that can hurt someone.Back to Blog Index