How we evaluate our LLM judge:
a perturbation-based approach

LLM-as-judge systems have a well-known failure mode: they tend to be overzealous, unnecessarily flagging correct answers. In a clinical context this is a serious problem, as every extraneous flag leads to additional steps that could delay a patient receiving a critical medication.

Insurance plans require prior authorizations (PAs) to decide whether to cover specialty medications for conditions like autoimmune diseases, cancer, and multiple sclerosis, based on detailed questions providers answer about patients’ clinical history. At Forus, we developed an LLM judge to verify generated PA answers against patient clinical records and specialty-specific guidance, ensuring claims are grounded in medical evidence. Because our platform manages PA submission end-to-end, the calibration of this judge is critical—a hand-wavy "it looks good on a few examples" approach would be an insufficient evaluation strategy.

This over-flagging tendency is consistent with industry findings: LLM judges tend to lean conservative when verifying claims against evidence, especially in incomplete or ambiguous cases [1]. Our system faced a calibration problem. A high false positive rate means the judge can't be trusted to act autonomously, since every flagged case requires human intervention. Calibrating against that bias became our focus.

However, standard eval options were all unsatisfying: human auditing of live outputs is slow and expensive, using another LLM to judge the judge is circular, and academic benchmarks are insufficient for the level of intricacy we see. Popular medical QA benchmarks like MedQA [2], PubMedQA [3], and the MultiMedQA [4] suite evaluate medical knowledge through USMLE-style multiple-choice questions and biomedical research abstracts. These are useful signals for general clinical reasoning, but don’t exactly align with answering PA form questions against a specific patient's medical history. Emerging work like MedAgentBench [5] is more targeted, and evaluates LLM agents in simulated EHR environments. Still, no standardized benchmark exists for PA question answering.

We needed something we could run during development (fast, repeatable, grounded in real clinical data), not something we'd discover during a production incident. Fortunately, we'd already done a lot of the hard work, building a rigorous gold standard dataset for our PA evals through a multi-annotator consensus workflow followed by expert clinical review, sampled to be representative of our real data distribution across drugs, payers, and question types. That investment gave us high-quality ground truth, but we still needed to figure out how to use it to evaluate the judge.

Our verification pipeline has four stages, adapted for the PA domain from the decompose-retrieve-judge paradigm used in factual verification work like VeriFact [6] and FActScore [7].

Atomic decomposition

PA form questions often ask the provider to confirm multiple clinical conditions in a single field: a diagnosis, the therapies tried, the patient's response to each, and the timing of those trials. We break each answer into atomic claims, the smallest standalone statements that can be independently checked.

Query expansion and retrieval

Clinical language is full of synonyms and phrasing variations: a claim about "atopic dermatitis" needs to match a chart note saying "eczema," and "inadequate response" might be documented as "failed therapy" or "discontinued due to lack of efficacy." We expand each claim into multiple semantic search queries before retrieving relevant evidence from the patient's record.

Claim-level judgment

For each claim, the judge produces one of three verdicts.

• Supported: the evidence affirms the claim.
• Not Supported: the claim conflicts with the evidence, either through direct contradiction or through a significant omission or fabrication.
• Not Addressed: the evidence is silent on the question–this category is critical because in healthcare, records are routinely incomplete, and a missing lab result doesn't mean the lab was never drawn. Collapsing silence into "not supported" would make the judge unfairly punitive.

Question-level aggregation

Claim verdicts roll up to an answer-level result using worst-verdict aggregation, so if any claim is unsupported, the whole answer is flagged.

Our gold standard tells us whether the judge correctly approves accurate answers. If we run the judge over a set of correct PA answers and count how often it flags them anyway, we get the false positive rate. A high rate makes the judge unhelpful. But the more dangerous failure is the opposite—a judge that misses a real error, quietly validating a claim the evidence doesn't support, can affect a patient's access to medication and undermine trust in the process.

However, gold standards are built to show what right looks like. We needed the opposite: examples of what wrong looks like, but on the same patients, with the same evidence. Our idea was to take a subset of correct answers and synthetically perturb them into plausible-but-wrong versions, then evaluate whether the judge flags the errors. This is essentially mutation testing but for LLM evals: we introduce known bugs and check if our tests catch them.

This gives us two evaluation runs per gold standard PA, with a design resembling a randomized controlled trial; we have all of the same clinical data, but only the PA form answers change. Any difference in judge behavior between the runs is attributable to the answer itself, not to noise in the underlying record. The clean run measures the false positive rate; the perturbed run measures the detection rate. We review both, but the clean-run false positive rate is the one we anchor on, because detection rate is only meaningful once we know the judge isn't excessively flagging.

A concrete example

Consider a PA for a biologic drug where one question asks for the patient's diagnosis and another asks for the supporting clinical features. The original answers list "Crohn's disease" as the diagnosis, alongside features like strictures, fistulas, and transmural inflammation, which are characteristic of Crohn's. We perturb the diagnosis to "ulcerative colitis" (UC) but leave the supporting features untouched. The judge flags two things. First, the perturbed diagnosis: the chart doesn't support UC, so this is the error we introduced and expected to catch. But the judge also flags the unchanged supporting-features answer, because strictures and fistulas point to Crohn's, not UC, so the features no longer fit the diagnosis they're meant to support. One perturbation, two flags, and the second one is, interestingly, a behavior we hadn't explicitly designed the judge to produce.

Synthetic perturbations are cheap to produce, so we can run a proper train/test split: tune prompts on one set, validate on a held-out set. Hand-labeled eval data usually forces a choice between a tiny test set or no test set at all; synthetic perturbation sidesteps the tradeoff.

Our initial instinct was to start simple by flipping yes-no answers, swapping multiple-choice options, and using regular expressions to alter free-text answers. But naive perturbations only test robustness to shallow changes, not clinical judgment. The harder problem became clear: how do you generate plausible wrong answers: ones that appear sound but are factually inaccurate for a specific patient? This requires the model to invent errors that respect the surrounding clinical context, including the patient's diagnoses, medication history, and therapies actually tried. An error incoherent with the rest of the chart is too easy to catch and is an insufficient measure of how the judge handles realistic mistakes.

Our approach was to build a perturbation model, seeded with structured context from the patient's record, so its wrong answers stay anchored to the patient's actual medical history. For free-text questions, the model generates a plausible wrong answer. For multiple-choice questions, it selects a different option that's clinically coherent in context but factually unsupported for the patient.

Three failure modes surfaced during this work, each one teaching us something about what makes a perturbation actually useful:

Together these constraints shaped the perturbation design: definitive assertions only, specific and falsifiable claims, no attestations, and around 30% of questions perturbed per form. That last number is a calibration: enough perturbations to measure detection cleanly, few enough that the form still resembles a realistic submission with isolated errors rather than a uniformly suspicious document.

For systems with real consequences, evaluating an LLM judge takes more than spot-checks and confusion matrices. The alternative we built taught us two things worth highlighting.

The first is that a single gold standard can do double duty. By generating known-wrong answers anchored to real patient records, we got both directions of measurement out of an asset we'd already built, without paying twice for annotation.

The second is that a well-designed eval doesn't just measure a model, it surfaces what the model is doing that you didn't ask it to do. The Crohn's-to-UC perturbation earlier is one example of a broader pattern, where one perturbation causes the judge to flag a different, unperturbed answer because the two no longer fit together. In production, real errors often cascade through related questions, and we want the judge to catch that. In eval, we have to separate direct detection (the judge flags the answer we perturbed) from consistency-based detection (the judge flags a different, unperturbed answer because it conflicts with the perturbed one), since conflating them would overstate how well the judge catches the specific errors we introduced.

The harder problem, the one we're still working on, is the gap between synthetic errors and real ones. Production mistakes are subtler, more contextual, and more idiosyncratic than what any perturbation model generates today.

We're exploring a few directions to close that gap. One is mining production data for confirmed errors and using them to seed more realistic perturbations. Another is building an error taxonomy from real cases so that perturbations cover the actual distribution of mistakes rather than the ones a model finds easy to invent. A third is adversarial generation, where the perturbation model is trained to produce errors the judge misses, so that the eval gets harder as the judge gets better.

There's also a deeper question we don't have a clean answer to yet: how do you know when an eval is good enough to trust? Detection rate on synthetic perturbations and agreement with human reviewers on production cases are both proxies. Neither represents what we actually care about, which is whether the judge is making the right call on the next patient's PA. We're increasingly convinced this isn't a problem any single eval solves, and that the right answer involves layering signals, with development-time evals like this one, production monitoring, and ongoing clinical review each catching what the others miss.

If you're thinking about the same problems, we'd love to compare notes.