Pharmacy Roulette: The Deceptively Hard Routing Problem Behind Every Prescription
Maria's prescription for a specialty medication just got approved. After weeks of back-and-forth with her insurance company, she finally has the green light. All that's left is to send it to a pharmacy so she can start treatment.
Picking a pharmacy. How hard could that be?
Her doctor's note says "send to CVS Specialty." The electronic health record lists two preferred pharmacies, neither of which is CVS. Her insurance plan historically routes to Accredo. She texted us last month saying she moved to a new state. And a successful transfer for the same drug and insurance went to Optum Specialty two months ago, but that pharmacy is no longer in-network as of last week.
Five signals, all pointing in different directions. And if we pick wrong, there’s no immediate failure signal. The prescription sits at a pharmacy until someone tries to process it, or Maria calls asking where her medication is. By then, days have passed and her access to treatment has been severely delayed.
This is the pharmacy selection problem. We’ve iterated over five different approaches over two years. Each one fixed something the last one couldn't handle.
Pharmacy selection looks simple: given some features about a prescription, predict the right pharmacy. Under the surface, a few key challenges emerge:
The network is a moving target. Specialty drugs can only go to certified specialty pharmacies (your local CVS can't dispense them), and which of those an insurer counts as in-network changes constantly. There's no way to look this up. You find out when routing fails.
The data is messy and incomplete. Insurance plans are identified by a trio of codes (BIN, PCN, and Group) that together determine how a claim gets routed and processed. But you might only have two of the three. A doctor note might say "CVS" with no store number, city, or distinction between CVS Retail and CVS Specialty. A patient might have moved since their last prescription. You have to make decisions with the pieces of data you have.
Signals live in unstructured places. A pharmacy preference might be buried in a patient text message, a footnote in a doctor's note, an EHR field from three years ago, or the fine print of a prior authorization (PA) approval letter.
Wrong answers fail silently and slowly. A prescription sent to a pharmacy that can't fill it doesn't return an error immediately. Instead, it enters the pharmacy's queue until a pharmacist notices or the patient calls a week later. The feedback loop is days and every miss results in a patient not receiving their medication on time.
Multiple answers can be correct. If Maria has filled at both CVS and Walgreens recently and both are in-network with her insurance, either is a valid choice. This makes evaluation hard. We can't solely measure accuracy because a different answer isn't necessarily wrong. So, we measure success by re-route rate (how often a prescription had to be sent to a different pharmacy after the initial selection).
Our first approach was a rules engine. For retail drugs, we built a 14-case if/else cascade that checked signals in a fixed order:
Case 1: No EHR preferred pharmacy + no fill history → manual review
Case 2: Single EHR preferred pharmacy + no history → select it
Case 3: Single EHR preferred pharmacy + all Rx history agrees → select it
Case 4: No EHR preferred pharmacy + all Rx history agrees → select it
Case 5: Single pharmacy in last year → select it
...
Case 14: Multiple preferred + history tiebreaker → best matchThe first match wins. For specialty drugs, we maintained a hardcoded dictionary that mapped roughly 30 insurance BIN+PCN combinations directly to pharmacies. When a patient’s insurance matched the dictionary, routing succeeded. Otherwise, the case was escalated to an expert for manual review.
This worked for the easy cases. When a patient has a single preferred pharmacy and all their prior prescription history confirms it, the answer is obvious. But it fails at scale. Every new insurance required updating the hardcoded mapping. The specialty dictionary went stale within weeks because pharmacy networks change without notice. We kept adding branches for months, and our clinical experts were still handling the majority of cases because the rules were too brittle to cover the real complexity.
To replace the deterministic approach, we trained an XGBoost model. Our data was tabular with lots of missing values, which is exactly where gradient-boosted trees shine. The model took 9 input features:
| Feature | Source |
|---|---|
| Drug name | Prescription |
| Patient's state | Prescription |
| Year prescribed | Prescription |
| Insurance BIN | Insurance card |
| Insurance PCN | Insurance card |
| Insurance GROUP | Insurance card |
| Insurance plan name | Insurance card |
| PA submission target | Prior authorization |
| PA form name | Prior authorization |
It predicted a specialty pharmacy with per-pharmacy confidence thresholds: only automate when the model is confident enough to be right >97% of the time, and escalate everything else to manual review.
The model generalized well initially. It found patterns in feature combinations (specific BIN-drug -state triples) that were hard to write out as explicit rules.
Then it started drifting.
The temporal drift problem
Contracts between payers and pharmacies are constantly renegotiated, and the in-network pharmacy for a given BIN/PCN/Group can change overnight. The model had no way to know. It would predict "Accredo" for a BIN/PCN combination with 95% confidence, and be completely wrong because the Accredo contract expired two weeks ago. Retraining helped, but it didn’t close the gap. Between the moment a network changed and when an updated model deployed, we were still sending prescriptions to the wrong pharmacy.
We needed a model that stayed current without retraining.
Instead of asking "what do these insurance features predict?", we started asking a different question: which pharmacy did we actually transfer to last time for a prescription that looked like this one? And how recently?
This flips the whole approach. Instead of a frozen feature snapshot, the model uses a live history of transfer outcomes, weighted by recency. When a pharmacy drops out of a network, we see evidence of it happening because successful transfers stop. The model is able to pivot from this evidence with no retraining necessary.
Getting there took months of watching the V2 model drift and asking why we were retraining a model to learn things our own transfer history already knew.
We shipped this as a four-stage pipeline.
Stage 1: Rule matching via powerset
Say a prescription comes in for Humira, with insurance BIN 610014, PCN OHCARD, in Texas. In practice, we have up to 8 features (insurance identifiers, drug, state, prescriber, plan name, PA form) but many are null. We ask: have we ever successfully transferred a prescription with this exact combination before? If so, where did it go? Then we relax the search: what about just this BIN + PCN, any drug, any state? What about just this BIN + Humira, any state?
We do this systematically. For every possible combination of the available features (at least two at a time), we check if our database has a matching historical rule.
Known features: {bin, pcn, drug, state}
Search from most specific to least:
(bin, pcn, drug, state) → 3 matching transfers, all to CVS Specialty
(bin, pcn, drug) → 12 matching transfers, 10 to CVS, 2 to Accredo
(bin, pcn, state) → 8 matching transfers, 6 to CVS, 2 to Optum
(bin, drug, state) → 0 matches
(pcn, drug, state) → 0 matches
(bin, pcn) → 140 matching transfers, mixed across 4 pharmacies
(bin, drug) → 0 matches
...Each match is a combination of feature values we've actually seen before in real transfers. The most specific match (all 4 features) has only 3 data points but high precision. The broadest match (just BIN + PCN) has 140 data points but lower precision. Both contribute signal.
Rules, not features
This is a fundamentally different approach from V2. A traditional classifier learns feature importance: BIN matters a lot, patient state matters less. The powerset approach searches for specific feature-value combinations (rules) in historical data. It doesn't just know that BIN matters. It knows that BIN 610014 + PCN OHCARD historically routes to Accredo, while BIN 610014 + PCN OHCARD + Humira routes to CVS Specialty.
Here's what's interesting: when the broad and specific rules disagree, that disagreement is useful. In our example, the broad rule (just BIN + PCN) says this insurance plan usually goes to Accredo. But the specific rule (BIN + PCN + Humira + Texas) says the last 3 transfers all went to CVS Specialty. Maybe Humira has always gone to a different pharmacy than the plan's default. Or maybe Accredo recently lost the contract and the broad rule just hasn't caught up yet. Either way, the model is able to factor in this conflict.
The powerset also handles incomplete data naturally. If a prescription only has 3 non-empty features, we search 4 subsets. If it has all 8, we search 247. The point is that we find whatever historical evidence exists.
Stage 2: Evidence retrieval and temporal decay
For each matching rule, we pull all historical transfer records with timestamps. In our example, the broad rule (BIN + PCN) has 140 records spanning 18 months. Six months ago, most went to Accredo. The last month is almost entirely CVS Specialty. The raw counts say Accredo; the recent pattern says CVS Specialty. We need a way to weigh recent evidence more heavily.
We do this with a sigmoid decay function:
w(Δd) = 1 / (1 + exp(√Δd − 5))Where Δd is the number of days between the current prescription date and the historical transfer date. The curve holds steady at first, then drops sharply, with the inflection point at 25 days (√25 = 5).
| Days since transfer | Weight |
|---|---|
| 1 | 0.98 |
| 7 | 0.91 |
| 14 | 0.78 |
| 25 | 0.50 |
| 45 | 0.15 |
| 60 | 0.06 |
| 90 | 0.01 |
The curve is aggressive by design: evidence older than three months is effectively ignored. From these weighted results, we extract four signals:
Time-weighted most common pharmacy: Sum each pharmacy’s weights; highest total wins. If Accredo dominated six months ago but CVS has taken the last month, CVS comes out ahead, since recent transfers weigh ~0.9 and anything past 90 days rounds to zero.
Time-weighted percent match: How much of the weighted evidence agrees on the winner? A rule where 95% of the weight points to one pharmacy is much stronger signal than another at 55%.
Cross-tier agreement: Do rules at different specificity levels agree? If 3 out of 4 matching rules point to the same pharmacy, that's a strong signal. If they're split, something is probably changing.
Evidence freshness: The weighted average age of the evidence in days, computed using the same sigmoid weights. A low number means the prediction is based on recent transfers; a high number means the model is relying on older, less reliable data.
Stage 3: XGBoost prediction
An XGBoost model trained on these signals makes the final call. Its inputs are the four features plus some metadata about the matches (how many rules matched and how much evidence each had). It outputs a pharmacy prediction and a confidence score.
V3 solved the freshness problem. But we kept hitting cases where the model fell short because the answer was buried in a faxed PA letter or a patient text message that ourno feature vector didn’t capture.
A doctor's note saying "send to CVS Specialty in Fort Worth." A PA letter that says "this medication must be dispensed by Accredo." A patient texting "my new pharmacy is Walgreens on Main St." These aren't edge cases. They're often the most authoritative signals available, and we needed a way to feed them into our model.
We used LLM extractors running before the ML models. An LLM reads PA approval letters and pulls out the mandated pharmacy. Another parses free-text doctor notes ("send to CVS/pharmacy 4537 in Dallas," "pt prefers Accredo," "use the specialty pharmacy from last time") into structured pharmacy names. Another extracts preferences from patient text messages. The extracted name is fuzzy matched against our pharmacy database to select a specific pharmacy.
The limits of isolated extraction
At this point, our cascade had 11 stages. When signals conflicted (and they often did) the waterfall resolved by priority ordering. Doctor note beats insurance rule. Insurance rule beats transfer history. But a doctor note from a year ago saying "send to Accredo" shouldn't override a fresh insurance rule pointing to CVS Specialty. The cascade has no concept of why a signal might be stale or which one should win in a given case. And the edge cases kept growing.
Instead of pushing signals through a fixed cascade, we pass an LLM agent all the context at once and let it reason.
The agent gets the raw signals, not the processed outputs. It doesn't call the XGBoost model or read the LLM extractors' structured outputs. It reads the doctor note itself and resolves pharmacy names through its own tool calls.
Those tools let it search pharmacies, read documents (including insurance cards via vision), check dispensing restrictions, look up insurance details, request patient preferences, schedule phone calls, store a pharmacy selection, or escalate to an expert. It reads everything, decides which pieces of information to rely on, investigates further if it needs to, and then acts. Unlike the cascade, it can hold every conflicting signal at once and decide which should win in this particular case.
The first version of the agent performed worse than the waterfall on many cases. But when it got things right, its reasoning reflected the kind of cross-signal thinking that our clinical experts do.
Take Maria. The cascade takes the doctor note ("send to CVS Specialty") at face value because doctor notes rank high. The agent sees the full picture: the note is from a year ago, Optum's transfer evidence has gone cold (suggesting a network change), the patient texted that she moved to a new state, and the EHR-preferred pharmacies don't include CVS. So it discounts the stale note, recognizes the state change means prior prescription history may not apply, and either selects the in-network EHR pharmacy or requests an updated preference from Maria. Five conflicting signals,resolved by r context instead of rank.
Shipping safely: shadow mode
We couldn't A/B test with live prescriptions flowing through our system. So, we built shadow mode: the agent runs on real data, genuinely reading documents and searching pharmacies, but its action tools are intercepted and return plausible simulated responses. The agent can’t tell the difference, and no side effects are executed. Shadow mode lets us compare the agent's decisions against the existing rules engine n every prescription for weeks before enabling any real actions. During rollout, we whitelisted specific actions like storing a pharmacy selection and escalating to an expert. As we validated the shadow data, we unlocked more actions.
When we find something that works, we want to exploit it. Successfully transferring 50 prescriptions with BIN 610014 + PCN OHCARD to Accredo in the last 3 months is a strong signal and we should keep routing there. But if Accredo's contract with that payer ends, the first few transfers will fail. Each failed transfer is a patient without medication. How do we get signal early that something has changed?
The sigmoid decay gives us passive detection: the system naturally loses confidence in a pharmacy that stops receiving successful transfers. But this only works after transfers start failing. There's an inherent lag between the network changing and the model catching up, and the decay rate is a tradeoff: too aggressive and you incorrectly decrease confidence in stable routings, too slow and you keep routing to a pharmacy that can't fill.
This is where the agent changes the game. The statistical model can only react passively. You can't explore by trying an uncertain pharmacy on a real patient's prescription. But the agent can explore by gathering information instead: it notices early warning signs (a denial from a pharmacy that used to be valid, a provider note pointing somewhere new) and investigates before committing. The agent may request a preference from the patient, suggest a phone call to verify network status, or escalate to a clinical expert. It converts the explore problem from "try something risky" to "gather more information before deciding."
This tension shows up everywhere we automate, not just pharmacy selection.
Every agent run is traced end-to-end: every tool call, every response, every decision point is recorded, tied to a specific agent version, and scored against its final outcome (did the pharmacy selection succeed, or did a human re-route it?). We have the data to know exactly where the agent fails and why.
What’s missing is the ability to automatically test whether a proposed change actually fixes those failures.
The simulation problem
We once shipped a prompt change that performed well on the cases we reviewed by hand, then broke on a subtle pattern we hadn't checked. Manual testing wasn't going to scale.
The obvious fix is replay. We have thousands of historic traced runs, so we can run the new version on the same inputs and compare. That works as long as the new version follows the same path as the old one. It breaks upon any divergence: a different search_pharmacy query, a pharmacy the old run never considered, a preference request instead of a direct selection. Now, the replay can’t tell us about what happens next. The trace recorded what searching for Pharmacy A returned, not what would happen if the query was for Pharmacy B instead, because the original run didn’t search for B.
We’re building a simulator that can handles these divergences. Tool calls that match the trace return the recorded response. Divergent calls get a simulated one, based on patterns from thousands of other runs. Every production run adds more data on how tools behave, so the simulator gets more realistic over time.
From simulator to continuous improvement
With a working simulator, the feedback loop closes. A proposed change is backtested against thousands of real historical cases. We score it on two things: did it reason about the signals correctly, and was the pharmacy selection correct? If the new version demonstrates improvement, we ship it. Every failure in production becomes a regression test case.
Today, our team of clinical experts audits every new agent version before it rolls out. The simulator changes that from a manual gate to a continuous loop. We're not there yet, but the pieces are coming together. The goal is agents that propose, test, and ship their own improvements.
From our experience iterating on the solution to the pharmacy selection challenge a few patterns stand out:
Start deterministic, earn your way to ML. The rules engine reliably caught the easy cases Introducing ML was only worth the complexity when the scale and long tail of unhandled cases was large enough to justify it.
Production ML is constantly at risk of temporal drift. Accuracy can degrade significantly within months if the distribution shifts and the model doesn't. Build systems that learn from outcomes continuously rather than from snapshots periodically.
Design for incomplete data from the start. In healthcare, you rarely have clean data. The powerset approach (finding the best evidence that matches whatever features you do have, rather than requiring a complete set) substantially outperformed approaches that treated missing features as unknowns.
Multiple correct answers demand different metrics. When two pharmacies are both acceptable, accuracy against a single label is misleading. Measure what matters: did the selection work, or did the prescription have to be re-routed?
Trust is built incrementally. Shadow mode, per-tool autonomy, gradual rollout. In a system where wrong answers delay patient care, there's no shortcut to building production confidence.
We're running agents like these in production right now, resolving cases like Maria's every day. If these problems are interesting to you: temporal ML, explore/exploit under distribution shift, self-improving agents, building reliable AI for high-stakes decisions with incomplete data, we're hiring.
