We built agents to solve the last mile problem: pharmacy selection
Maria's prescription for a biologic, a specialty medication that costs tens of thousands of dollars per year that she depends on to manage a chronic condition, just got approved. After weeks of back-and-forth with her insurance company, she finally got the green light. All that's left is to send it to the right pharmacy so she can start treatment.
Picking a pharmacy. How hard could that be?
Her doctor's note says "send to PineCare Pharmacy." The electronic health record lists two preferred pharmacies, neither of which is PineCare. Her insurance plan historically routes to RxGate. She texted us last month saying she moved to a new state. Another prescription for the same drug and insurance was successfully routed to MedRoute Specialty two months ago, but that pharmacy is no longer in-network as of last week.
Five signals, all pointing in different directions. And if we pick wrong, there’s no immediate failure signal. The prescription sits at a pharmacy until someone tries to process it, or Maria calls asking where her medication is. By then, days have passed and her access to treatment has been severely delayed.
This is the pharmacy selection problem. We’ve iterated through five different approaches over two years.
Pharmacy selection looks simple: given some features about a prescription, predict the right pharmacy. Under the surface, a few key challenges emerge:
Rules can change without notice. Specialty drugs can only go to certified specialty pharmacies (i.e. your local Walgreens can't dispense them). For any given insurer, which specialty pharmacies are considered "in-network" changes constantly, with no way to look it up. You find out when a routing fails.
The data is messy and incomplete. Insurance plans are identified by a trio of codes (BIN, PCN, and Group) that together determine how a claim gets routed and processed. But you might only have two of the three. A doctor note might say "CVS" with no store number, city, or distinction between the retail and specialty branches. A patient might have moved since their last prescription. You have to make decisions with the pieces of data you have.
Signals live in unstructured places. A pharmacy preference might be buried in a patient text message, a footnote in a doctor's documentation, an EHR field from three years ago, or the fine print of a prior authorization (PA) approval letter.
Wrong answers fail silently and slowly. A prescription sent to a pharmacy that can't fill it doesn't return an error immediately. Instead, it enters the pharmacy's queue until a pharmacist notices or the patient calls a week later. You don’t hear back for days and every miss results in a patient not receiving their medication on time.
Multiple answers can be correct. If Maria has used both PineCare Pharmacy and RxGate recently and both are in-network with her insurance, either is a valid choice. This makes evaluation hard. We can't solely measure accuracy because a different answer isn't necessarily wrong. So, we measure success by re-route rate (how often a prescription had to be sent to a different pharmacy after the initial selection).
Two years ago, our first approach was a rules engine. For retail drugs, we built a 14-case if/else cascade that checked signals in a fixed order:
Case 1: No EHR preferred + no fill history → manual review
Case 2: Single EHR preferred + no history → select it
Case 3: Single EHR preferred + all history agrees → select it
Case 4: No EHR preferred + all history agrees → select it
Case 5: Single pharmacy in last year → select it
...
Case 14: Multiple preferred + history tiebreaker → best match
The first match wins. For specialty drugs, we hardcoded ~30 insurance rules. When a patient’s insurance matched our hardcoded list, routing succeeded. Otherwise, the case was escalated to an expert for manual review.
But this fails at scale. Every new insurance required updating the hardcoded mapping. Our hardcoded rules went stale within weeks because pharmacy networks changed without notice. Our clinical experts were still handling the majority of cases because the rules were too brittle to cover the real complexity.
To replace the deterministic approach, we trained an XGBoost model. Our data was tabular with lots of missing values, which is exactly where gradient-boosted trees shine. The model took 9 input features:
| Feature | Source |
|---|---|
| Drug name | Prescription |
| Patient's state | Prescription |
| Year prescribed | Prescription |
| Insurance BIN | Insurance card |
| Insurance PCN | Insurance card |
| Insurance GROUP | Insurance card |
| Insurance plan name | Insurance card |
| PA submission target | Prior authorization |
| PA form name | Prior authorization |
It predicted a specialty pharmacy with per-pharmacy confidence thresholds: only automate when the model is confident enough to be right >97% of the time, and escalate everything else to manual review.
The model generalized well initially. It found patterns in feature combinations (specific BIN-drug -state triples) that were hard to write out as explicit rules.
Then it started drifting.
The temporal drift problem
Contracts between payers and pharmacies are constantly renegotiated, and the in-network pharmacy for a given BIN/PCN/Group can change overnight. The model had no way to know. It would predict "RxGate" for a BIN/PCN combination with 95% confidence, and be completely wrong because the RxGate contract expired two weeks ago. Retraining helped, but it didn’t close the gap. Between the moment a network changed and when an updated model deployed, we were still sending prescriptions to the wrong pharmacy.
We needed a model that stayed current without retraining.
Rather than asking "what do these insurance features predict?", we started asking a different question: which pharmacy did we actually route to last time for a prescription that looked like this one? And how recently?
Instead of a frozen feature snapshot, the model uses a live history of routing outcomes, weighted by recency. When a pharmacy leaves a network, we don’t see any more successful routings. The model adjusts on its own, no retraining necessary.
We shipped this as a three-stage pipeline.
Stage 1: Rule matching via powerset
Say a prescription comes in for Humira, with insurance BIN 610014, PCN OHCARD, in Texas. In practice, we have up to 8 features (insurance identifiers, drug, state, prescriber, plan name, PA form) but many are null. We ask: have we ever successfully routed a prescription with this exact combination before? If so, where did it go? If not, we relax the search: what about just this BIN + PCN, any drug, any state? What about just this BIN + Humira, any state?
We do this systematically. For every possible combination of the available features (at least two at a time), we check if our database has a matching historical rule.
Known features: {bin, pcn, drug, state}
Search from most specific to least:
(bin, pcn, drug, state) → 3 historical routings, all to PineCare Pharmacy
(bin, pcn, drug) → 12 historical routings, 10 to PineCare Pharmacy, 2 to RxGate
(bin, pcn, state) → 8 historical routings, 6 to PineCare Pharmacy, 2 to MedRoute
(bin, drug, state) → 0 matches
(pcn, drug, state) → 0 matches
(bin, pcn) → 140 historical routings, mixed across 4 pharmacies
(bin, drug) → 0 matches
...Each match is a combination of feature values we've actually seen before in real routings. The most specific match (all 4 features) has only 3 data points but high precision. The broadest match (just BIN + PCN) has 140 data points but lower precision. Both are useful signals.
Rules, not features
This approach is different from V2. V2 learned feature importance: BIN matters a lot, patient state matters less. This powerset approach searches for specific feature-value combinations (rules) in historical data. It doesn't just know that BIN matters. It knows that BIN 610014 + PCN OHCARD historically routes to RxGate, while BIN 610014 + PCN OHCARD + Humira routes to PineCare Pharmacy.
Here's what's interesting: when the broad and specific rules disagree, that disagreement is useful. In our example, the broad rule (just BIN + PCN) says this insurance plan usually goes to RxGate. But the specific rule (BIN + PCN + Humira + Texas) says the last 3 routings all went to PineCare Pharmacy. Maybe Humira has always gone to a different pharmacy than the plan's default. Or maybe RxGate recently lost the contract and the broad rule just hasn't caught up yet. Either way, the model is able to factor in this conflict.
The powerset also handles incomplete data naturally. If a prescription only has 3 non-empty features, we search 4 subsets. If it has all 8, we search 247. The point is that we find whatever historical evidence exists.
Stage 2: Evidence retrieval and temporal decay
For each matching rule, we pull all historical routing records with timestamps. In our example, the broad rule (BIN + PCN) has 140 records spanning 18 months. Six months ago, most went to RxGate. The last month is almost entirely PineCare Pharmacy. The raw counts say RxGate; the recent pattern says PineCare Pharmacy. We need a way to weigh recent evidence more heavily.
We do this with a sigmoid decay function:
w(Δd) = 1 / (1 + exp(√Δd − 5))Here, Δd is the number of days between the current prescription date and the historical routing date. The curve holds steady at first, then drops sharply, with the inflection point at 25 days (√25 = 5).
The curve is aggressive by design: evidence older than three months is effectively ignored. From these weighted results, we extract four signals:
Time-weighted most common pharmacy: For each pharmacy, sum the time weights of all its routings. The pharmacy with the highest total wins. This naturally handles network changes: if RxGate was dominant 6 months ago but has left the network and now PineCare Pharmacy has dominated the last month, PineCare Pharmacy's cumulative weight will exceed RxGate's because recent weights are ~0.9 while anything older than 90 days is essentially zero.
Time-weighted percent match: How much of the weighted evidence agrees on the winner? A rule where 95% of the weight points to one pharmacy is a much stronger signal than another at 55%.
Cross-tier agreement: Do rules at different specificity levels agree? If 3 out of 4 matching rules point to the same pharmacy, that's a strong signal. If they're split, something is probably changing.
Evidence freshness: The weighted average age of the evidence in days, computed using the same sigmoid weights. A low number means the prediction is based on recent routings; a high number means the model is relying on older, less reliable data.
Stage 3: XGBoost prediction
An XGBoost model trained on these signals makes the final call. Its inputs are the four features plus some metadata about the matches (how many rules matched and how much evidence each had). It outputs a pharmacy prediction and a confidence score.
V3 solved the freshness problem. But we kept hitting cases where the model fell short because the answer was buried in a faxed PA letter or a patient text message that our feature vector didn’t capture.
A doctor's note saying "send to PineCare Pharmacy in Fort Worth." A PA letter that says "this medication must be dispensed by RxGate." A patient texting "my new pharmacy is MedRoute on Main St." These are often the most authoritative signals available, and we needed a way to feed them into our model.
We used LLM extractors running before the ML models. An LLM reads PA approval letters and pulls out the mandated pharmacy. Another parses free-text doctor notes ("send to PineCare Pharmacy #4537 in Dallas", "pt prefers RxGate", "use the specialty pharmacy from last time") into structured pharmacy names. Another extracts preferences from patient text messages. The extracted name is fuzzy matched against our pharmacy database to select a specific pharmacy.
The limits of isolated extraction
At this point, our cascade had 11 steps. When signals conflicted (and they often did) the waterfall resolved by the priority ordering we set. Doctor note beats insurance rule. Insurance rule beats routing history. But a doctor note from a year ago saying "send to RxGate" shouldn't override a fresh insurance rule pointing to PineCare Pharmacy. The cascade has no concept of why a signal might be stale or which one should win in a given case. And the edge cases kept growing.
When we started this work, LLMs weren't capable enough to handle this kind of reasoning reliably. By the time we'd built V3 and V4, the models had caught up. We could now give an agent the full context and tools that a human would have, and trust it to reason across conflicting signals.
Those tools let it search pharmacies, read documents (including insurance cards via vision), check dispensing restrictions, look up insurance details, request patient preferences, schedule phone calls, store a pharmacy selection, or escalate to an expert. It reads everything, decides which pieces of information to rely on, investigates further if it needs to, and then acts. Unlike the cascade, it can hold every conflicting signal at once and decide which should win in this particular case.
The first version of the agent performed worse than the waterfall in many cases. But when it got things right, its reasoning reflected the kind of cross-signal thinking that our clinical experts do.
Take Maria. The cascade takes the doctor note ("send to PineCare Pharmacy") at face value because doctor notes rank high. The agent sees the full picture: the note is from a year ago, MedRoute's routing evidence has gone cold (suggesting a network change), Maria texted that she moved to a new state, and the EHR-preferred pharmacies don't match the doctor's note. So it discounts the stale note, recognizes the state change means prior prescription history may not apply, and either selects the in-network EHR pharmacy or requests an updated preference from Maria.
Shipping safely: shadow mode
We couldn't A/B test with live prescriptions flowing through our system. So, we built shadow mode: the agent runs on real data, reading documents and searching pharmacies, but its write tools are mocked to return plausible simulated responses. The agent can’t tell the difference, and no side effects are executed.
Shadow mode lets us compare the agent's decisions against the existing rules engine on every prescription for weeks before enabling any real actions. During rollout, we whitelisted specific actions like storing a pharmacy selection and escalating to an expert. As we validated the shadow data, we unlocked more actions.
When we find something that works, we want to exploit it. Successfully routing 50 prescriptions with BIN 610014 + PCN OHCARD to RxGate in the last 3 months is a strong signal and we should keep routing there. But if RxGate's contract with that payer ends, the first few routings will fail. Each failure is a patient without medication. How do we get signal early that something has changed?
The sigmoid decay gives us passive detection: the system naturally loses confidence in a pharmacy that stops receiving successful routings. But this only works after routings start failing. There's an inherent lag between the network changing and the model catching up, and the decay rate is a tradeoff: too aggressive and you incorrectly decrease confidence in stable routings, too slow and you keep routing to a pharmacy that can't fill.
This is where the agent changes the game. The statistical model can only react passively. You can't explore by trying an uncertain pharmacy on a real patient's prescription. But the agent can explore by gathering information instead: it notices early warning signs (a denial from a pharmacy that used to be valid, a provider note pointing somewhere new) and investigates before committing. The agent may request a preference from the patient, suggest a phone call to verify network status, or escalate to a clinical expert. It converts the explore problem from "try something risky" to "gather more information before deciding."
This tension shows up everywhere we automate, not just pharmacy selection.
From our experience iterating on the solution to the pharmacy selection challenge, a few patterns stand out:
Start deterministic, earn your way to ML. The rules engine reliably caught the easy cases. Introducing ML was only worth the complexity when the scale and long tail of unhandled cases was large enough to justify it.
Production ML is constantly at risk of temporal drift. Accuracy can degrade significantly within months if the distribution shifts and the model doesn't. Build systems that learn from outcomes continuously rather than from snapshots periodically.
Design for incomplete data from the start. In healthcare, you rarely have clean data. The powerset approach (finding the best evidence that matches whatever features you do have, rather than requiring a complete set) substantially outperformed approaches that treated missing features as unknowns.
Multiple correct answers demand different metrics. When two pharmacies are both acceptable, accuracy against a single label is misleading. Measure what matters, i.e. did the selection work, or did the prescription have to be re-routed?
Trust is built incrementally. Shadow mode, per-tool autonomy, gradual rollout. In a system where wrong answers delay patient care, we cannot take shortcuts in deployment.
We're running agents like these in production right now, resolving cases like Maria's every day. If problems like temporal ML, explore/exploit under distribution shift, self-improving agents, and building reliable AI for high-stakes decisions with incomplete data interest you, we're hiring.
