We built agents to solve the last mile problem: pharmacy selection

Maria's prescription for a biologic drug she depends on to manage a chronic condition just got approved. After weeks of back-and-forth with her insurance company, all that's left is to send it to the right pharmacy so she can start treatment.

Picking a pharmacy. How hard could that be?

Her doctor's note says "send to RxGate." The electronic health record (EHR) lists two preferred pharmacies, neither of which is RxGate. Her insurance plan historically routes to PineCare Pharmacy. She texted us last month saying she moved to a new state. Another prescription for the same drug and insurance was successfully routed to MedRoute Specialty two months ago, but that pharmacy is no longer in-network as of last week.

Five signals, all pointing in different directions. And if we pick wrong, there’s no immediate failure signal. The prescription sits at a pharmacy until staff tries to process it, or Maria calls asking where her medication is. By then, days have passed and her access to treatment has been severely delayed.

This is the pharmacy selection problem. Over two years, we evolved a rules-based cascade into an agent that can weigh conflicting evidence and investigate before acting.

Rules change without notice. Not every pharmacy can dispense a specialty drug, and which eligible pharmacies are in-network can change without warning. Often, we find out when a routing fails.

Data is incomplete and scattered. Insurance routing depends on BIN, PCN, and Group. We may only have two of those three codes. A pharmacy preference might be buried in a patient text, a doctor's note, or a prior authorization (PA) approval letter.

Multiple answers can be correct. Two pharmacies may both be valid choices, so accuracy against a single label can mislead. We measure re-route rate, or how often a prescription has to be sent to another pharmacy after the initial selection.

Our first approach was a rules engine with a 14-case if/else cascade for retail drugs and ~30 hardcoded insurance rules for specialty drugs. The first matching rule took precedence. Unresolved cases went to manual review.

But the rules went stale within weeks as pharmacy networks changed. Every new insurance plan required another mapping, and our clinical experts were still handling the majority of cases.

To improve the cascade, we added an XGBoost model using 9 features from prescriptions, insurance cards, and prior authorizations. Our data was tabular with lots of missing values, which is exactly where gradient-boosted trees shine.

It predicted a specialty pharmacy. We set per-pharmacy confidence thresholds to target >97% accuracy on automated selections. The rest went to manual review.

The model generalized well initially. It found patterns in feature combinations (specific BIN-drug-state triples) that were hard to write out as explicit rules.

Then it started drifting.

The temporal drift problem

A payer's network could change overnight while the model kept confidently predicting RxGate. Retraining helped, but between the contract change and the updated model, we were still sending prescriptions to the wrong pharmacy.

We needed a model that stayed current without retraining.

Rather than asking what these insurance features predict, we started with past routings. Which pharmacy did we actually route to last time for a prescription that looked like this one? And how recently?

Instead of a frozen feature snapshot, the model uses a live history of routing outcomes, weighted by recency. That evidence updates as new routings come in, so predictions can change without retraining.

We shipped this as a three-stage pipeline.

Stage 1: Rule matching via powerset

Say a prescription comes in for Humira, with insurance BIN 610014, PCN OHCARD, in Texas. In practice, we have up to 8 features, but many are null. We search every combination of at least two available features for matching successful routings. This lets us find evidence using whatever data we have. The powerset approach substantially outperformed approaches that treated missing features as unknowns.

In the illustrative example below, all four features match 48 successful routings. BIN + PCN + Drug matches 144. BIN + PCN matches 960. We keep every matching rule. If State is missing, the four-feature match disappears, but rules built from the other features still provide evidence.

A four-field rule is more specific than a two-field rule. The number of fields is itself a model feature. But we don't just pick the most specific rule. When broad and specific rules disagree, that disagreement is useful. Maybe Humira has always gone to a different pharmacy than the plan's default. Or maybe PineCare Pharmacy’s contract recently expired and the broad rule hasn't caught up yet. The model uses evidence across the matching rules, including how much they agree.

Stage 2: Evidence retrieval and temporal decay

For each matching rule, we pull historical routings with timestamps. In the BIN + PCN + Drug history below, older routings mostly went to PineCare Pharmacy, while the recent routings went to RxGate. The raw counts say PineCare Pharmacy. The recent pattern says RxGate.

Pharmacy selection patterns can change over time, so we give recent evidence more weight using a time decay function.

From these weighted results, we extract four signals:

Time-weighted most common pharmacy: For each pharmacy, sum the weights of its historical routing cases. The pharmacy with the highest total becomes that rule’s top pharmacy.

Time-weighted percent match: How much of the weighted evidence agrees on the winner? A rule at 95% is a much stronger signal than one at 55%.

Cross-tier agreement: Do rules at different specificity levels agree? Agreement is a strong signal. A split may suggest something is changing.

Evidence freshness: The weighted average age of the evidence, using the same time decay weights. A low number means the prediction is based on recent routings.

Stage 3: XGBoost prediction

One XGBoost model scores each matching rule using its weighted results, evidence freshness, routing count, and the number of fields in the rule, along with agreement across rules. The score estimates how likely that rule’s top pharmacy is to be a valid choice. We use the highest-scoring rule’s pharmacy as our prediction and its score as our confidence.

V3 let us use fresh routing evidence without retraining. But we kept hitting cases where the answer was buried in a faxed PA letter or a patient text message that our feature vector didn’t capture.

We used LLM extractors running before the ML models. One reads PA approval letters and pulls out the mandated pharmacy. Another parses free-text doctor notes (“send to PineCare Pharmacy #4537 in Dallas”, “use the specialty pharmacy from last time”). Another extracts preferences from patient text messages. Each extracted name is fuzzy matched against our pharmacy database to select a specific pharmacy.

The limits of isolated extraction

At this point, our cascade had 11 steps. Signals often conflicted. The waterfall resolved them using the priority order we set. Doctor note beats insurance rule. Insurance rule beats routing history. But a doctor note from a year ago saying "send to RxGate" shouldn't override a fresh insurance rule pointing to PineCare Pharmacy. The fixed priority order can’t account for why a signal might be stale or which one should win in a given case. And the edge cases kept growing.

When we started this work, LLMs weren't capable enough to handle this kind of reasoning reliably. By the time we'd built V3 and V4, the models had caught up. We could now give an agent the same context and tools that our clinical experts have, and trust it to reason across conflicting signals.

Those tools let it search pharmacies, read documents (including insurance cards via vision), check dispensing restrictions, look up insurance details, request patient preferences, schedule phone calls, store a pharmacy selection, or escalate to an expert. It reads everything, decides which pieces of information to rely on, investigates further if it needs to, and then acts.

Take Maria. The doctor’s note is a year old, she has moved states, and the EHR preferences conflict with the note. The agent can weigh those signals together and ask Maria for updated information before choosing a pharmacy.

The first version of the agent performed worse than the waterfall in many cases. But when it got things right, its reasoning reflected the kind of cross-signal thinking that our clinical experts exhibit.

Shipping safely: shadow mode

We couldn't A/B test with live prescriptions flowing through our system, so we deployed the agent in shadow mode. Here, the agent runs on real data, reading documents and searching pharmacies, but its actions write to non-production pathways.

Shadow mode lets us compare the agent's decisions against the existing rules engine on every prescription for weeks before enabling any real actions. During rollout, we allowlisted specific actions like storing a pharmacy selection and escalating to an expert. As we validated the shadow data, we unlocked more actions.

When we find something that works, we want to exploit it. Successfully routing 50 prescriptions with BIN 610014 + PCN OHCARD to PineCare Pharmacy in the last 3 months is a strong signal. But if PineCare Pharmacy's contract ends, the first few routings will fail. Each failure is a patient without medication. How do we get signal early that something has changed?

Time decay lets old evidence fade as new successful routings arrive, but it doesn’t tell us that a network has changed. The decay rate is a tradeoff. If we decay evidence too quickly, we discount useful history. If we decay it too slowly, outdated routings retain too much influence.

You can't explore by trying an uncertain pharmacy on a real patient's prescription. But the agent can gather information instead. A denial from a previously valid pharmacy or a provider note pointing somewhere new can prompt it to request a patient preference, suggest a phone call to verify network status, or escalate to a clinical expert. It converts the explore problem from “try something risky” to “gather more information before deciding.”

We're running agents like these in production right now, resolving cases like Maria's every day. If you're interested in tackling problems like temporal ML, explore/exploit under distribution shift, self-improving agents, and building reliable AI for high-stakes decisions with incomplete data, come join us.

Acknowledgments

The multiple iterations on our pharmacy routing algorithm required thoughtful investment across our Engineering team. Special thanks to Anirudh Jain, Sang Jun Park, Ishaan Radia, and James Xu for their efforts in building this system.