October 30, 2026

Product·8 min read

Building reliable EHR automation with agents and deterministic workflows

Shayaan AzeemMTS Intern

TL;DR: Forus automates prior authorization, which means pulling clinical data out of providers’ EHRs. We used to hand-write a browser script for every EHR, so handing an agent a browser felt tempting, but risky. I built an authoring system where an agent explores the EHR, proposes each action, a human approves it with the target highlighted in the live browser, and only the executed actions are included in a deterministic Playwright script.

Why not just give an agent a browser?

The easiest way to get data out of an unfamiliar electronic health record (EHR) is to give an agent a browser and let it figure things out, but this means finding a lot of other stuff too. The same agent that finds a lab result can also delete a chart or edit a prescription, and it only has to misread one button.

At Forus we hit this problem constantly. We automate prior authorization (PA), the process where an insurer requires approval before covering a medication, and part of that work is extracting clinical information from a provider’s EHR system. Different practices and specialties use different EHRs, and we often lack access to a reliable API for the information we need to extract. Historically, we used deterministic Selenium scripts to extract data. Those scripts are brittle and require configuration for each EHR.

I wanted the agent’s ability to explore a new system without ever letting it act unsupervised in production. Here’s how I built an authoring system to do that.

Agents for authoring, but not for execution

As Forus scales, we face the challenge of integrating with many new EHRs so we can work with more practices. Each EHR is unique; even instances of the same EHR can have different interfaces and selectors. Writing and maintaining browser automation for all those workflows and edge cases takes significant engineering time.

The tempting answer was to give an agent a browser and ask it to extract the data on every run. But this was a naive approach for this workload:

Because we run it so often, we'd be paying a lot for a model to repeatedly work out navigation that we could already describe in code.
More importantly, this was a provider's EHR. We would be giving an agent access to an authenticated system with patient records and controls; it could do more than read information.

Even if our goal was extraction, the model still had to decide where to click, and it could be wrong. We couldn’t take that risk because an agent could unintentionally delete or edit a patient record.

I wanted to build a system that let me use the agent's capabilities while building the automation, then preserve the workflow as deterministic code for subsequent runs. Generated scripts can still break when an EHR changes. This approach reduces the time it takes to author and revise them while keeping execution deterministic. The existing scripts used Selenium; the authoring system I built generated Playwright code because it offered more reliable element targeting, automatic waiting, and better support for dynamic EHR interfaces.

Three design constraints

A tool to author code with a human in the loop.

Reusable Scripts: The output should run in production without a person approving every action.

Fallible Code: An agent can generate deterministic code that’s wrong.

Human Review: Human review and approval had to be an explicit part of generating and authoring code.

Showing the human exactly what the agent will click

The authoring system connects an agent to a live browser through MCP. The server exposes tools for observing the page, inspecting the DOM and network activity, and performing actions.

The agent would propose an action, but I needed evidence that it would do what we expected. For example, it might propose clicking #patient-chart-save, an illustrative EHR selector. The name suggests a button that saves the patient’s chart, but the selector alone doesn’t show which control it matches or what clicking it will do. We didn’t want to approve an ambiguous action.

I realized I could manipulate the DOM to highlight the agent’s target in the live browser, then wait for a person to approve the action. This was much better because the human could visually confirm what the agent was about to do. Only after the action runs successfully can its corresponding code enter the script.

The distinction between observing and acting matters.

observe_page returns interactive elements with reusable selectors.
inspect_dom supplies a pruned structural view for unusual controls.
inspect_network helps locate requests behind page content.

These discovery tools don’t add steps to the generated script.

Tools such as act, capture_response, and extract_to_results perform workflow actions or produce script steps and require approval.

The highlight gives the operator something clear to evaluate or approve.

If the operator rejects the action, they can explain why, giving the agent context for its next proposal. If they approve it, the server executes the action and returns the corresponding Playwright statement once the action succeeds.

Approving a click isn’t approving the code

Human approval of tool calls left another gap. The agent was also editing a code file. It could execute approved actions through our tools, then independently add a plausible action to the script.

To close that gap, I added a ledger that tracks actions that have been executed and tested.

A simplified entry in ledger.jsonl:

{
  "timestamp": "2026-08-01T12:00:00Z",
  "tool": "act",
  "line": "await page.get_by_role(\"button\", name=\"Search\").click()"
}

Before a script edit, a check requires any added browser action lines to appear in that ledger. Failed actions don't produce a ledger entry.

This gives us two distinct checks:

Agent to live browser. Human approval, with the target highlighted where applicable. Review what is about to happen in the EHR.
Agent to generated script. Action line must exist in the executed action ledger. Keep unexecuted actions out of the output.

The second check matters because a browser session approved by a human and the generated file aren’t automatically the same thing. I needed to constrain both.

The ordering inside the action path matters. This is a simplified reconstruction:

async def act(selector, action, value=None, parameter=None):
    line = render_action(selector, action, value, parameter)
    locator = resolve_selector(page, selector)

    await require_fresh_human_approval(locator)
    await perform(locator, action, value)

    ledger.append(line)
    return {"canonical_line": line}

Rendering happens before execution, so invalid action combinations fail before anyone is asked to approve them. Appending happens after execution, so a failed click can't become a verified script step.

The check before writing then compares browser action lines against those recorded statements. This is an enforceable contract to ensure determinism.

What does this selector actually point to?

For highlighting to be useful, the highlighter has to know what a selector actually addresses and where it is. EHR websites make this surprisingly annoying.

The pages can contain nested iframes, duplicate IDs, custom controls, and frames that don’t respond when inspected. A selector that looks reasonable in a text description may not do what it seems to do, or even match an existing element.

For ordinary controls, the observe_page tool prefers accessible role and name, then labels, unique IDs, and scoped CSS selectors.

The inspect_dom tool returns a structural DOM view with scripts, styles, and irrelevant containers removed so the agent can inspect custom controls without reading the entire page source.

The agent is instructed to reuse the selectors emitted by the tooling.

When the frame name isn’t a name

One recurring issue when highlighting or finding a selector is that Playwright's reported frame name can fall back to the iframe element's ID. Consider this element:

<iframe id="GlobalNavigation"></iframe>

A frame name of GlobalNavigation doesn’t mean the element has a name attribute. Generating iframe[name="GlobalNavigation"] would match nothing.

The fix was to inspect the actual iframe element in its parent document. A candidate selector has to match exactly one element, and that element has to be the frame being inspected. A simplified version of that check looks like this:

function verifiedSelector(iframe, candidates) {
  const root = iframe.getRootNode();

  for (const selector of candidates) {
    const matches = root.querySelectorAll(selector);
    if (matches.length === 1 && matches[0] === iframe) {
      return selector;
    }
  }
  return null;
}

Candidate generation tries escaped IDs, actual name attributes, and positional fallbacks to tie the selector back to the frame.

For a control inside nested frames, a single frame selector isn't enough. The tool walks the frame's ancestry back to the main document, reverses it, and derives a selector for each frame boundary. The resulting locator preserves that path:

target = (
    page.frame_locator("#workspace")
    .frame_locator("#patient-chart")
    .get_by_role("button", name="Lab results")
)

The representation also needs to distinguish two cases that are easy to conflate accidentally in Python:

[] means the element is in the main document.
None means an addressable frame chain can’t be established.

A check like if not chain would treat both alike and could cause a child frame element to be emitted as if it belonged to the main page. Unaddressable frames have to be skipped explicitly.

There are practical performance constraints too. Sibling frames often share ancestors, so the tool caches derived frame selectors within an observation pass.

Keeping the cache scoped to that pass avoids treating a selector from an earlier DOM state as indefinitely valid.

Highlighting uses that same frame chain. If the target is missing, ambiguous, or can't be highlighted, the approval prompt reports that instead of telling the human to look for an outline that isn't there.

Where agents belong

The result was a system where agents could explore unfamiliar EHRs and help write browser automations much quicker, while the actual workflows remained deterministic.

This project taught me to think carefully about where to use agents. An agent could help discover and author an unfamiliar EHR workflow, with a human reviewing its actions. We could then test and reuse the resulting code without asking a model to work out the navigation on every run. Not everything needs to be agentic, and I came away with a greater appreciation for simple, deterministic systems.

This was one of the projects I worked on during my time as an Intern at Forus. I wrote more about my experience and what I learned during my internship here.

If you’re interested in browser automation, agent tooling, and deciding where to use models in production, we’re hiring.

When the frame name isn’t a name

Read next