← Blog

AI Agent Browser: How POLYROB Drives a Real Browser

AI Agent Browser: How POLYROB Drives a Real Browser

An AI agent browser is a real browser that an AI agent drives for you: it reads the page, picks an element, clicks or types, and repeats until the task is done. POLYROB’s browser tool does this with Playwright and Chromium on your own machine, and it blocks private network addresses by default.

This page covers how the loop works, what actions the agent has, what you install, which safety limits ship in the code, and where the tool is still thin. Every POLYROB claim comes from the public repository at release v1.7.8, and the files are listed at the end.

How does an AI agent browser work?

Vercel’s open-source agent-browser project, whose README we read on 2026-11-12, describes the pattern in one line: the agent gets the page as compact text with a reference for each element, acts on a reference, reads the page again to check the result, and repeats (vercel-labs/agent-browser). Most agent browsers follow that look, think, act cycle. They differ in where the browser runs, how the page is shown to the model, and what stops it from going somewhere it should not.

POLYROB’s version is one tool inside a larger agent, not a standalone browser product. The agent that uses it also has memory, a goal board and chat surfaces. If you are weighing the whole stack, start with the guide to a self-hosted AI agent.

What can the POLYROB agent do in a browser?

The tool registers about 30 actions. Grouped by job:

The page is shown to the agent as a list of interactive elements with an index, built by a DOM script in tools/dom. The agent clicks “index 12”, not a CSS selector. For reading, extract_page_content takes the browser’s accessibility snapshot, which is cleaner than raw HTML, adds the title and URL, and cuts the text at 25,000 characters. If no snapshot is available it falls back to the page’s HTML text.

When does the agent not need a browser?

For “read this page”, POLYROB has a lighter tool, web_fetch, which turns a URL into markdown with no browser installed. The getting-started guide says only the Playwright tool, the one for clicking, forms and multi-step navigation, needs the browser engine. Use the browser when the page needs a login, a click or a form.

How do you install the browser engine?

The browser is optional. Install the extra and download Chromium:

pip install "polyrob[browser]"
python -m playwright install chromium
# on a fresh Linux server, also pull the system libraries (needs sudo):
# python -m playwright install --with-deps chromium

The installer script at polyrob.dev/install also asks whether to set up the browser, and takes --browser or --no-browser to decide in advance. Two settings matter after that. BROWSER_HEADLESS is on by default, so Chromium runs without a window. MAX_BROWSER_CONTEXTS defaults to 6, the pool-wide limit on browser contexts open at once.

What stops the agent from visiting the wrong place?

A browser renders pages written by strangers, so the tool restricts where it can go. These rules are in the code, not in a prompt.

  1. Scheme allowlist. Navigation accepts only http, https, data and about. A file:// URL is refused, in any capitalization, and so is a web URL that embeds file:/// as a redirect lure.
  2. Private addresses are blocked. Before navigating, the URL is resolved and rejected if it points to loopback, a private (RFC 1918) range, a link-local address or the cloud metadata service. This stops a page from steering the agent to an internal service. BROWSER_ALLOW_PRIVATE_URLS turns this off for local development, and only an explicit 1, true, yes or on counts; a typo keeps the guard on.
  3. Every request is checked, not just page loads. A route guard validates each image, script and fetch the page makes, so a subresource cannot reach an internal address either.
  4. The Chromium sandbox stays on. Sandbox-disabling flags are refused unless you set POLYROB_LOCAL, meant for local development.
  5. Downloads are bounded. A file the page downloads is saved through a confined writer, with a 32 MiB limit and a sanitized name.

For the wider picture of what one turn of the agent can and cannot do, see the AI agent security guide. Pages can also try to talk to the agent, which the prompt injection examples page walks through.

How do you isolate the browser from the agent?

By default the browser runs as the same user as the agent. If the agent process holds signing keys (wallet custody), POLYROB refuses to launch Chromium inside it and connects to a separate browser instead. polyrob browser install sets one up as its own system user, with a systemd unit, an AppArmor profile and an egress chain that stops the browser reaching loopback services, private ranges and the metadata service. You then set BROWSER_CDP_URL (a local endpoint on 127.0.0.1) or BROWSER_WSS_URL (a Playwright server on a second host, preferred across a network because CDP has no authentication). Every session gets a fresh context. polyrob browser status shows whether the rail is in place. The AI agent sandbox page covers the other isolation layer, for code the agent runs.

How does it compare with agent-browser?

agent-browser is a native Rust command line tool that any coding agent can call. By its README, checked on 2026-11-12, a background daemon controls Chrome over the DevTools Protocol, each named session gets its own daemon and browser, Chrome for Testing is the default, and you can attach to a running Chrome. It is licensed under Apache-2.0. POLYROB’s browser is a Python tool inside the agent, built on Playwright, with the page shown as an indexed element list. If you want a browser command for an existing coding agent, agent-browser is built for that. If you want one agent that also keeps memory and reaches you in chat, the POLYROB tool comes with it. For a wider list of hosted and open options, Mastra published a ranking titled “The 6 Best AI Agent Browser Platforms (July 2026)” (Mastra).

Where does the browser tool fall short?

AI agent browser questions

Does the agent need a browser to read a web page?

No. web_fetch returns a page as markdown without one. The browser is for pages where the agent must log in, click or fill a form.

Can the agent open a local file or localhost?

Not by default. file:// is always refused, and loopback and private addresses are blocked unless you set BROWSER_ALLOW_PRIVATE_URLS for local development.

Does it run headless?

Yes, by default. Set BROWSER_HEADLESS to false to see the window.

Can the agent upload files and download them?

It uploads files from its workspace through a file field. Downloads are saved to the workspace, up to 32 MiB each.

Where do you start?

Install POLYROB, add the browser extra, and ask the agent to open a page and fill a form on a site you control. Watch the first run in a visible window by turning headless off. Then read the docs, or browse the browser tool source. POLYROB is MIT licensed.

Sources

All POLYROB facts are from the public repository at release v1.7.8 (github.com/theselfruleorg/polyrob).

POLYROB is an open-source AI agent you run and own. Install it or read the docs.