← Blog

Run LLM Locally and Connect an Agent to It

2026-11-11

To run an LLM locally, install a model server such as Ollama, pull a model, and use the OpenAI-compatible address it serves on your machine. To put an agent on top of that model, declare the address in ~/.polyrob/providers.yaml and start it with polyrob run -p ollama "hello". No API key is involved.

Running an LLM locally means a language model runs on hardware you control, so prompts are processed on your own machine instead of a provider’s servers. A model server loads the weights and answers requests on a local address. Other programs, including an AI agent, connect to that address like any other model API.

Most guides stop once the model answers in a chat window. This one goes one step further: it shows how to connect a self-hosted agent to the model you just started, with the exact configuration, the rules the loader enforces, and the limits you should know before you rely on it.

What do you need to run an LLM locally?

You need three things: a model server, a model, and enough memory to hold it.

• A model server. Ollama is a terminal tool: you pull a model and it serves it. LM Studio is a desktop app with a model browser and a local server. llama.cpp is the engine many of these tools build on, and you can run its server directly.
• A model. Pick one by size. Larger models need more RAM or GPU VRAM. Quantized builds, often shipped in the GGUF format, shrink a model so it fits on a laptop at some cost in quality.
• Memory. On a machine with a dedicated GPU, the model should fit in VRAM or it will spill into slower system memory. On an Apple Silicon Mac the CPU and GPU share one memory pool.

The best size for your machine depends on your hardware, so check the model page of whichever server you choose. POLYROB does not size models for you and publishes no speed or quality figures for any model.

How do you start a local model server?

With Ollama, two commands are enough:

ollama pull qwen3-coder:30b
ollama list

The server listens on http://127.0.0.1:11434, and its OpenAI-compatible API lives under /v1. LM Studio and llama.cpp print the address of their own server when you start it. Use that address, not a guess, in the next step.

Confirm the model answers before you add an agent. If it does not answer in the server’s own chat, no agent will fix that.

How do you connect an agent to a local LLM?

POLYROB is an open-source (MIT), self-hosted AI agent. Install it with the install page command, then declare your local server as a provider. The file is ~/.polyrob/providers.yaml:

providers:
  ollama:
    base_url: http://127.0.0.1:11434/v1
    auth_type: none
    transport: chat_completions
    default_model: qwen3-coder:30b
    models: [qwen3-coder:30b, llama3.3:70b]

Then run it:

polyrob run -p ollama "hello"
polyrob run -p ollama -m llama3.3:70b "summarize this folder"

The configuration guide lists a local Ollama, LM Studio, vLLM or llama.cpp server as supported the same way, as long as it speaks an OpenAI-compatible or Anthropic-compatible API (source: docs/guide/[email protected]). The getting-started guide shows the same Ollama row as the route for installs with no key at all (source: docs/guide/[email protected]).

What does each providers.yaml field do?

Each row names a wire shape, not a client class. These are the fields that matter for a local server:

The loader is cautious on purpose. A malformed row is skipped with a warning instead of crashing startup. A providers.yaml that is group-writable or world-writable is refused until you run chmod 600. The guide puts the reason in one sentence: “The file decides where your prompts and keys go.” (sources: modules/llm/[email protected], docs/guide/[email protected]). Keep the file under ~/.polyrob, where agent file tools are denied access to it.

How do you make the local model the default?

By default, POLYROB picks the first provider with a usable key, in this order: the six built-ins (openrouter, anthropic, openai, gemini, nvidia, deepseek), then the flat-rate and subscription rows, then the rows in your providers.yaml in file order. A hosted key already in ~/.polyrob/.env therefore wins over your local row. Pin the local one:

polyrob model set-default ollama qwen3-coder:30b
polyrob model list

polyrob model list prints every provider with its key status, models and default. In a running chat, /model <name> switches the model from the next step. The pinned model serves every run on that provider, including scheduled jobs and goals.

If your reason for a local model is that prompts must stay on your machine, also keep hosted provider keys out of ~/.polyrob/.env. POLYROB fails over to the next usable provider on a billing, quota or rate-limit error, and a hosted key gives it somewhere to go.

Can other programs use the same local model through POLYROB?

Yes. POLYROB has an OpenAI-compatible /v1 endpoint, behind a flag and off by default. The slug form addresses any provider explicitly, including one you declared: "model": "ollama/qwen3-coder:30b". GET /v1/models lists each available provider’s default plus every declared model (source: docs/guide/[email protected]). Note that only the last user message in a request is executed, because POLYROB keeps its own session history. The OpenAI-compatible API guide lists the refused fields and the model-name mapping.

What is the difference between Ollama and Ollama Cloud?

POLYROB ships a built-in row called ollama-cloud. It uses OLLAMA_API_KEY and the hosted endpoint at https://ollama.com/v1. It is not a local Ollama. If you set that key by mistake and expect local inference, your prompts go to a hosted service. For a local server, use your own row with base_url: http://127.0.0.1:11434/v1 and auth_type: none, as above (sources: docs/guide/[email protected], modules/llm/[email protected]).

What are the limits of running an agent on a local model?

• You run the server. POLYROB does not download, size or start a local model. It connects to one you already run (source: docs/guide/migration/[email protected]).
• Tool use depends on the model. An agent plans steps and calls tools, so the model has to call tools reliably. Small local models do this less well than large hosted ones. Try your model on a small task before you give it a goal.
• Limits still apply. An autonomous run stops at the step cap, 150 steps by default (AUTONOMOUS_MAX_STEPS), and at a per-run spend cap of 2.00 USD by default (AUTONOMOUS_RUN_BUDGET_USD). Autonomy itself is off until you run polyrob autonomy on (source: docs/[email protected]).
• Your data. Memory, goals and sessions are local SQLite files. With a local model, prompts stay on your machine. Tools you switch on, such as web fetch or the browser, reach their own services. Since 1.7.5 the installer sends one anonymous count after a successful fresh install (version, OS and install method, with no ID); --no-telemetry or DO_NOT_TRACK=1 turns it off (source: [email protected]).

Frequently asked questions

Can I run an LLM locally without a GPU?

Yes, on the CPU, with a small or heavily quantized model. It is slower. How slow depends on your hardware, and POLYROB publishes no figures.

Does POLYROB work offline with a local model?

The model side can. Memory and state are local files. Any tool that fetches a web page or calls a service still needs a network connection.

Which local model should I pick for an agent?

Pick one that supports tool calling, then test it on a small task. The example row above uses qwen3-coder:30b because it is the model in the project’s own documentation. It is an example, not a recommendation or a benchmark result.

Can I mix a local model and a hosted one?

Yes. Declare the local row, keep a hosted key, and pick the provider per run with -p, or switch live with /model. Pin the default so the choice is yours.

Run an LLM locally, then give it work

Running an LLM locally takes a model server and enough memory. Giving it work takes one row in providers.yaml. To see the rest of the setup, read how to self-host an AI agent, compare options in the open source AI agent guide, or see how a personal AI agent fits in your chats. The install page has the command, and the docs cover the rest. For a server on another machine, and for keeping prompts off hosted providers, see the guide to self-hosted AI models.

POLYROB is an open-source AI agent you run and own. Install it or read the docs.