Self-Hosted AI Models: Serve Them, Then Connect an Agent

2026-11-12

Self-hosted AI models are open-weight language models that you run on hardware you control, behind a server that speaks an OpenAI-compatible or Anthropic-compatible API. To use one with an agent, start the server, then add one row to ~/.polyrob/providers.yaml that points at its address.

Self-hosted AI models are language models whose weights and inference server run on a laptop, a home server or a rented GPU machine that you control, instead of behind a managed provider’s API. Your prompts go to your own server, and you decide which model version runs.

Most guides on this topic stop when the model answers in a chat window. This one covers the part after that: what to run, what it costs you in hardware and upkeep, how to connect an agent to a server on another machine, and how to stop the agent from using a hosted model when yours fails.

Why do people self-host AI models?

The reasons that come up most are the same few:

Prompts and files are processed on your own hardware, so they are not sent to a model provider.
You pick the model and the version, and the version does not change under you.
The cost is hardware and power instead of a per-token bill.
A local server keeps working when your internet connection does not, as far as the model goes.

The costs are real too. You buy or rent the hardware, you keep the server updated, and you own every outage. Small models also handle multi-step agent work less reliably than large hosted ones. POLYROB publishes no speed, quality or cost-saving figures for any model, and this page does not either. Test the model you pick on your own tasks.

What do you need to self-host an AI model?

You need a model server, a model, and enough memory to hold it. The server is the program that loads the weights and answers requests on a local address.

Server	What it is	Fits when
Ollama	A terminal tool: pull a model, and it serves it	You want the quickest start on one machine
LM Studio	A desktop app with a model browser and a local server	You prefer a window to a terminal
llama.cpp	The inference engine many tools build on, with its own server	You want direct control of the runtime
vLLM	A server for GPU machines that answers an OpenAI-style API	You run a GPU box and serve it to other machines

On a machine with a dedicated GPU, the model should fit in VRAM, or it spills into slower system memory. On an Apple Silicon Mac the CPU and GPU share one memory pool. Quantized builds, often shipped in the GGUF format, make a model smaller so it fits, at some cost in quality. The right size depends on your hardware, so check the model page of the server you choose.

Which self-hosted models work for an agent?

An agent plans steps and calls tools, so the model must call tools reliably. A model that chats well can still fail at that. Pick a model whose own page says it supports tool calling, then try it on one small task before you give it a goal.

The POLYROB documentation uses qwen3-coder:30b and llama3.3:70b in its own example. They are examples of the format, not a recommendation and not a benchmark result. POLYROB does not bundle, host or size models. It connects to a server you already run.

How do you connect an agent to a self-hosted model?

POLYROB is an open-source (MIT), self-hosted AI agent. Install it with the command on the install page, then declare your server as a provider. The file is ~/.polyrob/providers.yaml. A model server on the same machine needs no key:

providers:
  ollama:
    base_url: http://127.0.0.1:11434/v1
    auth_type: none
    transport: chat_completions
    default_model: qwen3-coder:30b
    models: [qwen3-coder:30b, llama3.3:70b]

Run it with polyrob run -p ollama "hello". The configuration guide names a local Ollama, LM Studio, vLLM or llama.cpp server as supported this way (source: docs/guide/[email protected]). For the full walk-through of a single local server, see run an LLM locally.

How do you use a model server on another machine?

A GPU box on your network is the common setup for larger models: the agent runs on a small machine, and the model runs on the box with the GPU. Declare the box by its address. If the server checks a key, name the environment variable that holds it:

providers:
  gpubox:
    base_url: http://192.168.1.50:8000/v1
    env_key: GPUBOX_API_KEY
    transport: chat_completions
    default_model: my-model
    models: [my-model]

Three rules from the loader matter here (source: modules/llm/[email protected]):

env_key must look like an API key variable, so the name must match *_API_KEY. A name that suggests a wallet, seed or token secret is rejected.
base_url must be http or https, and a cloud-metadata address is refused. Plain http to a LAN address is allowed. Traffic on a network you do not trust should go over https.
A row with no base_url is skipped, and a row with no models: list still works if you name it with -p gpubox -m my-model, but it does not appear in model listings.

A malformed row is skipped with a warning instead of stopping startup. A providers.yaml that is group-writable or world-writable is refused until you run chmod 600, because the file decides where your prompts and keys go. Keep it under ~/.polyrob, where the agent’s file tools are denied access to it.

If your server speaks the Anthropic wire format instead, set transport: anthropic_messages. Servers that need a bearer header can set bearer_auth: true on such a row (source: docs/guide/[email protected]).

What happens when the model cannot call tools?

Each row has a supports_native_tools field, and the code defaults it to true. If a provider is marked as not supporting native tools, POLYROB logs that and uses a JSON-based tool format instead of the provider’s structured function calls (source: agents/task/agent/core/[email protected]).

Set supports_native_tools: false on a row only when your server or model rejects structured tool calls. The JSON fallback gets the agent working, and how well it works depends on the model. It is a way to make a weaker model usable, not a way to make it strong.

How do you stop the agent from using a hosted model?

This is the step the generic guides skip. If your reason for self-hosting is that prompts stay on your hardware, two things can undo it.

First, the default provider. POLYROB picks the first provider with a usable key, in this order: the six built-ins (openrouter, anthropic, openai, gemini, nvidia, deepseek), then the flat-rate and subscription rows, then your own rows in file order. A hosted key in ~/.polyrob/.env therefore wins over your local row. Pin yours:

polyrob model set-default gpubox my-model
polyrob model list

Second, failover. When a model call fails on a billing, quota or rate-limit error, POLYROB tries the next provider on a fallback ladder. The ladder is made of the fallback-eligible built-ins that have a key, and a providers.yaml row joins it only if you set fallback_eligible: true on it (source: modules/llm/[email protected]). The allowlist LLM_FALLBACK_PROVIDERS takes a comma list of the providers the ladder may use. A provider outside the list is refused, including a provider with a stray key (source: docs/[email protected]).

polyrob config set LLM_FALLBACK_PROVIDERS gpubox

With that set, a failure on your server does not move the session onto a hosted provider. The cost of that choice is that a failure on your server is a failure. For a stricter setup, also keep hosted provider keys out of the environment.

How do you check the setup works?

Run polyrob model list. It prints every provider with its key status, models and default. Run polyrob doctor for a health check that includes the providers file: it shows which rows loaded and why any row was rejected (source: modules/llm/[email protected], docs/guide/[email protected]). Then send one short prompt with polyrob run -p gpubox "hello".

If you want other programs to use the same model through the agent, POLYROB has an OpenAI-compatible endpoint behind a flag. It is covered in the OpenAI-compatible API guide.

What are the limits of self-hosted models with an agent?

You run and maintain the server. POLYROB does not download, start or update a model.
Tool use depends on the model. Small models do it less reliably, so test before you hand over a goal.
Limits still apply. An autonomous run stops at the step cap, 150 steps by default (AUTONOMOUS_MAX_STEPS), and at a per-run spend cap of 2.00 USD by default (AUTONOMOUS_RUN_BUDGET_USD). Autonomy itself is off until you run polyrob autonomy on (sources: docs/[email protected], docs/[email protected]).
A built-in row called ollama-cloud is a hosted service at https://ollama.com/v1 with its own key. It is not a local Ollama. Do not set OLLAMA_API_KEY if you want local inference only (source: modules/llm/[email protected]).
A self-hosted model keeps model prompts on your hardware. Tools you switch on, such as web fetch or the browser, still reach their own services. Since 1.7.5 the installer sends one anonymous count after a successful fresh install (version, OS and install method, with no ID), and --no-telemetry or DO_NOT_TRACK=1 turns it off (source: [email protected]).

For the wider security picture, including how the agent treats web pages and mail as data, read AI agent security.

Frequently asked questions

Can I self-host AI models without a GPU?

Yes, on the CPU, with a small or heavily quantized model. It is slower, and POLYROB publishes no figures for how much slower on your hardware.

Is a self-hosted model cheaper than a hosted API?

It changes the shape of the cost: hardware, power and your time instead of a per-token bill. Whether it is cheaper depends on how much you use it and what you already own. POLYROB makes no cost-saving claim.

Can I use a self-hosted model and a hosted one together?

Yes. Declare the self-hosted row, keep a hosted key, and pick the provider per run with -p, or switch live with /model. Pin the default and set LLM_FALLBACK_PROVIDERS so the choice stays yours.

Does the agent work offline with a self-hosted model?

The model side can. Memory and state are local SQLite files. Any tool that fetches a page or calls a service still needs a network connection.

Next step: give the model a job

Self-hosted AI models need a server, enough memory and one provider row. To see the rest of the setup, read how to self-host an AI agent, or how a personal AI agent lives in your chats. The install page has the command, and the docs cover the rest.

POLYROB is an open-source AI agent you run and own. Install it or read the docs.