OpenAI-Compatible API: Serve One From Your Own Agent

2026-11-11

An OpenAI-compatible API is a server that accepts the same /v1/chat/completions request and returns the same response shape as OpenAI’s API, so any client built for OpenAI works by changing its base URL. POLYROB, the open-source (MIT) self-hosted AI agent, can serve one: turn on a flag and an OpenAI SDK can talk to your agent, with a few documented refusals you should know before you build on it.

An OpenAI-compatible API is defined by its wire format, not by who runs it. If a server answers POST /v1/chat/completions with a choices array and lists models at GET /v1/models, the official SDKs and most tools that speak OpenAI can use it. The OpenAI Python SDK takes a base_url argument or the OPENAI_BASE_URL environment variable for exactly this reason.

Most pages on this topic cover model gateways and inference hosts. POLYROB sits on both sides of the idea, and this page covers both: how it serves the endpoint with an agent behind it, and how it uses other OpenAI-compatible servers as its model.

What does an OpenAI-compatible API give you?

It gives you portability on the client side. Code written against one OpenAI-shaped server keeps working against another, because only the base URL, the key and the model name change. That is why local model servers such as Ollama, LM Studio and vLLM all expose the shape, and why aggregators and company gateways do too.

There are two directions, and people searching this phrase usually mean one of them:

You call an OpenAI-compatible server. Your tool or agent is the client. POLYROB does this when you declare a server in ~/.polyrob/providers.yaml.
You expose one. Other programs call you as if you were OpenAI. POLYROB does this through its /v1 router.

How does POLYROB serve an OpenAI-compatible API?

POLYROB mounts two routes when you switch on OPENAI_COMPAT_API_ENABLED: POST /v1/chat/completions and GET /v1/models. The flag is off by default (docs/guide/[email protected], api/openai_compat/[email protected]). Set it, then start the server:

polyrob config set OPENAI_COMPAT_API_ENABLED true
polyrob serve

polyrob serve binds to 127.0.0.1:9000 by default and exits with a clear message if no model provider key is configured (docs/guide/[email protected]). Then point the OpenAI SDK at it:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:9000/v1", api_key="<your POLYROB API key>")

reply = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Summarize https://example.com"}],
)
print(reply.choices[0].message.content)

What answers is not a bare model call. The request goes to POLYROB’s chat agent, which builds its system prompt from its own identity, skills and tools and keeps its own session history. The router describes it as a tool-light chat agent (api/openai_compat/[email protected], docs/guide/[email protected]).

Which key do you send?

Two different keys are in play, and mixing them up is an easy first mistake. The key you send to /v1 is a POLYROB API key, an inbound credential that identifies the caller to POLYROB. It is not your OPENAI_API_KEY or any other model provider key, which is the outbound credential POLYROB uses to reach a model (docs/guide/[email protected]).

Send it as X-API-KEY, or as Authorization: Bearer <api-key>, which is what the OpenAI SDK sends. New keys start with polyrob_. Minting one with POST /api/auth/api-keys needs the account system: it answers 503 with the remedy until you set ENABLE_AUTH=true, configure a database and restart. Using an existing key never needs that (docs/guide/[email protected]).

How does POLYROB map the model name?

The model string you send decides which provider answers. The router resolves it in this order (api/openai_compat/[email protected]):

A model id that one provider’s catalog owns exactly goes to that provider, preferring one you have a key for.
A provider/model slug names the provider outright, for example ollama/qwen3-coder:30b. This works for any provider you declared in providers.yaml.
A known prefix such as gpt-, claude- or gemini- routes to that provider.
Anything else goes to your default provider, with the string passed through unchanged.

The last rule has a consequence: an unrecognized model name never returns a 400. A typo gets answered by your default provider instead of failing, so check the name against GET /v1/models, which lists each available provider’s default plus every declared model. The response echoes your model string back verbatim even when routing used a different pair (api/openai_compat/[email protected]).

What does the endpoint refuse or ignore?

A compatibility layer that silently drops parameters is hard to debug, because a client cannot tell a refusal from a bad answer. POLYROB’s source says this in a comment, and the router refuses what it cannot honour with a 400 that names the field and the reason (api/openai_compat/[email protected]).

Request field	What happens
`tools`, `tool_choice`, `functions`, `function_call`	Refused with 400. POLYROB runs its own tools; caller-supplied tools are not executed.
`response_format`	Refused with 400. Structured-output modes are not supported.
`stop`, `logprobs`, `top_logprobs`, `seed`, `logit_bias`	Refused with 400.
`n` above 1	Refused with 400. One choice is returned.
`max_tokens`, `max_completion_tokens`, `top_p`, `presence_penalty`, `frequency_penalty`	Accepted and ignored.
`temperature`	Passed to the turn.
Image, audio or file content parts	Refused with 400 and the part type. Text parts only.

A field that is present but null or empty does not trigger the refusal (api/openai_compat/[email protected]).

Why does only the last message run?

The endpoint is single-turn. Only the last user message in messages is executed, and system messages are not applied. POLYROB keeps its own conversation history, keyed on the user field or else your authenticated id, so replaying a client-side transcript would duplicate it. The agent’s system prompt comes from its own identity and skills (docs/guide/[email protected]).

In practice: send one new user message per call and let the server hold the thread. Use a stable user value to keep one conversation, and a different one to start another.

How does streaming and usage work?

With stream: true you get server-sent events, but the stream is buffered. The agent turn runs to completion and the full reply arrives as one content chunk, so there are no token-by-token deltas yet. While a long turn runs, the stream sends an SSE comment frame (: keep-alive) every 15 seconds so clients and proxies do not time out. If the turn fails after the headers went out, the error arrives in the OpenAI error shape inside the stream (api/openai_compat/[email protected]).

Pass stream_options with include_usage to get OpenAI’s final usage chunk. The usage numbers are an estimate: the response carries usage.estimated: true, and the counts are recomputed from the request text and the reply. They leave out the system prompt, skills, memory recall, tool schemas and every intermediate agent step, so treat them as a floor and not a bill. Authoritative cost is in the instance’s own usage records (docs/guide/[email protected], api/openai_compat/[email protected]).

Errors on /v1 use the OpenAI envelope, {"error": {"message", "type", "code", "param"}}, so an SDK reports the real reason. The one exception is 402 on a billed route, which carries a payment challenge and is passed through untouched. A non-streamed turn is cancelled if the client disconnects (docs/guide/[email protected], [email protected]).

How does POLYROB call other OpenAI-compatible APIs?

The other direction needs no flag. Declare the server in ~/.polyrob/providers.yaml with transport: chat_completions, and POLYROB uses it as a model provider: a local Ollama, LM Studio, vLLM or llama.cpp server, an aggregator, or a company gateway (docs/guide/[email protected]). The exact file, the default-model rules and the loader’s checks are in the guide to running an LLM locally.

Declared models also become reachable through the /v1 endpoint, so one declared local row can serve other programs on the same machine through the slug form.

Where does the POLYROB endpoint fall short?

It is an agent endpoint, not a model proxy. You cannot pass your own tools, force JSON output or set a stop sequence.
Single-turn only, and system messages are ignored.
No true token streaming. The reply arrives in one chunk.
Token counts are estimates, and no speed or throughput figures are published, so none are claimed here.
It is off by default and needs an API key.
POLYROB is a young project with a small community.

If you need a transparent gateway to many model hosts, a dedicated gateway fits better. If you want your own agent reachable from tools that only know the OpenAI shape, this endpoint does that.

Frequently asked questions

Is the POLYROB OpenAI-compatible API on by default?

No. Set OPENAI_COMPAT_API_ENABLED=true and restart the server. Until then the /v1 routes are not mounted.

Can I use function calling through POLYROB’s /v1 endpoint?

No. A request that carries tools or functions is refused with a 400 that names the field. POLYROB runs its own toolset, which includes an MCP client for adding servers.

Which model does POLYROB use when I send an unknown name?

Your default provider, with the string passed through. It does not return an error, so confirm names with GET /v1/models.

Can I use the OpenAI Python SDK with POLYROB?

Yes. Set base_url to http://localhost:9000/v1 and use a POLYROB API key as the api_key. The SDK sends it as a Bearer header, which POLYROB accepts.

Does it work with Claude or local models?

Yes. The model field can name any provider you have configured, including Anthropic models and declared local servers. See the Claude AI agent guide for the Anthropic side.

Next steps

Install POLYROB from the install page, read the docs, and see what self-hosting an AI agent involves. The open-source AI agent overview covers the wider picture, and the code for the endpoint is in the public repository. To point the agent at models you host yourself, read the guide to self-hosted AI models.

Sources, all in https://github.com/theselfruleorg/polyrob at tag v1.7.7: api/openai_compat/router.py, api/openai_compat/models.py, api/openai_compat/model_map.py, docs/guide/api.md, docs/guide/configuration.md, docs/CONFIGURATION.md, CHANGELOG.md.

POLYROB is an open-source AI agent you run and own. Install it or read the docs.