For developers building apps with multiple LLM APIs

Manage LLM routing outside your apps.

ngx_mruby is a module that runs Ruby inside nginx. In v3, manage LLM routes and provider keys at your proxy. Change the upstream settings while your apps keep the same endpoint.

Run it locally See how model routing works

v3 · In developmentStable 2.7 documentation

One proxy URL. The right API for each model.

Your app sends a model name. Ruby selects the upstream and its credential; nginx forwards the request and returns the response.

Your app

POST /v1/responsesOne URL, the same API format

ngx_mruby

Check the client key
Select the route by model
Apply the provider key

LLM API

Model AProvider key A

Model BProvider key B

A/B are test models from next. Selecting one changes this illustration without sending an API request.

Change routes at the proxy

Define the model-to-API mapping in Ruby. Your apps can keep the same proxy URL when that mapping changes.

Keep provider keys out of clients

Check the client’s key, then replace it with the provider credential before forwarding upstream.

Messages / Responses relay is covered by integration tests on next. API formats are not translated. Ruby can also supply the client key for nginx’s limit_conn to bound concurrent requests. Status and requirements

Start with one request and one Ruby handler.

Begin with the Hello example; no LLM key is needed. Run the smallest nginx-to-Ruby setup, then extend that handler into model routing.

Use Docker Compose, git and curl in examples/start from the next branch. The first run builds nginx and mruby from source.

Start nginx

docker compose up --build --wait

Edit the handler

hello.rbChange Hello to another greeting

See the new response

curl localhost:8080/helloWithout cache, the next request reads your edit

Open the complete setup walkthrough Installing into nginx on Linux

Then put your routing rules in Ruby.

After the first handler, read the request’s model and choose an upstream. This code is loaded directly from the integration test.

Ruby handleragent_proxy_route.rb

r = Nginx::Request.new
v = Nginx::Var.new
routes = {
  "mock-model-a" => ["agent_proxy_llm_a", "Bearer upstream-test-key-a"],
  "mock-model-b" => ["agent_proxy_llm_b", "Bearer upstream-test-key-b"]
}
route = routes[JSON.parse(r.body)["model"]]
if route
  v.agent_proxy_backend = route[0]
  v.agent_proxy_upstream_auth = route[1]
  Nginx.return Nginx::DECLINED
else
  Nginx.return Nginx::HTTP_FORBIDDEN
end

Show checked output (mock)

# What test/t/cases/agent_proxy.rb compares for routing by model: status,
# port of the upstream, Authorization and x-api-key at the upstream, and
# whether the upstream got the client's body unchanged.
model_a=200 12372 Bearer upstream-test-key-a none true
model_b=200 12373 Bearer upstream-test-key-b none true
unknown_model=403

Show nginx.conf

upstream agent_proxy_llm_a {
    server 127.0.0.1:12372;
}

upstream agent_proxy_llm_b {
    server 127.0.0.1:12373;
}

# Concurrent requests per client key: agent_proxy_auth.rb sets the key.
limit_conn_zone $agent_proxy_stream_key zone=agent_proxy_streams:1m;

server {
    listen 18132;

    # Declares the variables that the hooks assign; never requested.
    location = /.agent-proxy-vars {
        internal;
        set $agent_proxy_key_id "";
        set $agent_proxy_stream_key "";
        set $agent_proxy_backend "";
        set $agent_proxy_upstream_auth "";
        return 404;
    }

    # Most variables stay unset on most requests (a rejected key, a
    # request that no hook routed).
    uninitialized_variable_warn off;

    # The client key, checked for every request of this server before a
    # location is chosen. A request without a known key gets 401, and the
    # server-level error_page gives it a JSON body.
    mruby_server_rewrite_handler build/nginx/html/agent_proxy_auth.rb;
    error_page 401 @agent_proxy_401;

    # The upstream gets the credential that agent_proxy_route.rb picked and
    # never the client's.
    proxy_http_version 1.1;
    proxy_set_header Connection close;
    proxy_set_header Authorization $agent_proxy_upstream_auth;
    proxy_set_header x-api-key "";
    proxy_buffering off;

    # The Anthropic Messages API, and the OpenAI Responses API below with
    # the same recipes.
    location = /v1/messages {
        # One request at a time per client key; a second one gets 429 from
        # @agent_proxy_too_many. limit_conn_status is a status that no
        # upstream sends, so that an upstream's own 429 passes unchanged.
        limit_conn agent_proxy_streams 1;
        limit_conn_status 460;
        limit_conn_log_level notice;
        error_page 460 = @agent_proxy_too_many;

        # Picks the upstream block and the credential by the model of the
        # JSON body.
        mruby_enable_read_request_body on;
        mruby_access_handler build/nginx/html/agent_proxy_route.rb;

        # Logs $upstream_status after the response.
        mruby_log_handler build/nginx/html/agent_proxy_log.rb;

        proxy_pass http://$agent_proxy_backend/v1/messages;
    }

    # The zone of limit_conn is per client key, so a stream of a key on
    # either location holds the slot of that key on both.
    location = /v1/responses {
        limit_conn agent_proxy_streams 1;
        limit_conn_status 460;
        limit_conn_log_level notice;
        error_page 460 = @agent_proxy_too_many;

        mruby_enable_read_request_body on;
        mruby_access_handler build/nginx/html/agent_proxy_route.rb;
        mruby_log_handler build/nginx/html/agent_proxy_log.rb;

        proxy_pass http://$agent_proxy_backend/v1/responses;
    }

    # The proxy's own error bodies, in the error format of the API that the
    # client asked for: a named location keeps the request's $uri.
    location @agent_proxy_401 {
        default_type application/json;
        if ($uri = /v1/responses) {
            return 401 '{"error":{"message":"invalid API key","type":"invalid_request_error","param":null,"code":null}}';
        }
        return 401 '{"type":"error","error":{"type":"authentication_error","message":"invalid API key"}}';
    }

    location @agent_proxy_too_many {
        default_type application/json;
        add_header retry-after 1 always;
        if ($uri = /v1/responses) {
            return 429 '{"error":{"message":"too many concurrent requests","type":"rate_limit_error","param":null,"code":null}}';
        }
        return 429 '{"type":"error","error":{"type":"rate_limit_error","message":"too many concurrent requests"}}';
    }
}

On every CI run, the case test/t/cases/agent_proxy.rb sends the requests of its client and compares the values that the client prints with the values shown on this page. The four files last changed on 2026-10-29 in commit 300e251c.

Continue with authentication, configuration and streaming

Architecture candidate · MCP integration unverified

Keep search and heavy document work on separate backends.

The same routing approach can sit in front of an MCP service. Keep one client endpoint while directing search and source-document processing to separate backends.

MCP URL

Ruby routing rules

SearchDocuments / source files

Authorization, discovery and protocol validation remain the service’s responsibility. ngx_mruby itself is not an MCP server.

Read the design and its boundaries

Planned · not available yet

What v3 will add

Shared usage accounting, processing individual stream events and first-byte deadlines are on the development plan. Their status is tracked separately from the relay features that work today.

See the status of each capability