Route LLM requests with Ruby.

ngx_mruby extends nginx with Ruby. Check client keys and route LLM requests by model before they reach an API. Keep upstream credentials out of your apps and change routing rules at the proxy.

Try model routingRead the proxy guide

v3 · In development · Stable 2.7 installation and reference

Route each model to its API

A configuration for apps using several models, such as chat and document summarization. Requests go to the same URL; the model and your Ruby mapping determine the upstream.

Request from your app

POST /v1/responses

Authorization: Bearer local-client{"model":"mock-model-a"}

Ruby routing rule

    "mock-model-a" => ["local_api_a", "Bearer local-upstream-a"],

See the complete Ruby mapping

route.rb

  routes = {
    "mock-model-a" => ["local_api_a", "Bearer local-upstream-a"],
    "mock-model-b" => ["local_api_b", "Bearer local-upstream-b"]
  }

Forward to API A

local_api_aAuthorization: Bearer local-upstream-a

200 · mock response

{
  "upstream":"a",
  "model":"mock-model-a",
  "credential_replaced":true
}

An illustration of the local mock you can run below. Selecting a model here does not send a network request.

An unknown client key gets 401; an unmapped model gets 403. For an accepted request, nginx applies the upstream credential and relays the response. Messages and Responses keep their original API formats.

Authentication, concurrency limits and streamed responses

Additional integration tests: concurrency and streamed relay

Ruby handleragent_proxy_route.rbRUBY

r = Nginx::Request.new
v = Nginx::Var.new
routes = {
  "mock-model-a" => ["agent_proxy_llm_a", "Bearer upstream-test-key-a"],
  "mock-model-b" => ["agent_proxy_llm_b", "Bearer upstream-test-key-b"]
}
route = routes[JSON.parse(r.body)["model"]]
if route
  v.agent_proxy_backend = route[0]
  v.agent_proxy_upstream_auth = route[1]
  Nginx.return Nginx::DECLINED
else
  Nginx.return Nginx::HTTP_FORBIDDEN
end

Show checked output (mock)

test/html/agent_proxy_route.expected: the values that the case compares

# What test/t/cases/agent_proxy.rb compares for routing by model: status,
# port of the upstream, Authorization and x-api-key at the upstream, and
# whether the upstream got the client's body unchanged.
model_a=200 12372 Bearer upstream-test-key-a none true
model_b=200 12373 Bearer upstream-test-key-b none true
unknown_model=403

Show nginx.conf

test/conf/conf.d/60-agent-proxy.conf

upstream agent_proxy_llm_a {
    server 127.0.0.1:12372;
}

upstream agent_proxy_llm_b {
    server 127.0.0.1:12373;
}

# Concurrent requests per client key: agent_proxy_auth.rb sets the key.
limit_conn_zone $agent_proxy_stream_key zone=agent_proxy_streams:1m;

server {
    listen 18132;

    # Declares the variables that the hooks assign; never requested.
    location = /.agent-proxy-vars {
        internal;
        set $agent_proxy_key_id "";
        set $agent_proxy_stream_key "";
        set $agent_proxy_backend "";
        set $agent_proxy_upstream_auth "";
        return 404;
    }

    # Most variables stay unset on most requests (a rejected key, a
    # request that no hook routed).
    uninitialized_variable_warn off;

    # The client key, checked for every request of this server before a
    # location is chosen. A request without a known key gets 401, and the
    # server-level error_page gives it a JSON body.
    mruby_server_rewrite_handler build/nginx/html/agent_proxy_auth.rb;
    error_page 401 @agent_proxy_401;

    # The upstream gets the credential that agent_proxy_route.rb picked and
    # never the client's.
    proxy_http_version 1.1;
    proxy_set_header Connection close;
    proxy_set_header Authorization $agent_proxy_upstream_auth;
    proxy_set_header x-api-key "";
    proxy_buffering off;

    # The Anthropic Messages API, and the OpenAI Responses API below with
    # the same recipes.
    location = /v1/messages {
        # One request at a time per client key; a second one gets 429 from
        # @agent_proxy_too_many. limit_conn_status is a status that no
        # upstream sends, so that an upstream's own 429 passes unchanged.
        limit_conn agent_proxy_streams 1;
        limit_conn_status 460;
        limit_conn_log_level notice;
        error_page 460 = @agent_proxy_too_many;

        # Picks the upstream block and the credential by the model of the
        # JSON body.
        mruby_enable_read_request_body on;
        mruby_access_handler build/nginx/html/agent_proxy_route.rb;

        # Logs $upstream_status after the response.
        mruby_log_handler build/nginx/html/agent_proxy_log.rb;

        proxy_pass http://$agent_proxy_backend/v1/messages;
    }

    # The zone of limit_conn is per client key, so a stream of a key on
    # either location holds the slot of that key on both.
    location = /v1/responses {
        limit_conn agent_proxy_streams 1;
        limit_conn_status 460;
        limit_conn_log_level notice;
        error_page 460 = @agent_proxy_too_many;

        mruby_enable_read_request_body on;
        mruby_access_handler build/nginx/html/agent_proxy_route.rb;
        mruby_log_handler build/nginx/html/agent_proxy_log.rb;

        proxy_pass http://$agent_proxy_backend/v1/responses;
    }

    # The proxy's own error bodies, in the error format of the API that the
    # client asked for: a named location keeps the request's $uri.
    location @agent_proxy_401 {
        default_type application/json;
        if ($uri = /v1/responses) {
            return 401 '{"error":{"message":"invalid API key","type":"invalid_request_error","param":null,"code":null}}';
        }
        return 401 '{"type":"error","error":{"type":"authentication_error","message":"invalid API key"}}';
    }

    location @agent_proxy_too_many {
        default_type application/json;
        add_header retry-after 1 always;
        if ($uri = /v1/responses) {
            return 429 '{"error":{"message":"too many concurrent requests","type":"rate_limit_error","param":null,"code":null}}';
        }
        return 429 '{"type":"error","error":{"type":"rate_limit_error","message":"too many concurrent requests"}}';
    }
}

On every CI run, the case test/t/cases/agent_proxy.rb sends the requests of its client and compares the values that the client prints with the values shown on this page. The four files last changed on 2026-10-29 in commit 300e251c.

Try it locally without a provider key

Start nginx and two mock APIs with Docker Compose. Change the model to choose an API, then edit the Ruby mapping to reroute the same request. The first run builds from source.

Start locally

git clone --branch next https://github.com/matsumotory/ngx_mruby.git
cd ngx_mruby/examples/start
docker compose up --build --wait

The walkthrough includes the request, expected response, edit-and-retry steps and shutdown. This mock example is separate from a real-provider integration test.

Run and customize the example

Write HTTP and TCP policies in Ruby

Choose a backend for each tenant

Select an upstream from a request header and your routing table.

HTTP configuration and example

Check the client before relaying TCP

Check the client address against an allowlist before opening the upstream connection.

TCP configuration and example

Documentation for your implementation

Guides, API reference and operationMCP use-case design and unverified scopev3 implementation status and planMarkdown index for agents