Route LLM requests with Ruby.

Building a chat, summarization app or LLM agent? ngx_mruby extends nginx with Ruby. Put a proxy between your apps and LLM APIs to manage client authentication, model routes and upstream API keys in one place.

Try model routing Read the proxy guide

v3 · In developmentStable 2.7 installation and reference

Route each model to its API

For example, use different models for chat and document summarization. Configure your apps with the same proxy URL and client key, then select a destination by model. You do not need to distribute upstream API keys to each app. Switch the model below to follow the request and response.

An illustration, not a live request. Run the same mock below. All keys shown are demonstration values.

Send from your app
Your chat or summarization app calls the proxy URL. Choose a different model without changing that URL or the client key.
POST http://localhost:8080/v1/responsesAuthorization: Bearer local-client{"model":"mock-model-a"}

Choose in nginx with Ruby

Check the client key, then look up the destination and its API key by model. Keep this mapping at the proxy, not in each app.

  routes = {
    "mock-model-a" => ["local_api_a", "Bearer local-upstream-a"],
    "mock-model-b" => ["local_api_b", "Bearer local-upstream-b"]
  }

Relay API A's response
nginx forwards with the API key and relays the response to your app. This mock reports the destination and credential replacement, not generated text.
local_api_aAuthorization: Bearer local-upstream-a
200 · Response received by your app
{ "upstream":"a", "model":"mock-model-a", "credential_replaced":true }
200 · Response received by your app
{ "upstream":"b", "model":"mock-model-b", "credential_replaced":true }

An unknown client key gets 401; an unmapped model gets 403 before reaching an API. There is no API-format conversion: your app must send the path and body expected by its upstream. Messages, Responses and concurrency limits are covered by separate integration tests.

Authentication, concurrency limits and streamed responses

Additional integration tests: concurrency and streamed relay

Ruby handleragent_proxy_route.rb

r = Nginx::Request.new
v = Nginx::Var.new
routes = {
  "mock-model-a" => ["agent_proxy_llm_a", "Bearer upstream-test-key-a"],
  "mock-model-b" => ["agent_proxy_llm_b", "Bearer upstream-test-key-b"]
}
route = routes[JSON.parse(r.body)["model"]]
if route
  v.agent_proxy_backend = route[0]
  v.agent_proxy_upstream_auth = route[1]
  Nginx.return Nginx::DECLINED
else
  Nginx.return Nginx::HTTP_FORBIDDEN
end

Show checked output (mock)

# What test/t/cases/agent_proxy.rb compares for routing by model: status,
# port of the upstream, Authorization and x-api-key at the upstream, and
# whether the upstream got the client's body unchanged.
model_a=200 12372 Bearer upstream-test-key-a none true
model_b=200 12373 Bearer upstream-test-key-b none true
unknown_model=403

Show nginx.conf

upstream agent_proxy_llm_a {
    server 127.0.0.1:12372;
}

upstream agent_proxy_llm_b {
    server 127.0.0.1:12373;
}

# Concurrent requests per client key: agent_proxy_auth.rb sets the key.
limit_conn_zone $agent_proxy_stream_key zone=agent_proxy_streams:1m;

server {
    listen 18132;

    # Declares the variables that the hooks assign; never requested.
    location = /.agent-proxy-vars {
        internal;
        set $agent_proxy_key_id "";
        set $agent_proxy_stream_key "";
        set $agent_proxy_backend "";
        set $agent_proxy_upstream_auth "";
        return 404;
    }

    # Most variables stay unset on most requests (a rejected key, a
    # request that no hook routed).
    uninitialized_variable_warn off;

    # The client key, checked for every request of this server before a
    # location is chosen. A request without a known key gets 401, and the
    # server-level error_page gives it a JSON body.
    mruby_server_rewrite_handler build/nginx/html/agent_proxy_auth.rb;
    error_page 401 @agent_proxy_401;

    # The upstream gets the credential that agent_proxy_route.rb picked and
    # never the client's.
    proxy_http_version 1.1;
    proxy_set_header Connection close;
    proxy_set_header Authorization $agent_proxy_upstream_auth;
    proxy_set_header x-api-key "";
    proxy_buffering off;

    # The Anthropic Messages API, and the OpenAI Responses API below with
    # the same recipes.
    location = /v1/messages {
        # One request at a time per client key; a second one gets 429 from
        # @agent_proxy_too_many. limit_conn_status is a status that no
        # upstream sends, so that an upstream's own 429 passes unchanged.
        limit_conn agent_proxy_streams 1;
        limit_conn_status 460;
        limit_conn_log_level notice;
        error_page 460 = @agent_proxy_too_many;

        # Picks the upstream block and the credential by the model of the
        # JSON body.
        mruby_enable_read_request_body on;
        mruby_access_handler build/nginx/html/agent_proxy_route.rb;

        # Logs $upstream_status after the response.
        mruby_log_handler build/nginx/html/agent_proxy_log.rb;

        proxy_pass http://$agent_proxy_backend/v1/messages;
    }

    # The zone of limit_conn is per client key, so a stream of a key on
    # either location holds the slot of that key on both.
    location = /v1/responses {
        limit_conn agent_proxy_streams 1;
        limit_conn_status 460;
        limit_conn_log_level notice;
        error_page 460 = @agent_proxy_too_many;

        mruby_enable_read_request_body on;
        mruby_access_handler build/nginx/html/agent_proxy_route.rb;
        mruby_log_handler build/nginx/html/agent_proxy_log.rb;

        proxy_pass http://$agent_proxy_backend/v1/responses;
    }

    # The proxy's own error bodies, in the error format of the API that the
    # client asked for: a named location keeps the request's $uri.
    location @agent_proxy_401 {
        default_type application/json;
        if ($uri = /v1/responses) {
            return 401 '{"error":{"message":"invalid API key","type":"invalid_request_error","param":null,"code":null}}';
        }
        return 401 '{"type":"error","error":{"type":"authentication_error","message":"invalid API key"}}';
    }

    location @agent_proxy_too_many {
        default_type application/json;
        add_header retry-after 1 always;
        if ($uri = /v1/responses) {
            return 429 '{"error":{"message":"too many concurrent requests","type":"rate_limit_error","param":null,"code":null}}';
        }
        return 429 '{"type":"error","error":{"type":"rate_limit_error","message":"too many concurrent requests"}}';
    }
}

On every CI run, the case test/t/cases/agent_proxy.rb sends the requests of its client and compares the values that the client prints with the values shown on this page. The four files last changed on 2026-10-29 in commit 300e251c.

Try it locally without a provider key

With Docker Compose, git and curl, run the same nginx proxy and two mock APIs shown above. No provider account is needed. The first run downloads and builds from source; the mocks do not generate text.

git clone --branch next https://github.com/matsumotory/ngx_mruby.git
cd ngx_mruby/examples/start
docker compose up --build --wait

Run this in the example directory to receive the result from API A. Then edit the mapping in route.rb and see the same application request reach a different destination.

curl -K client.conf -d '{"model": "mock-model-a"}'

Run and customize the example

Connect your own application

Your app sends HTTP requests to the proxy. Configure upstreams and HTTP transport in nginx.conf, and client-key checks and model mappings in route.rb. Ruby passes the selected destination and credential through nginx variables; proxy_pass and proxy_set_header handle the relay.

For a real API, configure and verify its endpoint, authentication and transport requirements, including TLS. The local example does not establish real-provider compatibility. A public deployment also needs its own credentials and operational configuration.

Read the implemented recipe and its limits