Authenticate and route AI applications with Ruby.

ngx_mruby extends nginx with Ruby. Between an AI application and a model API, it can form an AI gateway whose Ruby rules check client keys and select model routes and upstream credentials.

Here an AI application is any software that calls a model API or an MCP server, from chat front ends to agents and MCP hosts.

The v3 development example below lets you run model-based routing locally. Later use cases describe proposed extensions, including an MCP gateway, with their design status.

See the use cases

v3 · In development

Stable 2.7: install and reference

Route each model to its API

For example, use different models for chat and document summarization. Configure your apps with the same proxy URL and client key, then select a destination by model. You do not need to distribute upstream API keys to each app. Switch the model below to follow the request and response.

An illustration, not a live request. Run the same mock below. All keys shown are demonstration values.

Send from your app
Your chat or summarization app calls the proxy URL. Choose a different model without changing that URL or the client key.
POST http://localhost:8080/v1/responsesAuthorization: Bearer local-client{"model":"mock-model-a"}

Choose in nginx with Ruby

Check the client key, then look up the destination and its API key by model. Keep this mapping at the proxy, not in each app.

  routes = {
    "mock-model-a" => ["local_api_a", "Bearer local-upstream-a"],
    "mock-model-b" => ["local_api_b", "Bearer local-upstream-b"]
  }

Relay API A's response
nginx forwards with the API key and relays the response to your app. This mock reports the destination and credential replacement, not generated text.
local_api_aAuthorization: Bearer local-upstream-a
200 · Response received by your app
{ "upstream":"a", "model":"mock-model-a", "credential_replaced":true }
200 · Response received by your app
{ "upstream":"b", "model":"mock-model-b", "credential_replaced":true }

An unknown client key gets 401; an unmapped model gets 403 before reaching an API. There is no API-format conversion: your app must send the path and body expected by its upstream. Messages, Responses and concurrency limits are covered by separate integration tests.

Authentication, concurrency limits and streamed responses

Additional integration tests: concurrency and streamed relay

Ruby handleragent_proxy_route.rb

r = Nginx::Request.new
v = Nginx::Var.new
routes = {
  "mock-model-a" => ["agent_proxy_llm_a", "Bearer upstream-test-key-a"],
  "mock-model-b" => ["agent_proxy_llm_b", "Bearer upstream-test-key-b"]
}
route = routes[JSON.parse(r.body)["model"]]
if route
  v.agent_proxy_backend = route[0]
  v.agent_proxy_upstream_auth = route[1]
  Nginx.return Nginx::DECLINED
else
  Nginx.return Nginx::HTTP_FORBIDDEN
end

Show checked output (mock)

# What test/t/cases/agent_proxy.rb compares for routing by model: status,
# port of the upstream, Authorization and x-api-key at the upstream, and
# whether the upstream got the client's body unchanged.
model_a=200 12372 Bearer upstream-test-key-a none true
model_b=200 12373 Bearer upstream-test-key-b none true
unknown_model=403

Show nginx.conf

upstream agent_proxy_llm_a {
    server 127.0.0.1:12372;
}

upstream agent_proxy_llm_b {
    server 127.0.0.1:12373;
}

# Concurrent requests per client key: agent_proxy_auth.rb sets the key.
limit_conn_zone $agent_proxy_stream_key zone=agent_proxy_streams:1m;

server {
    listen 18132;

    # Declares the variables that the hooks assign; never requested.
    location = /.agent-proxy-vars {
        internal;
        set $agent_proxy_key_id "";
        set $agent_proxy_stream_key "";
        set $agent_proxy_backend "";
        set $agent_proxy_upstream_auth "";
        return 404;
    }

    # Most variables stay unset on most requests (a rejected key, a
    # request that no hook routed).
    uninitialized_variable_warn off;

    # The client key, checked for every request of this server before a
    # location is chosen. A request without a known key gets 401, and the
    # server-level error_page gives it a JSON body.
    mruby_server_rewrite_handler build/nginx/html/agent_proxy_auth.rb;
    error_page 401 @agent_proxy_401;

    # The upstream gets the credential that agent_proxy_route.rb picked and
    # never the client's.
    proxy_http_version 1.1;
    proxy_set_header Connection close;
    proxy_set_header Authorization $agent_proxy_upstream_auth;
    proxy_set_header x-api-key "";
    proxy_buffering off;

    # The Anthropic Messages API, and the OpenAI Responses API below with
    # the same recipes.
    location = /v1/messages {
        # One request at a time per client key; a second one gets 429 from
        # @agent_proxy_too_many. limit_conn_status is a status that no
        # upstream sends, so that an upstream's own 429 passes unchanged.
        limit_conn agent_proxy_streams 1;
        limit_conn_status 460;
        limit_conn_log_level notice;
        error_page 460 = @agent_proxy_too_many;

        # Picks the upstream block and the credential by the model of the
        # JSON body.
        mruby_enable_read_request_body on;
        mruby_access_handler build/nginx/html/agent_proxy_route.rb;

        # Logs $upstream_status after the response.
        mruby_log_handler build/nginx/html/agent_proxy_log.rb;

        proxy_pass http://$agent_proxy_backend/v1/messages;
    }

    # The zone of limit_conn is per client key, so a stream of a key on
    # either location holds the slot of that key on both.
    location = /v1/responses {
        limit_conn agent_proxy_streams 1;
        limit_conn_status 460;
        limit_conn_log_level notice;
        error_page 460 = @agent_proxy_too_many;

        mruby_enable_read_request_body on;
        mruby_access_handler build/nginx/html/agent_proxy_route.rb;
        mruby_log_handler build/nginx/html/agent_proxy_log.rb;

        proxy_pass http://$agent_proxy_backend/v1/responses;
    }

    # The proxy's own error bodies, in the error format of the API that the
    # client asked for: a named location keeps the request's $uri.
    location @agent_proxy_401 {
        default_type application/json;
        if ($uri = /v1/responses) {
            return 401 '{"error":{"message":"invalid API key","type":"invalid_request_error","param":null,"code":null}}';
        }
        return 401 '{"type":"error","error":{"type":"authentication_error","message":"invalid API key"}}';
    }

    location @agent_proxy_too_many {
        default_type application/json;
        add_header retry-after 1 always;
        if ($uri = /v1/responses) {
            return 429 '{"error":{"message":"too many concurrent requests","type":"rate_limit_error","param":null,"code":null}}';
        }
        return 429 '{"type":"error","error":{"type":"rate_limit_error","message":"too many concurrent requests"}}';
    }
}

On every CI run, the case test/t/cases/agent_proxy.rb sends the requests of its client and compares the values that the client prints with the values shown on this page. The four files last changed on 2026-10-29 in commit 300e251c.

Try it locally without a provider key

With Docker Compose, git and curl, run the same nginx proxy and two mock APIs shown above. No provider account is needed. The first run downloads and builds from source; the mocks do not generate text.

git clone --branch next https://github.com/matsumotory/ngx_mruby.git
cd ngx_mruby/examples/start
docker compose up --build --wait

Run this in the example directory to receive the result from API A. Then edit the mapping in route.rb and see the same application request reach a different destination.

curl -K client.conf -d '{"model": "mock-model-a"}'

Run and customize the example

Connect your own application

Your app sends HTTP requests to the proxy, the part of the AI gateway that relays model API calls. Configure upstreams and HTTP transport in nginx.conf, and client-key checks and model mappings in route.rb. Ruby passes the selected destination and credential through nginx variables; proxy_pass and proxy_set_header handle the relay.

For a real API, configure and verify its endpoint, authentication and transport requirements, including TLS. The local example does not establish real-provider compatibility. A public deployment also needs its own credentials and operational configuration.

Read the implemented recipe and its limits

Write HTTP and TCP policies in Ruby

Choose a backend from a request header

Select an upstream from a request header and your routing table.

HTTP configuration and example

Check the client before relaying TCP

Check the client address against an allowlist before opening the upstream connection.

TCP configuration and example

Future use cases and their design status

The model routing example above can be run today. The following designs describe uses of nginx and Ruby for MCP calls from AI applications, model API fallback, and requests arriving at a site from AI crawlers and agents. Here an agent is an AI application whose model chooses the next search or read. In front of MCP servers and model APIs, an AI gateway would authorize and route the calls; in front of a site, the same nginx and Ruby would limit the rate and load of incoming requests. Service processing and agent decisions remain with their respective implementations. “Designing” means a first recipe and acceptance criteria are documented; “Direction” means the direction is documented. No test runs a complete use case yet, in either status.

Check permissions for MCP tool calls

Designing

An MCP service operator could restrict the MCP tools and argument ranges available to each user. The design places an MCP gateway between the AI application and the MCP server, where Ruby would compare tools/call with the authenticated user’s permissions. Only allowed calls would be forwarded; the MCP server would still enforce access to individual data objects.

Needs in v3: the JSON body read (body_json), the execution-context type, and a Ruby-built body with any status.

Read the design and required features

Limit bursts of MCP calls

Direction

To bound load when AI applications repeatedly call MCP tools or open many streams, Ruby would identify the authenticated user and call class, and nginx limit_req and limit_conn would apply configured limits. Legitimate rereads would not be treated as loops; rejection responses would describe the conditions for retry.

Needs in v3: the shared dictionary, which counts across workers, and mruby_variable.

Read the design and required features

Fit the MCP tools to each AI application

Designing

The same MCP service could support AI applications that require different MCP tool interfaces. The design would let an MCP gateway select a versioned backend MCP server by connection profile, with consistent tools/list and tools/call behavior. The aim is to adapt tool descriptions and calls without duplicating search and retrieval logic for each application.

Needs in v3: the shared dictionary for the cache of tool lists.

Read the design and required features

Retrieve the needed part of a document

Designing

An agent could read the relevant passage of a long document and request more when needed instead of fetching everything at once. The MCP gateway would select a retrieval profile and processing budget; the backend would own document positions, versions and continuation information. Retrieval would use a range-aware MCP tool, not arbitrary truncation of JSON or a stream.

Needs in v3: the shared dictionary for budgets that all workers share.

Read the design and required features

Return next steps when a search finds nothing

Direction

When an MCP search returns no result or retrieval fails, the agent should receive the reason and available next steps. The MCP gateway would preserve the distinction between no match, denied access and an unavailable source; the MCP service would supply domain-specific guidance. The MCP gateway would not change query meaning or promote instructions in retrieved documents into execution instructions. The agent would still decide whether to search again or change the query.

Needs in v3: a Ruby-built body with any status.

Read the design and required features

Observe sequences of MCP tool calls

Direction

An MCP service operator could examine the sequence and outcomes of search, document retrieval and repeated search performed by an agent. A correlation ID would connect MCP tool names, outcomes and durations. Queries and document bodies would not be collected by default, and retention and collection volume would be bounded. The agent would still decide whether to continue its investigation.

Needs in v3: log handlers that run before access_log, mruby_variable, worker timers and the non-blocking socket API.

Read the design and required features

Select another model API endpoint after an error

Designing

Instead of implementing model API fallback separately in each application, an operator could manage it at the AI gateway. After an upstream 429 or another configured retryable error, Ruby would select a compatible endpoint and credentials, and nginx would retry. Attempts and time would be bounded, and only requests safe to replay would be retried, so that processing and charges are not duplicated. Once the response starts, it would not be switched to another generated stream.

Needs in v3: the shared dictionary for cooldowns, the upstream peer hook, the first-byte deadline and the Ruby balancer API.

Read the design and required features

Limit the request rate of AI crawlers and agents

Direction

A website or API operator could select Ruby policies for requests from AI crawlers and agents, distinguishing collection by a crawler from access on a user’s behalf. Authentication state and the requested resource would inform rate and concurrency limits. A User-Agent claim alone would not prove identity; the design also needs independent checks, such as request-signature verification, and a policy for unidentified clients.

Needs in v3: the shared dictionary that all workers share, and the non-blocking socket API for an identity lookup in an external service.

Read the design and required features

Consult an external semantic cache

Direction

An external semantic cache could reuse a stored answer for a similar model API request without changing the application. The AI gateway would supply constraints such as the user’s access scope and model; the external service would assess similarity and reuse validity. A miss, uncertain match or timeout would fall through to the original model API endpoint. The mechanism for storing responses in the cache is not yet designed.

Needs in v3: the JSON body read, sub_request with a method, a body and headers, and the non-blocking socket API.

Read the design and required features

Export metadata recorded by the log handler

Direction

A service operator could inspect status, response time and routing decisions in an external collector. The current log handler cannot wait for a network response, so export is planned through a bounded worker queue, timers and nonblocking transport. Collector delays would not stall responses, and the behavior on a full queue or a failed delivery would be defined.

Needs in v3: log handlers that run before access_log, mruby_variable, the SSE event filter, worker timers and the non-blocking socket API.

Read the design and required features

Documentation for your implementation

Guides, API reference and operation MCP use-case design and unverified scope v3 implementation status and plan Markdown index for agents

Authenticate and route AI applications with Ruby.

Route each model to its API

Send from your app

Choose in nginx with Ruby

Relay API A's response

Try it locally without a provider key

Connect your own application

Write HTTP and TCP policies in Ruby

Choose a backend from a request header

Check the client before relaying TCP

Future use cases and their design status

Check permissions for MCP tool calls

Limit bursts of MCP calls

Fit the MCP tools to each AI application

Retrieve the needed part of a document

Return next steps when a search finds nothing

Observe sequences of MCP tool calls

Select another model API endpoint after an error

Limit the request rate of AI crawlers and agents

Consult an external semantic cache

Export metadata recorded by the log handler

Documentation for your implementation