Agent proxy

Development version. This page describes ngx_mruby 3.x on the branch next, which is not released: the current release is 2.7. The recipes below run on next today, and CI checks them. The diagrams show the agent proxy that the v3 plan builds for v3.0.0, and say for each step whether it runs today or which step of the plan adds it.

An agent proxy is nginx in front of the API of an LLM provider, between coding agents and the provider. Ruby decides per request: whether the client key is known, which upstream and which provider credential the request gets, and the key that limit_conn counts. nginx moves the bytes: it reads the request, relays the response of the provider as it arrives, and enforces the limits of its own modules. The proxy passes the Anthropic Messages API (/v1/messages) and the OpenAI Responses API (/v1/responses) through, with their streamed responses (server-sent events); it does not translate between them.

Implementation status and the v3 target

The completed v3 design is the destination described by the diagrams below. Use this status vocabulary when implementing from this page. implemented means present on next, not released in 2.7. planned means the agreed plan names the work, not that an API or executable example is available. under-design is reserved for unresolved details; no proposal becomes an implemented contract merely because it appears in documentation.

Status	Capability	Evidence or next step
implemented	Client-key checks, per-key concurrent-request limit, model routing and credential replacement	The recipes and `test/t/cases/agent_proxy.rb` below
implemented	Byte-for-byte streamed Messages/Responses relay and upstream-error passthrough	The same integration case and mock upstream
planned	Shared usage accounting, event framing/filtering and first-byte deadline	Plan steps 7a, 7b and 7c; diagram labels below
planned	Provider-specific Host/SNI, fallback policy and reference-proxy transport boundaries	Plan step 7e; request and endpoints diagrams below

The stable 2.7 documentation describes current behavior. Planned v3 APIs must not be copied into a 2.7 configuration. A future API example is not runnable evidence until its implementation and corresponding tests land.

Recipes that run today

The configuration below serves both APIs with the same recipes. In front of a mock of the two APIs, which the test client starts on ports 12372 and 12373 (test/soak/mock_llm.rb):

agent_proxy_auth.rb, a server rewrite handler, looks up the client key in Authorization (Bearer) or x-api-key. An unknown key gets 401, and error_page 401 @agent_proxy_401 gives it a JSON body in the error format of the API; the request does not reach the upstream.
limit_conn per client key allows one request at a time. A second request of the key while the first streams gets 429 with a JSON body and retry-after, through the status 460, which no upstream sends, so that an upstream’s own 429 passes unchanged.
agent_proxy_route.rb, an access handler, reads the model of the JSON body, chooses the upstream block, and sets the credential that the upstream gets in place of the client’s. A model without an upstream gets 403.
proxy_pass relays the response as it arrives (proxy_buffering off). An error of the upstream, such as 529, reaches the client with its body.
agent_proxy_log.rb, a log handler, writes one line per request with the key id and $upstream_status.

Development version, New in 3.0

upstream agent_proxy_llm_a {
    server 127.0.0.1:12372;
}

upstream agent_proxy_llm_b {
    server 127.0.0.1:12373;
}

# Concurrent requests per client key: agent_proxy_auth.rb sets the key.
limit_conn_zone $agent_proxy_stream_key zone=agent_proxy_streams:1m;

server {
    listen 18132;

    # Declares the variables that the hooks assign; never requested.
    location = /.agent-proxy-vars {
        internal;
        set $agent_proxy_key_id "";
        set $agent_proxy_stream_key "";
        set $agent_proxy_backend "";
        set $agent_proxy_upstream_auth "";
        return 404;
    }

    # Most variables stay unset on most requests (a rejected key, a
    # request that no hook routed).
    uninitialized_variable_warn off;

    # The client key, checked for every request of this server before a
    # location is chosen. A request without a known key gets 401, and the
    # server-level error_page gives it a JSON body.
    mruby_server_rewrite_handler build/nginx/html/agent_proxy_auth.rb;
    error_page 401 @agent_proxy_401;

    # The upstream gets the credential that agent_proxy_route.rb picked and
    # never the client's.
    proxy_http_version 1.1;
    proxy_set_header Connection close;
    proxy_set_header Authorization $agent_proxy_upstream_auth;
    proxy_set_header x-api-key "";
    proxy_buffering off;

    # The Anthropic Messages API, and the OpenAI Responses API below with
    # the same recipes.
    location = /v1/messages {
        # One request at a time per client key; a second one gets 429 from
        # @agent_proxy_too_many. limit_conn_status is a status that no
        # upstream sends, so that an upstream's own 429 passes unchanged.
        limit_conn agent_proxy_streams 1;
        limit_conn_status 460;
        limit_conn_log_level notice;
        error_page 460 = @agent_proxy_too_many;

        # Picks the upstream block and the credential by the model of the
        # JSON body.
        mruby_enable_read_request_body on;
        mruby_access_handler build/nginx/html/agent_proxy_route.rb;

        # Logs $upstream_status after the response.
        mruby_log_handler build/nginx/html/agent_proxy_log.rb;

        proxy_pass http://$agent_proxy_backend/v1/messages;
    }

    # The zone of limit_conn is per client key, so a stream of a key on
    # either location holds the slot of that key on both.
    location = /v1/responses {
        limit_conn agent_proxy_streams 1;
        limit_conn_status 460;
        limit_conn_log_level notice;
        error_page 460 = @agent_proxy_too_many;

        mruby_enable_read_request_body on;
        mruby_access_handler build/nginx/html/agent_proxy_route.rb;
        mruby_log_handler build/nginx/html/agent_proxy_log.rb;

        proxy_pass http://$agent_proxy_backend/v1/responses;
    }

    # The proxy's own error bodies, in the error format of the API that the
    # client asked for: a named location keeps the request's $uri.
    location @agent_proxy_401 {
        default_type application/json;
        if ($uri = /v1/responses) {
            return 401 '{"error":{"message":"invalid API key","type":"invalid_request_error","param":null,"code":null}}';
        }
        return 401 '{"type":"error","error":{"type":"authentication_error","message":"invalid API key"}}';
    }

    location @agent_proxy_too_many {
        default_type application/json;
        add_header retry-after 1 always;
        if ($uri = /v1/responses) {
            return 429 '{"error":{"message":"too many concurrent requests","type":"rate_limit_error","param":null,"code":null}}';
        }
        return 429 '{"type":"error","error":{"type":"rate_limit_error","message":"too many concurrent requests"}}';
    }
}

r = Nginx::Request.new
v = Nginx::Var.new
keys = { "agent-test-key-1" => "key-1", "agent-test-key-2" => "key-2" }
auth = r.headers_in["Authorization"].to_s
id = keys[auth.start_with?("Bearer ") ? auth[7..-1] : r.headers_in["x-api-key"].to_s]
if id
  v.agent_proxy_key_id = id
  v.agent_proxy_stream_key = id
  Nginx.return Nginx::DECLINED
else
  Nginx.return Nginx::HTTP_UNAUTHORIZED
end

r = Nginx::Request.new
v = Nginx::Var.new
routes = {
  "mock-model-a" => ["agent_proxy_llm_a", "Bearer upstream-test-key-a"],
  "mock-model-b" => ["agent_proxy_llm_b", "Bearer upstream-test-key-b"]
}
route = routes[JSON.parse(r.body)["model"]]
if route
  v.agent_proxy_backend = route[0]
  v.agent_proxy_upstream_auth = route[1]
  Nginx.return Nginx::DECLINED
else
  Nginx.return Nginx::HTTP_FORBIDDEN
end

r = Nginx::Request.new
v = Nginx::Var.new
Nginx.errlogger Nginx::LOG_NOTICE,
                "agent_proxy_log test_id=#{r.headers_in["x-test-id"]} key=#{v.agent_proxy_key_id} " \
                "upstream_status=#{v.upstream_status} status=#{v.status}"

# What test/t/cases/agent_proxy.rb compares for the client key, one value
# per line: status, Content-Type and body of the response; mock_requests
# counts the requests that reached the mock upstream.
wrong_key=401 application/json {"type":"error","error":{"type":"authentication_error","message":"invalid API key"}}
no_key=401 application/json {"type":"error","error":{"type":"authentication_error","message":"invalid API key"}}
mock_requests_after_rejects=0
api_key=200 12372 true
mock_requests=1

# What test/t/cases/agent_proxy.rb compares for routing by model: status,
# port of the upstream, Authorization and x-api-key at the upstream, and
# whether the upstream got the client's body unchanged.
model_a=200 12372 Bearer upstream-test-key-a none true
model_b=200 12373 Bearer upstream-test-key-b none true
unknown_model=403

# What test/t/cases/agent_proxy.rb compares for limit_conn per client key:
# held is the first stream of key-1 (status, Content-Type, how the client
# stopped reading, events read); same_key is a second request of key-1
# while it is open (status, retry-after, Content-Type, body); other_key and
# after_close are requests of key-2 and of key-1 after the stream (status,
# port of the upstream); limit_log_lines counts the lines of limit_conn in
# error.log.
held=200 text/event-stream stopped 3
same_key=429 1 application/json {"type":"error","error":{"type":"rate_limit_error","message":"too many concurrent requests"}}
other_key=200 12372
limit_log_lines=1
after_close=200 12372

# What test/t/cases/agent_proxy.rb compares for an upstream error and the
# log handler: ok and overloaded are the status (with Content-Type and body)
# of a request that the mock answers with 200 and with 529; a key with ~=
# is a text that the line of the log handler in error.log contains.
ok=200
overloaded=529 application/json {"type":"error","error":{"type":"overloaded_error","message":"mock: status 529"}}
ok_log~=key=key-1 upstream_status=200 status=200
overloaded_log~=key=key-1 upstream_status=529 status=529

On every CI run, the case test/t/cases/agent_proxy.rb sends the requests of its client and compares the values that the client prints with the values shown on this page. The nine files last changed on 2026-10-29 in commit 300e251c.

The client of the test (test/t/cases/_agent_proxy_client.rb) sends the requests of each recipe to /v1/messages and prints one value per line; the files above are the values that the case compares for the Messages API. The case compares the same recipes for /v1/responses, and that a stream of 50 deltas of each API arrives byte for byte.

The request in v3.0

Development version, New in 3.0

Today: on next, checked by agent_proxy.rb

Step 4: the next step of the v3 plan

Steps 7a to 7e: later steps of the v3 plan

Today: the server rewrite handler checks the client key; an unknown key gets 401 with a JSON body

Today: limit_conn per client key; a second stream of the key gets 429

Today: the access handler routes by the model of the body and swaps the credential; 403 for an unknown model

Steps 7a and 7b: the token and money budgets of the key in a shared dictionary; body_json reads the model in C

Today: proxy_pass relays the request to the chosen upstream with the provider's credential

Today: an upstream 529 reaches the client with its body

Step 7e: the reference proxy sets Host and SNI per provider, and sends a 529 to another provider with error_page 529 = @name

The steps of a request through the agent proxy of v3.0, each with whether it runs today or which step of the v3 plan adds it.Drawn from these lines of ngx_mruby at 8d1500bf0 (2026-10-29):

docs/proposals/v3-plan.md lines 407 to 447 and 558 to 576
test/conf/conf.d/60-agent-proxy.conf lines 14 to 111

The streamed response in v3.0

Development version, New in 3.0

Today: on next, checked by agent_proxy.rb

Step 4: the next step of the v3 plan

Steps 7a to 7e: later steps of the v3 plan

Step 7c: a first-byte deadline gives up a provider that sends no header; proxy_read_timeout stays the idle timeout

Today: nginx relays the events of the provider byte for byte as they arrive

Step 7b: the event filter frames the events in C and calls Ruby once per listed event

Step 7b: Ruby reads the usage from message_start and message_delta, or from the terminal response.* event

Step 7a: the counters of the shared dictionary are charged when the response ends or is cut

Step 4: the log handlers run before access_log, which then logs the cost and the tokens

How the agent proxy of v3.0 relays a streamed response and charges its usage, each step with whether it runs today or which step of the v3 plan adds it.Drawn from these lines of ngx_mruby at 8d1500bf0 (2026-10-29):

docs/proposals/v3-plan.md lines 407 to 447 and 558 to 576
test/conf/conf.d/60-agent-proxy.conf lines 44 to 55

The endpoints in v3.0

Development version, New in 3.0

Today: on next, checked by agent_proxy.rb

Step 4: the next step of the v3 plan

Steps 7a to 7e: later steps of the v3 plan

Today: the proxy answers 401 and 429 itself with JSON bodies, and 403 with nginx's error page

Steps 7c and 7e: GET /v1/models from the client's model allowlist, and 504 after the first-byte deadline

Today: POST /v1/messages and POST /v1/responses with their streams pass through unchanged

Today: an error of the upstream reaches the client with its body

Step 7e: the reference proxy refuses Upgrade, the WebSocket transport of the Responses API

What the agent proxy of v3.0 answers itself, what it passes through and what it refuses, as a list of the kinds of request, each with whether it runs today or which step of the v3 plan adds it.Drawn from these lines of ngx_mruby at 8d1500bf0 (2026-10-29):

docs/proposals/v3-plan.md lines 424 to 447
test/conf/conf.d/60-agent-proxy.conf lines 93 to 110

The plan of the agent proxy, its protocol scope and the order of its steps are in Agent proxy: the v3.0 use case of the v3 plan. A sample of a planned feature appears on this page when the feature and its test merge.

Cost per request

The perf lane counts the instructions that nginx runs per request with callgrind (test/perf/compare.sh). Its handlers agent_auth.rb and agent_route.rb of test/soak/handlers/ look up the client key and route by the model of the body as the recipes above do, in the perf lane’s own configuration. On 2026-10-14, with PERF_N=20000 and nginx 1.31.6, in an Ubuntu 22.04 container on aarch64, the tree of next at bc2488a with mruby 4.0.0 (#561) measured:

What	Instructions per request (without GC)
The authentication check of the server rewrite handler, over proxying a 2 KB POST without Ruby	25,276 (23,305)
Routing by `model` with `JSON.parse` of a 2 KB body, over the same proxying	85,228 (82,097)

Agent proxy scenarios in the comparison describes the scenarios.