--- title: "Agent proxy" language: "en" documentation_version: "3.x" channel: "development" release_context: null source_revision: "252018d861d008b380109414280f99e51433e238" working_tree_clean: true source_path: "docs/guides/agent-proxy.md" source_sha256: "326f4450adce05546783568a6acb0eaa8c75ce18d352cd1b18e3359dbb13488a" source_at_revision: true source_url: "https://github.com/matsumotory/ngx_mruby/blob/252018d861d008b380109414280f99e51433e238/docs/guides/agent-proxy.md" html_url: "https://ngx.mruby.org/guides/agent-proxy/" markdown_url: "https://ngx.mruby.org/markdown/guides/agent-proxy.md" last_updated: "2026-10-30" --- # Agent proxy **Development version.** This page describes ngx_mruby 3.x on the branch `next`, which is not released: the current release is 2.7. The recipes below run on `next` today, and CI checks them. The diagrams show the agent proxy that the v3 plan builds for v3.0.0, and say for each step whether it runs today or which step of [the plan](https://github.com/matsumotory/ngx_mruby/blob/next/docs/proposals/v3-plan.md#agent-proxy-the-v30-use-case) adds it. An agent proxy is nginx in front of the API of an LLM provider, between coding agents and the provider. Ruby decides per request: whether the client key is known, which upstream and which provider credential the request gets, and the key that `limit_conn` counts. nginx moves the bytes: it reads the request, relays the response of the provider as it arrives, and enforces the limits of its own modules. The proxy passes the Anthropic Messages API (`/v1/messages`) and the OpenAI Responses API (`/v1/responses`) through, with their streamed responses (server-sent events); it does not translate between them. ## Implementation status and the v3 target The completed v3 design is the destination described by the diagrams below. Use this status vocabulary when implementing from this page. `implemented` means present on `next`, not released in 2.7. `planned` means the agreed plan names the work, not that an API or executable example is available. `under-design` is reserved for unresolved details; no proposal becomes an implemented contract merely because it appears in documentation. | Status | Capability | Evidence or next step | |---|---|---| | implemented | Client-key checks, per-key concurrent-request limit, model routing and credential replacement | The recipes and `test/t/cases/agent_proxy.rb` below | | implemented | Byte-for-byte streamed Messages/Responses relay and upstream-error passthrough | The same integration case and mock upstream | | planned | Shared usage accounting, event framing/filtering and first-byte deadline | Plan steps 7a, 7b and 7c; diagram labels below | | planned | Provider-specific Host/SNI, fallback policy and reference-proxy transport boundaries | Plan step 7e; request and endpoints diagrams below | The stable 2.7 documentation describes current behavior. Planned v3 APIs must not be copied into a 2.7 configuration. A future API example is not runnable evidence until its implementation and corresponding tests land. ## Recipes that run today The configuration below serves both APIs with the same recipes. In front of a mock of the two APIs, which the test client starts on ports 12372 and 12373 (`test/soak/mock_llm.rb`): - `agent_proxy_auth.rb`, a server rewrite handler, looks up the client key in `Authorization` (`Bearer`) or `x-api-key`. An unknown key gets 401, and `error_page 401 @agent_proxy_401` gives it a JSON body in the error format of the API; the request does not reach the upstream. - `limit_conn` per client key allows one request at a time. A second request of the key while the first streams gets 429 with a JSON body and `retry-after`, through the status 460, which no upstream sends, so that an upstream's own 429 passes unchanged. - `agent_proxy_route.rb`, an access handler, reads the `model` of the JSON body, chooses the upstream block, and sets the credential that the upstream gets in place of the client's. A model without an upstream gets 403. - `proxy_pass` relays the response as it arrives (`proxy_buffering off`). An error of the upstream, such as 529, reaches the client with its body. - `agent_proxy_log.rb`, a log handler, writes one line per request with the key id and `$upstream_status`.

Development version, New in 3.0

```nginx title="test/conf/conf.d/60-agent-proxy.conf" upstream agent_proxy_llm_a { server 127.0.0.1:12372; } upstream agent_proxy_llm_b { server 127.0.0.1:12373; } # Concurrent requests per client key: agent_proxy_auth.rb sets the key. limit_conn_zone $agent_proxy_stream_key zone=agent_proxy_streams:1m; server { listen 18132; # Declares the variables that the hooks assign; never requested. location = /.agent-proxy-vars { internal; set $agent_proxy_key_id ""; set $agent_proxy_stream_key ""; set $agent_proxy_backend ""; set $agent_proxy_upstream_auth ""; return 404; } # Most variables stay unset on most requests (a rejected key, a # request that no hook routed). uninitialized_variable_warn off; # The client key, checked for every request of this server before a # location is chosen. A request without a known key gets 401, and the # server-level error_page gives it a JSON body. mruby_server_rewrite_handler build/nginx/html/agent_proxy_auth.rb; error_page 401 @agent_proxy_401; # The upstream gets the credential that agent_proxy_route.rb picked and # never the client's. proxy_http_version 1.1; proxy_set_header Connection close; proxy_set_header Authorization $agent_proxy_upstream_auth; proxy_set_header x-api-key ""; proxy_buffering off; # The Anthropic Messages API, and the OpenAI Responses API below with # the same recipes. location = /v1/messages { # One request at a time per client key; a second one gets 429 from # @agent_proxy_too_many. limit_conn_status is a status that no # upstream sends, so that an upstream's own 429 passes unchanged. limit_conn agent_proxy_streams 1; limit_conn_status 460; limit_conn_log_level notice; error_page 460 = @agent_proxy_too_many; # Picks the upstream block and the credential by the model of the # JSON body. mruby_enable_read_request_body on; mruby_access_handler build/nginx/html/agent_proxy_route.rb; # Logs $upstream_status after the response. mruby_log_handler build/nginx/html/agent_proxy_log.rb; proxy_pass http://$agent_proxy_backend/v1/messages; } # The zone of limit_conn is per client key, so a stream of a key on # either location holds the slot of that key on both. location = /v1/responses { limit_conn agent_proxy_streams 1; limit_conn_status 460; limit_conn_log_level notice; error_page 460 = @agent_proxy_too_many; mruby_enable_read_request_body on; mruby_access_handler build/nginx/html/agent_proxy_route.rb; mruby_log_handler build/nginx/html/agent_proxy_log.rb; proxy_pass http://$agent_proxy_backend/v1/responses; } # The proxy's own error bodies, in the error format of the API that the # client asked for: a named location keeps the request's $uri. location @agent_proxy_401 { default_type application/json; if ($uri = /v1/responses) { return 401 '{"error":{"message":"invalid API key","type":"invalid_request_error","param":null,"code":null}}'; } return 401 '{"type":"error","error":{"type":"authentication_error","message":"invalid API key"}}'; } location @agent_proxy_too_many { default_type application/json; add_header retry-after 1 always; if ($uri = /v1/responses) { return 429 '{"error":{"message":"too many concurrent requests","type":"rate_limit_error","param":null,"code":null}}'; } return 429 '{"type":"error","error":{"type":"rate_limit_error","message":"too many concurrent requests"}}'; } } ``` ```ruby title="test/html/agent_proxy_auth.rb" r = Nginx::Request.new v = Nginx::Var.new keys = { "agent-test-key-1" => "key-1", "agent-test-key-2" => "key-2" } auth = r.headers_in["Authorization"].to_s id = keys[auth.start_with?("Bearer ") ? auth[7..-1] : r.headers_in["x-api-key"].to_s] if id v.agent_proxy_key_id = id v.agent_proxy_stream_key = id Nginx.return Nginx::DECLINED else Nginx.return Nginx::HTTP_UNAUTHORIZED end ``` ```ruby title="test/html/agent_proxy_route.rb" r = Nginx::Request.new v = Nginx::Var.new routes = { "mock-model-a" => ["agent_proxy_llm_a", "Bearer upstream-test-key-a"], "mock-model-b" => ["agent_proxy_llm_b", "Bearer upstream-test-key-b"] } route = routes[JSON.parse(r.body)["model"]] if route v.agent_proxy_backend = route[0] v.agent_proxy_upstream_auth = route[1] Nginx.return Nginx::DECLINED else Nginx.return Nginx::HTTP_FORBIDDEN end ``` ```ruby title="test/html/agent_proxy_log.rb" r = Nginx::Request.new v = Nginx::Var.new Nginx.errlogger Nginx::LOG_NOTICE, "agent_proxy_log test_id=#{r.headers_in["x-test-id"]} key=#{v.agent_proxy_key_id} " \ "upstream_status=#{v.upstream_status} status=#{v.status}" ``` ```text title="test/html/agent_proxy_auth.expected: the values that the case compares" # What test/t/cases/agent_proxy.rb compares for the client key, one value # per line: status, Content-Type and body of the response; mock_requests # counts the requests that reached the mock upstream. wrong_key=401 application/json {"type":"error","error":{"type":"authentication_error","message":"invalid API key"}} no_key=401 application/json {"type":"error","error":{"type":"authentication_error","message":"invalid API key"}} mock_requests_after_rejects=0 api_key=200 12372 true mock_requests=1 ``` ```text title="test/html/agent_proxy_route.expected: the values that the case compares" # What test/t/cases/agent_proxy.rb compares for routing by model: status, # port of the upstream, Authorization and x-api-key at the upstream, and # whether the upstream got the client's body unchanged. model_a=200 12372 Bearer upstream-test-key-a none true model_b=200 12373 Bearer upstream-test-key-b none true unknown_model=403 ``` ```text title="test/html/agent_proxy_limit.expected: the values that the case compares" # What test/t/cases/agent_proxy.rb compares for limit_conn per client key: # held is the first stream of key-1 (status, Content-Type, how the client # stopped reading, events read); same_key is a second request of key-1 # while it is open (status, retry-after, Content-Type, body); other_key and # after_close are requests of key-2 and of key-1 after the stream (status, # port of the upstream); limit_log_lines counts the lines of limit_conn in # error.log. held=200 text/event-stream stopped 3 same_key=429 1 application/json {"type":"error","error":{"type":"rate_limit_error","message":"too many concurrent requests"}} other_key=200 12372 limit_log_lines=1 after_close=200 12372 ``` ```text title="test/html/agent_proxy_log.expected: the values that the case compares" # What test/t/cases/agent_proxy.rb compares for an upstream error and the # log handler: ok and overloaded are the status (with Content-Type and body) # of a request that the mock answers with 200 and with 529; a key with ~= # is a text that the line of the log handler in error.log contains. ok=200 overloaded=529 application/json {"type":"error","error":{"type":"overloaded_error","message":"mock: status 529"}} ok_log~=key=key-1 upstream_status=200 status=200 overloaded_log~=key=key-1 upstream_status=529 status=529 ```

On every CI run, the case test/t/cases/agent_proxy.rb sends the requests of its client and compares the values that the client prints with the values shown on this page. The nine files last changed on 2026-10-29 in commit 300e251c.

The client of the test (`test/t/cases/_agent_proxy_client.rb`) sends the requests of each recipe to `/v1/messages` and prints one value per line; the files above are the values that the case compares for the Messages API. The case compares the same recipes for `/v1/responses`, and that a stream of 50 deltas of each API arrives byte for byte. ## The request in v3.0 The steps of a request through the agent proxy of v3.0, each with whether it runs today or which step of the v3 plan adds it. Development version, New in 3.0 Diagram labels (order does not imply connections): - leg-now: Today: on `next`, checked by `agent_proxy.rb` - leg-next: Step 4: the next step of the v3 plan - leg-later: Steps 7a to 7e: later steps of the v3 plan - key: Today: the server rewrite handler checks the client key; an unknown key gets 401 with a JSON body - limit: Today: `limit_conn` per client key; a second stream of the key gets 429 - route: Today: the access handler routes by the `model` of the body and swaps the credential; 403 for an unknown model - budget: Steps 7a and 7b: the token and money budgets of the key in a shared dictionary; `body_json` reads the `model` in C - upstream: Today: `proxy_pass` relays the request to the chosen upstream with the provider's credential - overloaded: Today: an upstream 529 reaches the client with its body - fallback: Step 7e: the reference proxy sets Host and SNI per provider, and sends a 529 to another provider with `error_page 529 = @name` Source: ngx_mruby 8d1500bf0 (2026-10-29) - `docs/proposals/v3-plan.md`: 407-447, 558-576 - `test/conf/conf.d/60-agent-proxy.conf`: 14-111 ## The streamed response in v3.0 How the agent proxy of v3.0 relays a streamed response and charges its usage, each step with whether it runs today or which step of the v3 plan adds it. Development version, New in 3.0 Diagram labels (order does not imply connections): - leg-now: Today: on `next`, checked by `agent_proxy.rb` - leg-next: Step 4: the next step of the v3 plan - leg-later: Steps 7a to 7e: later steps of the v3 plan - deadline: Step 7c: a first-byte deadline gives up a provider that sends no header; `proxy_read_timeout` stays the idle timeout - relay: Today: nginx relays the events of the provider byte for byte as they arrive - frame: Step 7b: the event filter frames the events in C and calls Ruby once per listed event - usage: Step 7b: Ruby reads the usage from `message_start` and `message_delta`, or from the terminal `response.*` event - charge: Step 7a: the counters of the shared dictionary are charged when the response ends or is cut - log: Step 4: the log handlers run before `access_log`, which then logs the cost and the tokens Source: ngx_mruby 8d1500bf0 (2026-10-29) - `docs/proposals/v3-plan.md`: 407-447, 558-576 - `test/conf/conf.d/60-agent-proxy.conf`: 44-55 ## The endpoints in v3.0 What the agent proxy of v3.0 answers itself, what it passes through and what it refuses, as a list of the kinds of request, each with whether it runs today or which step of the v3 plan adds it. Development version, New in 3.0 Diagram labels (order does not imply connections): - leg-now: Today: on `next`, checked by `agent_proxy.rb` - leg-next: Step 4: the next step of the v3 plan - leg-later: Steps 7a to 7e: later steps of the v3 plan - own-errors: Today: the proxy answers 401 and 429 itself with JSON bodies, and 403 with nginx's error page - models: Steps 7c and 7e: `GET /v1/models` from the client's model allowlist, and 504 after the first-byte deadline - pass: Today: `POST /v1/messages` and `POST /v1/responses` with their streams pass through unchanged - upstream-errors: Today: an error of the upstream reaches the client with its body - upgrade: Step 7e: the reference proxy refuses `Upgrade`, the WebSocket transport of the Responses API Source: ngx_mruby 8d1500bf0 (2026-10-29) - `docs/proposals/v3-plan.md`: 424-447 - `test/conf/conf.d/60-agent-proxy.conf`: 93-110 The plan of the agent proxy, its protocol scope and the order of its steps are in [Agent proxy: the v3.0 use case](https://github.com/matsumotory/ngx_mruby/blob/next/docs/proposals/v3-plan.md#agent-proxy-the-v30-use-case) of the v3 plan. A sample of a planned feature appears on this page when the feature and its test merge. ## Cost per request The perf lane counts the instructions that nginx runs per request with callgrind (`test/perf/compare.sh`). Its handlers `agent_auth.rb` and `agent_route.rb` of `test/soak/handlers/` look up the client key and route by the `model` of the body as the recipes above do, in the perf lane's own configuration. On 2026-10-14, with `PERF_N=20000` and nginx 1.31.6, in an Ubuntu 22.04 container on aarch64, the tree of `next` at `bc2488a` with mruby 4.0.0 (#561) measured: | What | Instructions per request (without GC) | |---|---| | The authentication check of the server rewrite handler, over proxying a 2 KB POST without Ruby | 25,276 (23,305) | | Routing by `model` with `JSON.parse` of a 2 KB body, over the same proxying | 85,228 (82,097) | [Agent proxy scenarios in the comparison](/contributing/test/#agent-proxy-scenarios-in-the-comparison) describes the scenarios.