nginxとRubyのLLMプロキシ

認証情報と転送ルールを、ゲートウェイで管理。Rubyが判定し、nginxがストリームを中継します。

動くプロキシの例を見る APIリファレンス

3.x・開発中

複数のエージェントを運用する開発者へ。クライアントごとの認証と転送ルールを、プロキシ側にまとめます。

ngx_mruby 3.x実装済み · nextのテストで検証

使う側

コーディングエージェントCLI・自作ツール

運用する側nginx

mruby / Ruby

キーを確認モデルで振り分け

同時接続数の制限認証情報を差し替え

LLM UPSTREAMS

Provider Amock-model-a

Provider Bmock-model-b

ストリームをそのまま中継

Messages / Responsesを中継。図はテストの規則を示し、外部APIへは送信しません。形式の相互変換は行いません。

MCPの入口を保ち、処理先を分ける。

検索と重い原本処理を持つサービス向け。利用者の接続設定を増やさず、処理ごとの規則をRubyに置く構成案です。

AI CLIENT

POST /mcp接続先は一つ

Rubyの規則nginx + Ruby

検索search

本文取得read_text

原本処理process_source

用途案 · MCP連携は未検証。認可・discovery・プロトコルの検査はMCPサービス側で行います。

設計条件と実装の境界 →

Rubyを書き換え、応答の変化を見る。

まず小さなハンドラを動かし、その設定を自分のプロキシへ。v3開発版はソースからビルドします。

起動

docker compose up --build --wait初回はコンパイルが必要

編集

hello.rbcacheなし: 次のリクエストに反映

確認

curl localhost:8080/hello実際の応答を確かめる

コンテナで試す ↗Linuxのnginxに組み込む →

モデル名から、転送先を選ぶ。

テストで使うRubyと、その検証結果。キーと転送先はmock用です。

Rubyハンドラagent_proxy_route.rb

r = Nginx::Request.new
v = Nginx::Var.new
routes = {
  "mock-model-a" => ["agent_proxy_llm_a", "Bearer upstream-test-key-a"],
  "mock-model-b" => ["agent_proxy_llm_b", "Bearer upstream-test-key-b"]
}
route = routes[JSON.parse(r.body)["model"]]
if route
  v.agent_proxy_backend = route[0]
  v.agent_proxy_upstream_auth = route[1]
  Nginx.return Nginx::DECLINED
else
  Nginx.return Nginx::HTTP_FORBIDDEN
end

検証結果（mock）agent_proxy_route.expected

# What test/t/cases/agent_proxy.rb compares for routing by model: status,
# port of the upstream, Authorization and x-api-key at the upstream, and
# whether the upstream got the client's body unchanged.
model_a=200 12372 Bearer upstream-test-key-a none true
model_b=200 12373 Bearer upstream-test-key-b none true
unknown_model=403

nginx.confを開く

upstream agent_proxy_llm_a {
    server 127.0.0.1:12372;
}

upstream agent_proxy_llm_b {
    server 127.0.0.1:12373;
}

# Concurrent requests per client key: agent_proxy_auth.rb sets the key.
limit_conn_zone $agent_proxy_stream_key zone=agent_proxy_streams:1m;

server {
    listen 18132;

    # Declares the variables that the hooks assign; never requested.
    location = /.agent-proxy-vars {
        internal;
        set $agent_proxy_key_id "";
        set $agent_proxy_stream_key "";
        set $agent_proxy_backend "";
        set $agent_proxy_upstream_auth "";
        return 404;
    }

    # Most variables stay unset on most requests (a rejected key, a
    # request that no hook routed).
    uninitialized_variable_warn off;

    # The client key, checked for every request of this server before a
    # location is chosen. A request without a known key gets 401, and the
    # server-level error_page gives it a JSON body.
    mruby_server_rewrite_handler build/nginx/html/agent_proxy_auth.rb;
    error_page 401 @agent_proxy_401;

    # The upstream gets the credential that agent_proxy_route.rb picked and
    # never the client's.
    proxy_http_version 1.1;
    proxy_set_header Connection close;
    proxy_set_header Authorization $agent_proxy_upstream_auth;
    proxy_set_header x-api-key "";
    proxy_buffering off;

    # The Anthropic Messages API, and the OpenAI Responses API below with
    # the same recipes.
    location = /v1/messages {
        # One request at a time per client key; a second one gets 429 from
        # @agent_proxy_too_many. limit_conn_status is a status that no
        # upstream sends, so that an upstream's own 429 passes unchanged.
        limit_conn agent_proxy_streams 1;
        limit_conn_status 460;
        limit_conn_log_level notice;
        error_page 460 = @agent_proxy_too_many;

        # Picks the upstream block and the credential by the model of the
        # JSON body.
        mruby_enable_read_request_body on;
        mruby_access_handler build/nginx/html/agent_proxy_route.rb;

        # Logs $upstream_status after the response.
        mruby_log_handler build/nginx/html/agent_proxy_log.rb;

        proxy_pass http://$agent_proxy_backend/v1/messages;
    }

    # The zone of limit_conn is per client key, so a stream of a key on
    # either location holds the slot of that key on both.
    location = /v1/responses {
        limit_conn agent_proxy_streams 1;
        limit_conn_status 460;
        limit_conn_log_level notice;
        error_page 460 = @agent_proxy_too_many;

        mruby_enable_read_request_body on;
        mruby_access_handler build/nginx/html/agent_proxy_route.rb;
        mruby_log_handler build/nginx/html/agent_proxy_log.rb;

        proxy_pass http://$agent_proxy_backend/v1/responses;
    }

    # The proxy's own error bodies, in the error format of the API that the
    # client asked for: a named location keeps the request's $uri.
    location @agent_proxy_401 {
        default_type application/json;
        if ($uri = /v1/responses) {
            return 401 '{"error":{"message":"invalid API key","type":"invalid_request_error","param":null,"code":null}}';
        }
        return 401 '{"type":"error","error":{"type":"authentication_error","message":"invalid API key"}}';
    }

    location @agent_proxy_too_many {
        default_type application/json;
        add_header retry-after 1 always;
        if ($uri = /v1/responses) {
            return 429 '{"error":{"message":"too many concurrent requests","type":"rate_limit_error","param":null,"code":null}}';
        }
        return 429 '{"type":"error","error":{"type":"rate_limit_error","message":"too many concurrent requests"}}';
    }
}