Add test
ngx_mruby test uses mruby test. ngx_mruby test is very earlier and experimental version.
Welcome pull-request!
Add test
Section titled “Add test”Add location config to test/conf/nginx.conf
Section titled “Add location config to test/conf/nginx.conf”# Nginx.hello testlocation /mruby { mruby_content_handler build/nginx/html/unified_hello.rb cache;}Add hook script into test/html/ if you need the script for location config
Section titled “Add hook script into test/html/ if you need the script for location config”# test/htdocs/unified_hello.rbif server_name == "NGINX" Server = Nginxelsif server_name == "Apache" Server = Apacheend
Server::rputs "Hello #{Server::module_name}/#{Server::module_version} world!"Add test code to test/t/ngx_mruby.rb
Section titled “Add test code to test/t/ngx_mruby.rb”assert('ngx_mruby', 'location /mruby') do res = HttpRequest.new.get base + '/mruby' assert_equal 'Hello ngx_mruby/0.0.1 world!', res["body"]endThe test client (the test build of build_config.rb) is built with
NO_SSL, so HttpRequest and SimpleHttp cannot send https requests.
For HTTPS, run curl -k with a URL from base_ssl(port), or
openssl s_client -connect localhost:<port>, as the existing tests do (for
example `curl -s -k #{base_ssl(18082) + '/'}` in test/t/ngx_mruby.rb).
Add a test as a fragment and a case file
Section titled “Add a test as a fragment and a case file”Instead of editing test/conf/nginx.conf and test/t/ngx_mruby.rb, a test
can live in files of its own:
test/conf/conf.d/<name>.conf: a fragment included at the end of thehttp {}block. It holds its ownserver {}on a port of its own (18080 to 18199; 18116 is taken by the second nginx thattest/t/cases/_second_instance.rbstarts, and 18180, 18181 and 18185 by the nginx oftest/t/cases/_workers.rb). Fragments forstream {}go totest/conf/conf.d/stream/(ports 12345 to 12399; 12357 and 12385 to 12388 are taken by the second nginx, and 12379 and 12393 by the nginx of_workers.rb; 12393-12396 belong to_udp_wait_session.rband its echo upstream, in a separate, sequential case). Write each port as five digits in these ranges, so thatTEST_SH_PORT_OFFSETmoves it (“Ports” below). A case ofmasterreachesnextby the merge, so a new case takes a port that is free on both branches:git grepthe number onorigin/masterand onorigin/next. Ports below 18110 are allowed.test/t/cases/<name>.rb: a test file run aftertest/t/ngx_mruby.rb, withtest/t/cases/_prelude.rbprepended. The prelude definesbase,base_ssl,http_host,html_path,nginx_t(runsnginx -ton a generated configuration),tcp_client,second_instance,workers_nginx(runs nginx with a master process and several workers, see “Several workers and reloads” below) andsuite_path(the path of a file of the suite in the copy thattest.shruns: run a CRuby helper as`ruby #{suite_path('test/t/cases/_x.rb')}`, see “Ports” below). It starts withSimpleTest.newand ends witht.report. Files whose name starts with_are helpers and are not run as cases;_tcp_client.rb,_filter_connection_client.rb,_request_body_client.rb,_second_instance.rb,_workers.rband_udp_wait_session.rbare CRuby scripts that the prelude helpers and the cases run withruby, and_wait_for_listen.rbis the CRuby script that test.sh runs to wait for nginx (see “Waiting for nginx to listen” below)._agent_proxy_client.rbis a CRuby script (3.0 or later) thatagent_proxy.rbruns itself: it loadstest/soak/mock_llm.rbandtest/soak/http_client.rb, runs the mock LLM upstream on 12372 and 12373 while it sends its requests (see “Mock LLM upstream” below), and printskey=valuelines.- Hook scripts go to
test/html/as before; the fragment refers to them asbuild/nginx/html/<name>.rb.
test.sh copies the fragments, from its copy of the suite (“Ports” below),
into the built nginx’s conf/conf.d/ with the same substitutions as
nginx.conf, and runs every case file in turn. A
failing case makes test.sh exit non-zero, like a failing assertion in
test/t/ngx_mruby.rb.
async_sleep_many.rb checks one content handler that sleeps 1,000 times
for 1 ms on port 18188, in its own server without server-level handlers.
It requires HTTP 200 with slept:1000, then the same timers and
gc_root_fibers from Nginx::Debug.stats as before the request. Each
exchange uses tcp_client with its 60-second limit and completes at EOF;
missing counters or an unavailable stats endpoint fail the case. These
counters observe pending timers and held fibers, not the request pool size.
test.sh defines NGX_MRUBY_DEBUG_STATS for nginx and ngx_mruby so this
check runs in every suite build. After updating an existing build, run a
full sh test.sh after removing build/nginx-X.Y.Z/objs/Makefile
(build_dynamic/ for the dynamic module), so nginx configures the flag
before using ONLY_BUILD_NGX_MRUBY=1 again. For a dynamic module, run
BUILD_DYNAMIC_MODULE=1 sh test.sh. See “Testing” below.
A case that starts nginx itself (nginx_t, second_instance,
workers_nginx) also runs in
the sanitizer cell of CI, where LeakSanitizer appends its report to the
output of a process that leaked and changes that process’s exit status. Run
such a case once with the sanitizer build (see “Running the tests under
AddressSanitizer and UndefinedBehaviorSanitizer” below) before opening the
PR. nginx_t turns leak detection off for its nginx -t run, because
nginx -t returns without freeing the configuration it read, so every
allocation of nginx itself would be reported. The stream-only test at the end
of test/t/ngx_mruby.rb does the same for the nginx -s quit that stops the
nginx it started: that process returns from nginx’s main() without freeing
the pool that main() creates first, and its report would go to the test
runner’s stderr.log. second_instance returns the first sanitizer line of
the second nginx’s error.log or stderr.log as sanitizer, and the cases
print the second nginx’s error.log when the line is set.
Required status check
Section titled “Required status check”.github/workflows/test.yml runs on pull requests to master, next and
v*.x, on pushes to those branches and to v* tags, and on “Run workflow”.
Its ci-ok job is the status check that rulesets require: it fails unless
every job that it lists in needs passed or was skipped because the
changes job decided so. These are the changes job, every job of the
build matrix and the files job, and on next also the soak job and
the perf job (on master, ci-ok does not need perf yet; a later
pull request to master adds it). On
next, the job of the cell that runs nginx under valgrind on every run of
the matrix is one of them, so a valgrind error in that cell fails ci-ok
(valgrind in CI), and a FAIL or ERROR row of the
perf job fails it as well on next (see “In CI” under “Performance
comparison with callgrind”).
A run of test.yml for a pull request starts when the pull request is
opened or reopened, when a commit is pushed to its head, and when it is
marked ready for review (pull_request with the types opened,
synchronize, reopened and ready_for_review). A draft runs no job but
changes and ci-ok, and the job ci-ok is then named ci-ok (draft):
the run tests nothing, GitHub does not merge a draft (condition 7 of
“Review and merge” in AGENTS.md), and under its own name
the pass of that job leaves no ci-ok on the head commit, so the head has
no passing ci-ok until the run that marking the pull request ready
starts has passed. The other workflows that run on pull requests skip
their jobs for a draft in the same way. A new push to the head of a ready
pull request cancels the run of the previous head and starts a new one. A
move of the base alone starts no run; updating the branch
(gh api -X PUT repos/<owner>/<repo>/pulls/<number>/update-branch) pushes
a merge commit to the head and so starts one. The result stands for the
head commit and the base commit of the merge ref at that time, which is
what condition 2 of “Review and merge” asks the pull request to record;
that condition also says when a move of the base makes a new run needed.
A cancelled run of a draft reports no result. Marking a draft ready for
review starts the run that counts, and cancels the run of the draft when
that is still going (cancel-in-progress). In that cancelled run ci-ok
does not run: GitHub marks it cancelled, as it marks every job of the run
that had not finished, and a cancelled check is not a pass. The draft run
of #692 shows this
(run 41193944190);
before, ci-ok ran there on always() and failed, and the draft run of
#678 left a failed
ci-ok (draft) beside the passing ci-ok of the run that counted
(run 41191594780).
A job that GitHub cancels before it starts keeps the expression of its
name as its name, and gh pr checks lists a cancelled check as fail;
neither is the ci-ok of the run that counts.
In every other run ci-ok keeps always(): when the run is cancelled (by
a newer push, which leaves the failure on the head that the push
replaced, or by “Cancel workflow”) before a needed job has finished,
ci-ok runs and fails, because the run did not test the commit to the
end. GitHub’s documentation defines
cancelled() only as “the workflow was canceled” and does not say
whether a needed job that reaches its timeout-minutes makes it true; if
it did, ci-ok on !cancelled() would be skipped, and the skip would
pass the required check. site.yml runs on pushes to next and on
dispatches only, no ruleset requires its jobs, and its deploy job runs
only when its build job passed (Site CI).
CI is the final check in a clean environment, not the place where a
change is first tested (“Branches and pull requests” in
AGENTS.md). The checks that CI runs for a change run
locally first: the check of the added files (Files that do not belong
in the repository), the
suite with the static and the dynamic module, the suite under the
sanitizers and under valgrind, the callgrind comparison, the soak, the
steps of examples/start, nginx-tests, the report of the heads of the
GitHub gems when the change touches build_config.rb,
test/gem_heads.sh or .github/workflows/gem-heads.yml (Heads of the
GitHub gems), the site’s checks and zizmor.
The sanitizers, valgrind, the comparison and the soak need Linux; on
macOS they run in a Linux container, as the sections of this file show.
CI keeps for itself only what one machine does not hold, the other cells
of the build matrix (with OpenSSL 3.5.9 built from source), the x86-64
runner and a clean environment, to which it spreads the local checks,
and the checks that “Branches and pull requests” keeps in CI with their
reason: one, CodeQL (CodeQL). A failure is then found before the push that
would show it in CI, and when a run fails for a reason of the runner (on
#796, three attempts stopped in the step install, where the runner
could not resolve its package mirror), the local result of the same
check shows whether the code is at fault. Before marking a pull request
ready, its author runs the local commands of the checks that the change
touches: sh test/check_added_files.sh for every pull request, and when
test/ci/changes.sh gives build standard or full, the suite with
the static and the dynamic module, under the sanitizers and under
valgrind, as the four standard cells run it (this file describes them;
“Build and test” in AGENTS.md lists them). The author quotes them under
“How was this tested”, and the reviewer checks that they are there. A
check that can run locally is not left to CI. The run of a ready pull request runs only the jobs that the
change needs (the tables below) and finishes in about fifteen minutes. On
2026-10-31, a cell of the build matrix took 6 to 14 minutes, and the run of
test.yml with the whole matrix for #678 took 15 minutes from its start to
its end
(run 41191594773).
A proposal to add a check to CI says whether it can run locally, why CI
needs it, and how many minutes it adds to the run.
Which jobs a run starts
Section titled “Which jobs a run starts”test/ci/changes.sh BASE HEAD classifies the files that a change touches,
and every workflow that decides by the kind of change reads it, so that
the policy is in that one file (sh test/ci/changes_test.sh checks it).
It prints one line per key: the categories code, tests, docs,
site, examples and workflows, then soak and perf, each true or
false, and build, the cells of the build matrix: standard when
code or tests is true, and none otherwise. A renamed file counts
under its old and its new path (git diff --no-renames), so that moving a
source file to docs/ still builds. The job changes of test.yml also
passes the script the run (a draft, the labels of a ready pull request, a
push, a tag, “Run workflow” and its input valgrind), and the script
gives build full, every cell, or none for the runs of the second
table below.
The jobs of the tables, except codeql, run on the runner the checks
that the author has already run locally (Required status
check); codeql runs only in CI
(CodeQL). gem-heads (.github/workflows/gem-heads.yml)
decides by its own path filter, not by test/ci/changes.sh: it runs on a
ready pull request that changes build_config.rb, test/gem_heads.sh or
the workflow, and its local counterpart is bash test/gem_heads.sh build_config.rb "" .claude/tmp/gem-heads.tsv (Heads of the GitHub
gems), which the author runs for such a
change.
| Category | Paths | What a ready pull request runs |
|---|---|---|
code |
src/, mrbgems/, mruby/, dependence/, build_config.rb, build_config.rb.lock, configure, configure.in, config.in, Makefile.in, build.sh, nginx_version, and every path that no other category names |
test.yml: the four standard cells of the build matrix (below), soak and perf. nginx-tests, examples and codeql by their own path filters |
tests |
test/, test.sh, .github/workflows/test.yml |
test.yml: the four standard cells; soak when soak is true; perf compares when perf is true |
docs |
docs/, the Markdown files at the top, AUTHORS, LEGAL, LICENSE, MITL |
No job on next: the author runs npm run check:ci in site/ (Site CI), and site.yml checks the merge on the push to next. On master, the author checks the 2.x pages locally (The 2.x pages of the site) |
site |
site/, .github/workflows/site.yml |
No job: as for docs |
examples |
examples/ |
examples; the site as for docs |
workflows |
.github/ except test.yml and site.yml |
zizmor (all the workflows, these two included); codeql for .github/workflows/ and .github/codeql/ |
The four standard cells build each link mode and each line of nginx (mainline 1.31 and stable 1.30) at least once, and run both memory checks:
| Cell | nginx | Module | OpenSSL | nginx runs |
|---|---|---|---|---|
| nginx 1.31.8, static module | 1.31.8 | static | --with-openssl-src |
directly |
| nginx 1.30.6, dynamic module | 1.30.6 | dynamic | the installed OpenSSL | directly |
| the sanitizer cell | 1.31.8 | static | the installed OpenSSL | built with AddressSanitizer and UndefinedBehaviorSanitizer (Running the tests under AddressSanitizer and UndefinedBehaviorSanitizer) |
| the cell under valgrind | 1.31.8 | static | the installed OpenSSL | under valgrind (valgrind in CI) |
Pull requests run these four cells. Every cell
builds OpenSSL 3.5.9 and installs it before it builds nginx; the first cell builds it into nginx as well
(--with-openssl-src), the path of configure.in and Makefile.in that
the cell under valgrind, the same nginx and module against the installed
OpenSSL, does not take. Every cell (the 8 combinations of nginx 1.31.8
and 1.30.6, the static and the dynamic module, and --with-openssl-src or
not, among them the cell under valgrind, and the sanitizer cell) still runs in the nightly run under valgrind, on the push
of a tag, before a tag on the tip of the base, in the run of the check
pull request, which has the label full-matrix, or with “Run workflow” on
the base when the check needs no pull request (“Before the tag” in
docs/releases/README.md), and on “Run workflow”
without the input base (the second table). A change that only one of the
other cells can show (for example a change for the dynamic module with
nginx 1.31.8) is checked locally with that version (nginx_version, or the NGINX_VERSION of the
cell, with BUILD_DYNAMIC_MODULE), and the pull request can have the
label full-matrix to run every cell.
On master, a pull request for which test/ci/changes.sh gives build
standard or full also runs, locally, the suite with the static module
on the oldest nginx of its build matrix (1.26.3 there), because its four
standard cells do not include that version, and nginx accepts different
configurations in different versions (before 1.27.1, nginx refuses a
stream server without a content handler). On next, the oldest version
of the list NGINX_VERSION of full_matrix in
.github/workflows/test.yml is 1.30.6, the nginx of the dynamic standard
cell, so the run of every such pull request builds it, and no local run
is added. To run the suite with another version of the matrix locally (a
change that only one of the other cells can show, or the runs of “Security
fixes” in docs/releases/README.md), write
nginx_version as the cells of CI do from their NGINX_VERSION, in a
checkout of its own (a git archive of the head), so that the change of
nginx_version is not committed, and run test.sh there without
ONLY_BUILD_NGX_MRUBY (Testing says why):
$ V=1.30.6$ printf 'NGINX_SRC_MAJOR=%s\nNGINX_SRC_MINOR=%s\nNGINX_SRC_PATCH=%s\nNGINX_SRC_VER=nginx-%s\n' $(echo $V | tr . ' ') $V > nginx_version$ sh test.shsoak is true when code is, or when test/soak/ or
test/build_release.sh changed; perf when code is, or when the
measurement of the performance comparison changed (test/perf/,
test/build_release.sh and the scenario files of test/soak/). A file of
no category counts as code, so a file of a new kind runs everything until
a pull request names it in the script, with the reason. A pull request
that changes several categories runs what each of them runs.
| Run | build, soak and perf |
|---|---|
| Draft pull request | Skipped, and files too; the job is named ci-ok (draft) |
| Ready pull request | By the categories of the files that its merge commit (refs/pull/N/merge) changes against its first parent, the base: the table above. perf starts when build is standard or full, and compares when perf is true or the pull request has the label perf |
Ready pull request with the label full-matrix |
Every cell and soak, whatever the files; perf as without the label |
Push to master, next or a v*.x branch |
Skipped |
Push of a v* tag |
Every cell and soak (perf runs on pull requests only) |
“Run workflow” without the input base, or with the input valgrind (the nightly run) |
Every cell and soak (perf runs on pull requests only) |
“Run workflow” with the input base alone |
As for a pull request against base, without labels (perf runs on pull requests only) |
The job changes reads the labels from the event of the run. A label
added to a ready pull request counts from its next run, which a push to
its head starts, or marking it a draft and ready again (gh pr ready --undo, then gh pr ready); a re-run of the jobs keeps the event, and so
the labels it had. The label full-matrix goes on the pull request when it
is opened (gh pr create --draft --label full-matrix), so that marking it
ready starts the run with every cell; gh label create full-matrix adds
the label to a repository that has none.
A change of test.yml alone is tests, so its run starts the four
standard cells, but not every cell, soak or the comparison of perf. A
pull request that changes those jobs in test.yml runs them with the
label full-matrix, with “Run workflow” on its branch (gh workflow run test.yml --ref <branch>), whose job is named ci-ok (manual), or the
comparison with the label perf.
The pull request quotes that run as evidence; it is not the run that
condition 2 of “Review and merge” counts, which is the run of the pull
request.
For “Run workflow” with the input base (a branch or a commit id), the
files are those that the branch changes since its merge base with base,
so that a branch can show what its pull request would run:
$ gh workflow run test.yml --ref claude/topic -f base=nextThe result of such a run is the job ci-ok (dry run), not ci-ok: the
person who starts the run chooses base, so its result is not the check
that the rulesets require of the commit. In the same way, a run with the
input valgrind names it ci-ok (valgrind): such a run builds the tip of
the branch when it starts, so a run that is cancelled would otherwise
leave a failed ci-ok on that commit. Every other “Run workflow” names it
ci-ok (manual): it tests the head of the branch, not the merge commit of
a pull request, and as ci-ok on the head it would pass the required
check of a pull request before the pull request’s own run has passed. valgrind-nightly.yml reads the
result of the run, not the name of the job.
A push to master or next runs only changes, files and ci-ok. Both
branches take changes only through pull requests, and condition 2 of
“Review and merge” in AGENTS.md has the session merge a
pull request only while the run of the pull request still stands for the
merge: when the base has moved since that run, CI runs again only when both
the pull request and the base’s new commits change code or tests. The rulesets require ci-ok, but not that the
branch is up to date with its base. To build and test the tip of a branch
by itself, use “Run workflow” (gh workflow run test.yml --ref next). The
nightly run under valgrind, which the schedule of valgrind-nightly.yml on
master starts on master and on next, is such a run (valgrind in
CI). The other workflows decide on their own when they
run: site deploys on every push to next, and codeql scans pushes to
master and next that change src/, mrbgems/, the workflows or its
configuration (CodeQL); nginx-tests, examples and zizmor
run on pull requests only. After a deploy, the verify-live job of site
reads the GitHub Pages environment and deployment through the REST API with
the job’s own GITHUB_TOKEN (contents: read), so that the requests do not
share the anonymous limit of the runner’s IP address; on a rate limit
(HTTP 429, or 403 with Retry-After or no requests left) it waits
(Retry-After, X-RateLimit-Reset, else 60 s doubling) and repeats at most
5 times per call, 240 s of waiting in all; any other error fails at once.
ci-ok passes a skipped build, soak, perf or files job only when changes
passed and decided that the run does not need it. For perf, which runs
on pull requests only, that is a run whose build is none, or a run
that is not one of a pull request. A perf job that starts and finds no
file to compare (perf false and no label perf) passes without
measuring. A job that was skipped
for another reason (a failed changes job) fails ci-ok, as does a needed
job that did not pass. A pull request whose run skipped jobs that its
files do not need meets condition 2 of “Review and merge” with that run.
Files that do not belong in the repository
Section titled “Files that do not belong in the repository”The files job runs test/check_added_files.sh BASE HEAD TIP. The script
checks two lists of files:
- the files that the change adds: the paths that
HEADhas andBASEdoes not have (git diff --no-renames --diff-filter=A, so a file that is renamed or moved counts as added at its new path); - the files that its commits add: each commit from
TIPback along first parents toBASE(git log --first-parent BASE..TIP), compared with its first parent. A file that one commit adds and a later commit removes stays in the history after the merge, so it is checked as well.
A file of either list fails with exit status 1 when it matches a pattern of
test/forbidden_paths.txt: a session directory or the local settings of an AI
coding agent under .claude/ (everything there except
.claude/settings.json, .claude/commands/ and .claude/skills/, which also
fail below another .claude/), a build directory at the top (build/,
build_dynamic/, build_soak/, build_perf/) or mruby/build/, the build
of the site (site/dist/, site/build-screenshots/, site/generated/,
site/.astro/), a file
that ./configure or the Makefile writes at the top (Makefile, config,
config.status, autom4te.cache/, mrbgems_config, .mruby_version, the
.openssl_*_done files), .vagrant/, a tag file of GNU Global or ctags, a
.log file, .env or .env.*, private key material, an editor or OS
leftover, node_modules/, __pycache__/ or a .pyc file, or a core dump. A
file of the first list also fails when it is empty (0 bytes), unless its name
is .gitkeep or .keep. Both lists include submodule entries (gitlinks),
whatever the git configuration or .gitmodules says about ignoring
submodules.
The script prints each failing path with the pattern and its line in the pattern file, or with “empty file”, and with the commit that added it. When every file passes, it prints how many files it checked and exits 0. It exits 2 when the pattern file is missing, a pattern is invalid, a revision is not in the repository, the repository is a shallow clone or a git command fails. It reads only the pattern file and the git repository.
The commits follow first parents only. A merge commit among them is compared
with its first parent, so the files that its second parent brings in are
checked at the paths they have in the repository, and the commits of the
second parent are not checked one by one. This keeps out the squashed commit
of a subtree merge (./update-mruby-subtree), whose paths lack the mruby/
prefix. It also means that the check does not see a file that a commit of a
merged side branch adds and a later commit of that side branch removes.
For a pull request, the job checks out the merge commit of the pull request
with the whole history (fetch-depth: 0). It compares the merge commit with
its first parent, the base branch that the pull request was merged into, and
checks the commits from the head of the pull request (the second parent) back
to that base. So a pull request whose branch is behind its base is not blamed
for the files that the base removed in the meantime. For a push to a branch,
the job compares the previous tip of the branch (github.event.before) with
the pushed commit, and checks the commits between them. A tag push, a push
that creates a branch and a manual run have nothing to compare, and the job
passes. A pull request that changes the script or the pattern file is checked
with its own version of them, so the review of the pull request covers that
change.
test/forbidden_paths.txt holds one POSIX extended regular expression (as
grep -E reads it) per line, matched against the path from the top of the
repository. A line that starts with ! is an exception, and the last line
that a path matches decides, as in .gitignore: .claude/skills/a/SKILL.md
matches (^|/)\.claude/ and then the exception !(^|/)\.claude/skills/, so
it passes, while .claude/skills/a/.DS_Store also matches (^|/)\.DS_Store$
further down and fails. Empty lines and lines that start with # are
comments. The patterns must be generic: no names of other projects, people or
hosts.
To run the check before opening a pull request, commit the change and compare
it with the base branch (origin/next for a change to next). With two
arguments, the script also checks the commits from HEAD back to the base.
It needs the whole history, so run it in a clone that is not shallow:
$ git fetch origin master$ sh test/check_added_files.sh origin/master HEADcheck_added_files.sh: 2 added file(s) checked between origin/master and HEAD and in the 2 commit(s) of origin/master..HEAD, none forbidden or emptyWhen a file fails, remove it from the change (git rm --cached <path> keeps
it on disk), or, if it belongs in the repository, change
test/forbidden_paths.txt (for an empty file, the script) in the same pull
request and say why in its description. A file that a commit adds and a later
commit removes stays in the history: remove it from the commit that added it
(for example with git rebase -i), or squash the commits.
The 2.x pages of the site
Section titled “The 2.x pages of the site”The site ngx.mruby.org is built on next (site/ there, and
docs/proposals/site.md there for the plan). Its 2.x pages under /v2/
are built from docs/ on master. A pull request to master runs no
job for them: local verification covers the site before a PR is ready,
while the push to next builds and deploys it. The author of a pull request
that changes docs/, the Markdown files at the top, or a file that they
link to, runs these checks locally before marking it ready, and quotes
them under “How was this tested”:
-
The relative links and anchors of the Markdown files that git tracks, with
check_links.pyofnext, whichmasterdoes not have. From the top of the checkout of the pull request, withTa directory of your own outside the repository or under.claude/tmp/:$ git fetch origin next$ git show origin/next:site/scripts/check_links.py > "$T/check_links.py"$ git ls-files -z -- '*.md' ':!:mruby/' ':!:dependence/' | xargs -0 python3 "$T/check_links.py" . -
The 2.x pages built from the pull request: in a checkout of
next, with the Node.js ofsite/.node-versionin that checkout (on a machine with another version, a version manager ornpx -p node@<that version>gives it), afternpm ci --ignore-scriptsandnpx playwright install chromium webkitin itssite/,NGX_MRUBY_V2_TREE=DIR npm run check:ciwithDIRthe top of the checkout of the pull request (“Site CI” indocs/test/README.mdonnext). It builds the 3.x pages ofnextand the 2.x pages fromDIR, and runs the checks of the site on both, the browser checks among them; it compiles the Ruby blocks of the pages withMRBCormruby/build/host/bin/mrbcof the bundled mruby; when neither exists, it says that it skipped them. It prints the commit ofnext; quote it with the head of the pull request. The generator stops on a link ofdocs/to a file that git does not track, and the checks fail on a broken link of the built site, on a 2.x page without its banner, and when the search of a version shows pages of the other.The release notes in
docs/releases/(vX.Y.Z.mdandvX.Y.Z.ja.md) are also 3.x pages (/releases/...and/ja/releases/...) oncemasteris merged intonext. This check builds the 3.x pages fromdocs/of the checkout ofnext, so it does not check a release note that the pull request adds or changes as a 3.x page, and the checks of the Japanese pages ofnext(the spaces between Japanese and Latin text, the code blocks, the line breaks) do not read that file. A pull request that adds or changes a release note therefore copies that file intodocs/releases/of the checkout ofnextbefore the run, and commits it there in a local commit that is never pushed:$ git add docs/releases/vX.Y.Z.md docs/releases/vX.Y.Z.ja.md$ git commit -m 'Release notes to check (local only, not pushed)'The generator reads only the files that git tracks, and
site/scripts/export-docs.mjsstops withsource must exist at HEAD before exporton a page whose source is not in the commitHEAD, so staging alone stops the build for a new file, while a change of a file thatHEADalready has would go through; the commit serves both. One run then checks the file as a 2.x page and as a 3.x page.check:ciprints the local commit: quote it with its parent, the commit ofnext. Afterwards, remove that checkout or reset it toorigin/next. This keeps the push tonextafter the merge ofmasterfrom stopping the deploy of the site on those checks.
Two workflows here serve the 2.x pages:
.github/workflows/site-dispatch.ymlruns on every push tomaster, and startssite.ymlonnextwithgh workflow run site.yml --ref next, so that the site is built again with the new 2.x pages, checked, and deployed only when the checks pass. It fails when GitHub refuses the dispatch or it cannot name the run that the dispatch started..github/workflows/site-v2.ymlruns only on “Run workflow” (gh workflow run site-v2.yml --ref <branch>). It checks outnextbeside the branch, builds the site withsite/and the 3.x pages ofnextand the 2.x pages from the branch’sdocs/(node site/scripts/build-v2.mjs --tree DIRofnext), and runs the link, banner and search checks above. The built site is the workflow artifactsite-v2. It does not run on pull requests or pushes: on a push, the run ofsite.ymlthatsite-dispatch.ymlstarts builds and checks the same 2.x pages. That run checks the links of the Markdown files ofnext, not ofmaster: the links of the Markdown files ofmasterthat the site does not render (AGENTS.md, for example, whose links the generator turns into GitHub URLs) are checked only by the author’s run ofcheck_links.pyabove.
Neither is in the needs of ci-ok.
Supply chain checks
Section titled “Supply chain checks”The badge above is the project’s OpenSSF Best Practices badge (passing level), which is separate from the Scorecard results below.
Four more workflows check the supply chain of the build and the code. None
of them is in the needs of ci-ok, so none of them blocks a merge through
the ruleset; “Review and merge” in AGENTS.md asks for every CI check on
the head commit to pass, which includes the zizmor and codeql jobs of a
pull request that runs them. Like test.yml, the ones that run on pull
requests skip their jobs for a draft, and marking it ready for review
starts them.
Static analysis of the workflows
Section titled “Static analysis of the workflows”.github/workflows/zizmor.yml runs zizmor on
ready pull requests that change .github/workflows/,
.github/dependabot.yml or .github/zizmor.yml, and on “Run workflow”;
pushes do not run it, because the run of the pull request checked the same
files. It
audits the files under .github/ with the default persona, including the
audits that query GitHub (for example known vulnerable actions and commits
that are not on the action’s repository), and fails on any finding. The
vendored mruby/ tree has workflows of its own, which ngx_mruby does not
run; they are not audited. The version of zizmor is pinned in the workflow
(version:): Dependabot updates the pin of the action but not that input.
A finding is fixed in the workflow, or accepted with its reason: by a
# zizmor: ignore[<audit>] comment on a line of the finding, with the
reason in a comment above it, or by an ignore rule in .github/zizmor.yml
with the reason next to it. To run the same check before opening a pull
request (the online audits need a GitHub token):
$ python3 -m venv .claude/tmp/zizmor && .claude/tmp/zizmor/bin/pip install zizmor==1.31.1$ GH_TOKEN=$(gh auth token) .claude/tmp/zizmor/bin/zizmor --config .github/zizmor.yml .githubOpenSSF Scorecard
Section titled “OpenSSF Scorecard”.github/workflows/scorecard.yml runs
OpenSSF Scorecard on pushes to
master, weekly, and on “Run workflow”. Scorecard supports only the default
branch, so pull requests do not run it. It publishes the results to the Scorecard API
(https://api.scorecard.dev/projects/github.com/matsumotory/ngx_mruby),
uploads them to code scanning (the repository’s Security tab), and keeps the
SARIF file as an artifact of the run for 5 days. Publishing makes the
Scorecard API check the workflow itself, so the workflow has no run steps and
no defaults; the comment at its top lists the restrictions.
GitHub disables the scheduled workflows of a public repository after 60 days
without activity in the repository. Then neither this workflow nor the next
one runs on its schedule, and nothing fails. “Run workflow”, or enabling the
workflow again (gh workflow enable scorecard.yml, gh workflow enable gem-heads.yml), starts them again.
CodeQL
Section titled “CodeQL”.github/workflows/codeql.yml runs CodeQL
code scanning for three languages, c-cpp, ruby and actions, and
uploads the results to code scanning (the repository’s Security tab). It
runs on ready pull requests to master and next and on pushes to these
branches that change src/, mrbgems/, .github/workflows/ or
.github/codeql/, every Wednesday at 04:37 UTC (GitHub runs a schedule
only from this file on master, so the weekly run scans master), and on
“Run workflow”. It
replaces GitHub’s default setup of code scanning, which ran on every pull
request and every push; the default setup is turned off in the repository
settings, which the owner manages. The file is the same on master and
next.
All three languages use build-mode: none: for C and C++ CodeQL reads the
sources without a build of nginx and mruby, so the job needs no build.
CodeQL’s documentation says that a database made without a build can give
less accurate results than one made from a build
(“Build options for compiled languages”).
The actions are pinned by commit id like those of the other workflows, and
zizmor checks the file.
CodeQL is the one check that “Branches and pull requests” in AGENTS.md keeps in CI: an author does not run it before marking a pull request ready. The job uploads its results to code scanning, and GitHub shows the new alerts on the lines that the pull request changes as annotations in the pull request (“Triaging code scanning alerts in pull requests”). A local run would need the CodeQL bundle, about 550 MB for Linux on arm64 in the version that the pinned action uses by default (2.27.3), and the system requirements of CodeQL list Linux on arm64 and Apple Silicon as beta (System requirements). CodeQL has not been run locally for this repository.
CodeQL does not analyze the vendored trees, mruby/ (mruby, with the code
that mruby itself vendors) and dependence/ (ngx_.github/codeql/codeql-config.yml, which the init step of the workflow
reads (config-file), lists them under paths-ignore. They are upstream
code that the repository takes with git subtree and does not edit by hand
(“Do not” in AGENTS.md), so an alert in them would not
be fixed here; a change of them comes with the next update of the subtree.
The owner decided on 2026-10-31 that CodeQL leaves them out, that the
alerts that code scanning had on them are dismissed as vendored upstream
code, and that nothing is reported upstream for them. GitHub’s
documentation says that paths-ignore applies to the interpreted
languages, Ruby among them, and to C and C++ analyzed without a build
(“Workflow configuration options for code scanning”).
Heads of the GitHub gems
Section titled “Heads of the GitHub gems”.github/workflows/gem-heads.yml runs test/gem_heads.sh every Monday, on
“Run workflow”, and on ready pull requests that change build_config.rb,
the script or the workflow. For each conf.gem github: line of
build_config.rb, it reports the head of the branch that a build clones
(the line’s branch:, or master, mruby’s default when the line has
none), whether the gem’s default branch is another branch, whether the
repository was renamed or archived, and whether the head moved since the
previous report. The conf.gem mgem: line (mruby-secure-random) is not
checked: mruby resolves it through the mgem-list repository.
The previous report is the artifact gem-heads of the newest successful
scheduled or manual run of the workflow on master; artifacts are kept for
90 days. master commits no build_config.rb.lock, so this report is the
only record of the heads that earlier builds took. A run with no previous
report reports every gem as new. A moved head is a warning annotation and
a row in the job summary, with a link that compares the two commits; the job
still passes. A gem whose repository or branch cannot be read fails the job,
and so does a failure to list the earlier runs. A previous report that cannot
be downloaded is a warning, and every gem is then reported as new.
To run the script on a checkout (it needs gh with a token):
$ bash test/gem_heads.sh build_config.rb "" .claude/tmp/gem-heads.tsvWhen a run reports a move, a session does the following. This is the sessions’ procedure, not a decision of the owner.
- Read the commits behind the comparison link. Every build of
master, CI included, and every source build of a released 2.x tag already takes the new head; CI does not run for a move by itself, so run thetestworkflow onmaster(gh workflow run test.yml --ref master) and read its result. - When the new head breaks the build or the suite, or changes what a 2.x
configuration, script or build observes, report it to the owner with the
run, and propose a pull request to
master: a fix, or achecksum_hash:that pins the gem to the last good commit, with its release notes entry (seedocs/releases/README.md). For the owner’s gems (matsumotory/mruby-*), “Other repositories” inAGENTS.mdapplies. - On
next,build_config.rb.lockpins each gem, so the move reachesnextonly through a pull request that moves the pin; theperfjob measures that pull request like any change of the lock.
Testing
Section titled “Testing”build nginx into ./build/nginx and test on ./build/nginx
Section titled “build nginx into ./build/nginx and test on ./build/nginx”sh test.shtest.sh configures nginx with --with-debug, --with-http_stub_status_module,
--with-http_ssl_module, --with-http_auth_request_module and --with-stream
(see NGINX_DEFAULT_OPT and NGINX_CONFIG_OPT in test.sh), so the test
configuration can use the directives of these modules. CI runs the same
test.sh on a fresh checkout. A local build directory keeps the options that
nginx was first configured with: the Makefile configures nginx again only when
build/nginx-X.Y.Z/objs/Makefile does not exist (build_dynamic/... for
BUILD_DYNAMIC_MODULE). After these options change, remove that file and run
test.sh once without ONLY_BUILD_NGX_MRUBY; otherwise nginx stops at the
test configuration with an “unknown directive” error. After changing
nginx_version or BUILD_DYNAMIC_MODULE, run test.sh once without
ONLY_BUILD_NGX_MRUBY as well: it then uses another build directory
(build/nginx-X.Y.Z of the new version, or build_dynamic/), and configures
nginx there when that file is missing in it.
To run nginx under valgrind, set the environment variable NGINX_RUNNER:
$ NGINX_RUNNER=valgrind sh test.shvalgrind writes its report when nginx exits at the end of test.sh, and
test.sh does not take valgrind’s errors into its exit status (it prints
the exit status of nginx, 99 with the option --error-exitcode=99 below
when valgrind counted errors). To check a
run as CI does, give valgrind the options of CI and read the ERROR SUMMARY
line of the log (see valgrind in CI):
$ rm -rf build/valgrind && mkdir -p build/valgrind$ NGINX_RUNNER="valgrind --error-exitcode=99 --leak-check=full --show-leak-kinds=definite,indirect,possible --errors-for-leak-kinds=definite,indirect --suppressions=$PWD/test/valgrind.supp --log-file=$PWD/build/valgrind/valgrind.%p.log" sh test.sh$ grep 'ERROR SUMMARY' build/valgrind/valgrind.*.logWaiting for nginx to listen
Section titled “Waiting for nginx to listen”test.sh starts the test nginx in the background ($NGINX_RUNNER build/nginx/sbin/nginx &) and sends the first request only after nginx
accepts TCP connections. It runs test/t/cases/_wait_for_listen.rb with
CRuby twice:
-
Before it starts nginx, the script waits until no process accepts a connection on the ports below. A port that another process listens on would make the wait after the start succeed without nginx, so the run fails when one still accepts after
NGINX_HEATTIMEseconds, with the ports:wait_for_listen: before nginx starts, 1 of 33 ports still accept connections after 60.05 seconds (limit 60.00): 18080; another process listens on them -
After it starts nginx, the script connects to each port every 0.1 seconds until every port has accepted one connection, and prints the time that took, counted from the start of the script, right after nginx was started:
wait_for_listen: nginx (pid 1234) accepted connections on all 33 ports after 0.21 seconds (limit 60.00)It fails when the nginx process ends first, or when
NGINX_HEATTIMEseconds pass, and names the ports that still refused connections:wait_for_listen: nginx (pid 1234) ended after 0.47 seconds, before it accepted connections on 33 of 33 ports: 18080, 18081, ...; see its output above and logs/error.logwait_for_listen: nginx (pid 1234) did not accept connections within 60.00 seconds (NGINX_HEATTIME) on 33 of 33 ports: 18080, 18081, ...
The ports are those of every listen directive in test/conf/nginx.conf
and test/conf/conf.d/*.conf, in the copy that “Ports” below describes,
with TEST_SH_PORT_OFFSET applied: the http {} servers (57 ports on next as of
2026-10-30). nginx opens all its listening sockets in one loop
(ngx_open_listening_sockets()) after it has read the whole configuration,
so one port would almost always show that the others are open. Each port
costs one connection of well under a millisecond, though, and checking every
one leaves no gap between the first socket and the last, and also catches a
fragment of conf.d that did not reach the installed configuration. The
stream {} ports (test/conf/nginx.stream.conf and
test/conf/conf.d/stream/) are left out: a connection to a stream server
runs that server’s session code, such as the Nginx::Stream::Async.sleep 3000 of the 12352 server, and the cases read what that code writes to
error.log. nginx opens the stream sockets in the same loop, right after
those of http {}, and the first test that connects to a stream port
runs after many HTTP tests of test/t/ngx_mruby.rb have passed. The ports that
Nginx::Server#add_listener and Nginx::Stream.add_listener add are opened
in the same loop as well, and are not checked.
NGINX_HEATTIME is the upper bound, in seconds, of each of the two waits.
The default is 60, and CI does not set it. A command that sets it, such as
NGINX_RUNNER=valgrind NGINX_HEATTIME=10 sh test.sh, waits at most that
long: the run goes on as soon as nginx listens, and fails at 10 seconds only
when nginx is not listening by then, where a first request sent at that time
would have been refused (Errno::ECONNREFUSED).
The default rests on these measurements. On 2026-10-19, the test nginx
built from master at 03aad7031 (nginx 1.31.6, static module) was started
the way test.sh starts it, in a container of Ubuntu 22.04 on an aarch64
host with 11 CPUs whose load average was about 15 from other builds, and
the script above measured the time until all ports accepted connections:
NGINX_RUNNER=exec, 10 starts: 0.10 to 0.11 seconds each, that is, every port accepted on the second round of attempts (the script tries every 0.1 seconds);NGINX_RUNNER=valgrind(valgrind 3.18.1), 5 starts: 2.27 to 2.61 seconds.
In CI, a cell under valgrind (nginx 1.28.3 on an x86_64 runner) was still
reading the configuration 10 seconds after the start: valgrind’s report at
the SIGTERM of test.sh shows nginx inside ngx_conf_parse()
(the run,
reported in
#564).
The time is longer on a loaded CI runner, under valgrind (here more than 20
times the time without it), with the sanitizer build (not measured here) and
with more configuration to read. 60 seconds is 6 times the 10 seconds of
that CI run and about 23 times the slowest start under valgrind above.
The upper bound can create these races:
- A start slower than the bound fails the run although nginx would have
started. Raise
NGINX_HEATTIMEfor slower setups; the wait still ends as soon as nginx listens, so a higher bound costs time only when nginx hangs without ending (an nginx that exits is reported at once). - A listening socket is not yet a running handler. The kernel accepts a
connection as soon as nginx listens, and the test nginx is a single
process (
master_process off) that runs theinit_workerhooks before it reads any connection. The first request then waits in the socket’s backlog until nginx reads it; the test client (mruby-simplehttp) reads without a timeout, so it waits too. - nginx later reads each probe connection of the wait, finds it closed
without a request, and closes it. No handler runs for it, and nginx writes
only
debuglines about it toerror.log.
Stopping nginx
Section titled “Stopping nginx”test.sh stops only the nginx processes that it started, by their pid. It
never signals a process by its name (killall, pkill), so the nginx
processes of other checkouts, of the soak test, of nginx-tests and of other
users keep running. Two runs of test.sh on one machine need different
ports (“Ports” below); with the same ports, the second run fails in the
wait before the start, as “Waiting for nginx to listen” above shows.
-
When
test.shexits, after the last case, after a failure, or on SIGHUP, SIGINT or SIGTERM, it stops the test nginx with SIGTERM and waits until the process has ended. It then prints how long that took and the exit status of nginx; the status does not change the exit status oftest.sh:test.sh: nginx (pid 374) ended within about 0.1 seconds of the first SIGTERM; SIGTERM sent 1 timestest.sh: nginx (pid 374) exit status 0The shell runs the trap of a signal only after the command that runs at that moment has ended, such as the wait for nginx,
test/t/ngx_mruby.rbor the current case file, so the stop starts then. A shell that is not interactive also starts a command run with&with SIGINT ignored, sosh test.sh &does not stop on SIGINT; send SIGTERM instead. -
Before each signal,
test.shchecks that the pid still runsbuild/nginx/sbin/nginxof this checkout: its command line, fromps -ww -o args=, contains that absolute path, also under valgrind. The shell collects a background process that has ended, so after a crash of nginx its pid may belong to another process by the time of the stop. Such a pid counts as ended and gets no signal:test.sh: nginx (pid 374) had ended before the stop; no signal sent -
Before it starts nginx,
test.shreadsbuild/nginx/logs/nginx.pid(build_dynamic/...forBUILD_DYNAMIC_MODULE). When an earlier run was killed before it could stop its nginx, that nginx still runs and the file names it.test.shstops it in the same way if the process runsbuild/nginx/sbin/nginxof this checkout, by the check above, and leaves any other process alone:test.sh: stopping the nginx (pid 7) that an earlier run left runningtest.sh: logs/nginx.pid names pid 7, which does not run /path/to/build/nginx/sbin/nginx; left running
The test nginx is a single process (master_process off). It checks for a
signal only after it has handled its events, in the loop of
ngx_single_process_cycle(): the signal handler sets ngx_terminate, and
the loop exits once ngx_process_events_and_timers() returns. A SIGTERM
that arrives after that check and before the next epoll_wait() does not
interrupt the wait, and with no timer and no connection, epoll_wait()
waits without a time limit. nginx then logs signal 15 (SIGTERM) received
and keeps running. A master process does not have this gap: it waits for
signals with sigsuspend(), and while its workers still run after a
SIGTERM, it sends them the command to terminate again, over the socket pair
of each worker that the worker’s event loop watches (SIGTERM only where that
write fails), with a delay that starts at 50 milliseconds and doubles, and
it sends SIGKILL once the delay exceeds one second, about 1.6 seconds after
the first signal (ngx_master_process_cycle()). test.sh does the same for
the single process with the signal: it sends SIGTERM again 1, 3, 7, 15 and
31 seconds after the first one while the process still runs, and a SIGTERM
that arrives during epoll_wait() ends the wait. It waits longer than the
master before SIGKILL because the test nginx may run under valgrind, which
checks for leaks at the exit, and SIGKILL would lose that report. After
NGINX_STOPTIME seconds (default 60) it sends SIGKILL, prints
test.sh: nginx (pid 374) did not end within 60 seconds (NGINX_STOPTIME) after 6 SIGTERM; sent SIGKILLand exits 1. test.sh counts these times in rounds of sleep 0.1 and a
ps, so the real times are somewhat longer than the printed ones. The
second nginx of test/t/cases/_second_instance.rb, also a single process,
is stopped with SIGQUIT, sent again in the same way.
How often a run hits the gap was measured on 2026-10-29, with the test
nginx built from next at c9c49023e (nginx 1.31.6, static module), started
as test.sh starts it, in a container of Ubuntu 22.04 on an aarch64 host
with 11 CPUs:
- 800 starts of the test nginx, each followed by one SIGTERM: 400 right after
the response to one request, and 400 at 0 to 20 milliseconds after three
requests were sent. nginx ended after the one SIGTERM every time. The gap
lasts from the check of
ngx_terminateto the call ofepoll_wait(), a debug log line and a lookup of the timer tree, so a loop of this size does not find it. - gdb attached to the test nginx, waited until nginx had closed the
connection of a request, stopped it at the next call of
ngx_epoll_process_events(), with no event and no timer left (timer=-1), and delivered SIGTERM there. nginx loggedsignal 15 (SIGTERM) received from ..., exitingand thenepoll timer: -1, and still ran 10 seconds later. A second SIGTERM ended it within 0.1 seconds. This is the state of the run oftest.shat7926f72(#600), where nginx logged the SIGTERM and stayed inepoll_wait()for 20 minutes.
The ports of the suite are the numbers from 12345 to 12399 and from 18080
to 18199: the listen directives of test/conf/, the addresses of
add_listener, the second nginx of test/t/cases/_second_instance.rb
(18116, 12357, and 12385 to 12388), the nginx of test/t/cases/config_hooks.rb (18158), the
nginx -t of test/t/cases/stream_duplicate_hooks.rb (12398), the
nginx with several workers of test/t/cases/_workers.rb (18180, 18181,
18185, 12379 and 12393, in test/conf/workers/), the independent UDP
session helper _udp_wait_session.rb (12393-12396, including its echo
upstream; it finishes before the workers case starts), the
mock LLM upstream of test/t/cases/_agent_proxy_client.rb (12372 and
12373), the stream-only test of test/t/ngx_mruby.rb (12399), and the
ports that the tests, the hook scripts and the expected outputs of
test/html/ name. TEST_SH_PORT_OFFSET (an integer, default 0) moves all
of them by the same amount, so that two checkouts can run test.sh at the
same time on one machine, each with its own offset:
$ TEST_SH_PORT_OFFSET=1000 sh test.sh # 13345 to 13399 and 19080 to 19199Before it installs the configuration, test.sh runs
test/t/cases/_stage.rb, which copies test/t, test/conf, test/html,
test/soak/mock_llm.rb and test/soak/http_client.rb to
build/nginx/harness/test/ (build_dynamic/... for
BUILD_DYNAMIC_MODULE). In every text file of the copy, each number of
those two ranges, written as five digits with no digit, letter or _ next
to it, becomes the number plus the offset:
stage: copied 164 files to /path/to/build/nginx/harness/test; 593 port numbers moved by 1000test.sh then installs the configuration and the hook scripts from the copy,
waits for the ports of the copy, and runs test/t/ngx_mruby.rb and the case
files of the copy. The tests run the CRuby helpers of the copy through
suite_path (test/t/cases/_prelude.rb, and test/t/ngx_mruby.rb for its
own use), which joins a path of the suite to TEST_SH_STAGE_DIR, the
directory that test.sh sets. Without that variable, as when a case runs
outside test.sh, suite_path gives the path in the checkout. The tests
still run from the root of the checkout, because the configuration names
the hook scripts with paths relative to it (build/nginx/html/<name>.rb).
With an offset of 0, the copy is the same as the source.
The rule is textual, so a new test keeps to it:
- Write a port as five digits in one of the two ranges, and write a number
in those ranges only where it is a port or a value that the test compares
with a number of the same file set.
_stage.rbfails, naming the file and the line, for such a number written with an underscore (18_132), which the rule would not see. A port built by arithmetic is moved only when its five-digit part is in the ranges:18110 + 1becomes19110 + 1with an offset of 1000, but18000 + 110and a port built from strings stay as they are, so write the number out. - One number in the ranges is not a port:
test/conf/conf.d/20-filter-connection.confoutputsf.output 12345, andtest/t/cases/filter_connection.rbcompares the body with'12345'and its length with'5'. Both files are moved alike, so the case still checks thatFilter#outputconverts an Integer withto_sinto five bytes. - The offset must keep every port between 10000 and 65535, so that every
number stays five digits long (an offset from -2345 to 47336);
_stage.rbfails otherwise. It must also keep the ports apart from those of other runs: the soak test (12360 to 12362,SOAK_PORT_BASE), the performance comparison (12370 to 12372,PERF_PORT_BASE) and nginx-tests (8000 to 8999). With offset 0,test.shand the performance comparison both use 12372.
Two runs need two checkouts as well as two offsets: one build directory holds one test nginx, its logs and its copy of the suite. The two offsets must keep the ranges of the two runs apart. The range from 18080 to 18199 has 120 numbers, so the offsets differ by 120 or more. The range from 12345 to 12399 of one run lands on the range from 18080 to 18199 of the other when the difference of the offsets is from 5681 (18080 - 12399) to 5854 (18199 - 12345), with either sign, so the difference is not in that interval. With an offset above 14568 (32767 - 18199), the ports of 18080 to 18199 reach the default ephemeral range of Linux (32768 to 60999), where a client socket can take a port before nginx binds it. The offsets 0 and 1000 of the example meet all three conditions.
Several workers and reloads
Section titled “Several workers and reloads”The test nginx of test.sh is a single process (master_process off), so
it cannot show what happens with several workers or across a reload. A case
that needs that runs test/t/cases/_workers.rb with CRuby through the
prelude helper workers_nginx(mode), as other cases run
_second_instance.rb. The helper starts an nginx of its own with
daemon off, master_process on and several workers, drives it, stops
it, and prints key=value lines, which workers_nginx returns as a Hash
of each key to the Array of its values; the case asserts on them. The
other cases keep the single process, and the test nginx of test.sh does
not change. test/t/cases/workers.rb has the three modes of today:
| Mode | Configuration | Workers | Ports | What the case asserts |
|---|---|---|---|---|
isolation |
test/conf/workers/isolation.conf |
4 | 18180 | Requests to /worker, one per connection, until every worker has answered at least twice (at most 200). Each answer is 200 with the pid of its worker (Nginx::Var#pid), a pid of one of the four workers, and the count of each handler of the location (mruby_set_code, the rewrite, access and content handlers) in a global variable of the state: in the answers of one worker the counts go 1, 2, 3, …, the same for the four handlers, so each worker has an mrb_state of its own that its requests share. Every worker answered. |
reload |
test/conf/workers/reload.conf |
2 | 18181, stream 12379 | A request to /sleep and a stream session start, whose handlers sleep (Nginx::Async.sleep, Nginx::Stream::Async.sleep) until the file release exists. The helper writes the configuration of generation 2 and a new script.rb (read with mruby_content_handler ... cache), and sends SIGHUP to the pid of the pid file. /generation then answers for generation 2 from a new worker, while both sleeps are still in flight, every old worker has logged gracefully shutting down, and the workers that hold the sleeps still run (an old worker without a sleep in flight may have ended). /script then answers with the new script from a new worker: it is read after the old workers have logged that they shut down, because until then an old worker may still accept a connection and answer with the script that it cached. After release, the old workers answer with generation 1 (200 with the pid of the worker that started the sleep, and stream generation 1), each after its gracefully shutting down in the error log, and then end. Generation 2 answers on both protocols from two new workers. |
shutdown |
test/conf/workers/shutdown.conf |
1 | 18185, stream 12393 | worker_shutdown_timeout 1s. A request to /sleep and a stream session start, whose handlers sleep for up to 10 seconds (Nginx::Async.sleep, Nginx::Stream::Async.sleep). The helper sends SIGHUP and waits until the old worker has ended: it ends less than 5 seconds after the reload, when the timeout has made nginx set c->close on the session and call its read handler, which ends the wait and the session. The session gets no bytes (the return of the server does not run), and the code logged that it started once and never that it resumed. nginx terminated the request: it gets no bytes, and its handler never logged that it resumed. |
The modes also assert on the end of the run: the stop gives master exited 0 with every worker of every generation ended, the error log has
no line at the level error or above (error, crit, alert, emerg),
no sanitizer line is in its logs, and under valgrind the ERROR SUMMARY of
the master and of every worker counts 0 errors.
What the helper does:
- It writes
build/nginx/workers/MODE/nginx.conf(build_dynamic/...forBUILD_DYNAMIC_MODULE) fromtest/conf/workers/MODE.confof the copy of the suite thattest.shruns, soTEST_SH_PORT_OFFSEThas moved its ports (“Ports” above), withdaemon off,master_process on,worker_processes N,pid build/nginx/workers/MODE/nginx.pid,user rootwhen it runs as root (as in a container, where the workers would run asnobodyotherwise), and, for a dynamic module build,load_modulewith an absolute path.__PREFIX__and__GENERATION__in the file become the prefix and the generation. The files are not inconf.d/, so the test nginx does not include them. - It fails when a port of the mode already accepts connections, starts
nginx under
NGINX_RUNNER(when that is set and notexec), waits until the http ports accept connections and the master runs all its workers (ps: the processes whose parent is the master), each wait bounded byNGINX_HEATTIME(default 60 seconds). The stream ports are not probed: a connection to them runs their session code. - The workers are told apart by their pid.
isolation.conflistens withreuseport, so each worker has a listening socket of its own and the kernel spreads the connections over them by the client’s address and port (Linux); without it one worker may take most connections. - A reload rewrites the configuration and sends SIGHUP to the master,
the pid in the pid file when that is the process the helper started. The helper waits for what it then asserts, never for a fixed
time: the new answer, and the line
gracefully shutting downof each old worker. A handler that should still be in flight across the reload waits for a file that the helper writes afterwards, so it is in flight however long the reload takes. - It stops nginx with SIGTERM to the master, the pid of the pid file when
that is the process it started (otherwise the pid it started), and waits
until the master has exited and every worker it saw, of every
generation, has ended. The master sends its
workers the command to terminate, waits for them and exits; it waits for
signals with
sigsuspend(), so it has none of the gap of the single process that “Stopping nginx” above describes, and one SIGTERM is enough. AfterNGINX_HEATTIMEseconds, the master and the workers that still run get SIGKILL. The stop of the test nginx bytest.shdoes not change. - The error log of the master and its workers is
build/nginx/logs/workers_MODE.error.log, at the levelnotice, and the standard output and error of the master go tobuild/nginx/logs/workers_MODE.stderr.log. CI’s step “Check for sanitizer reports” reads both, as the other*.logfiles ofbuild/nginx/logs/. Under valgrind, each worker is a fork of the master that valgrind follows, so with%pin--log-fileeach process writes a log of its own; the helper reads theERROR SUMMARYof the master and of each worker, and CI’s step “Check the valgrind reports” reads the same logs.
To add a case: add a mode to MODES in _workers.rb (the number of
workers and the ports), its configuration as test/conf/workers/MODE.conf
on free ports of the two ranges (“Ports” above), a function that drives it
and prints what it sees, and the assertions on those values to a case
file. Print the responses and the order of the log lines, not a verdict of
the helper, so that the case asserts on the real answers.
The case runs in every cell that runs the suite, so it is kept short. On
2026-11-01, with the build of next at bf1b742b7 and this case (nginx
1.31.8, static module) in a container of Ubuntu 22.04 on an aarch64 host
with 11 CPUs, test/t/cases/workers.rb took 0.87 to 0.91 seconds in 5 runs
without valgrind (the mode isolation about 0.15 seconds with 10 to 15
requests, the mode reload 0.37 to 0.42 seconds), and 5.1 seconds of the
141 seconds of test.sh (from the build to the end) under valgrind with
the options of CI, where each of the 10 processes of the two nginx writes
its leak report when it exits.
valgrind in CI
Section titled “valgrind in CI”The build matrix of .github/workflows/test.yml runs test.sh in each
cell. Which cells run nginx under valgrind depends on the run:
-
In a run that starts the
buildmatrix, every cell or the four standard cells (Which jobs a run starts), one cell runs nginx under valgrind: nginx 1.31.8, static module, without--with-openssl-src(the matrix gives itVALGRIND_ON_EVERY_RUN; in the whole matrix it is the jobbuild (1.31.8)). The other cells without a sanitizer run nginx directly (NGINX_RUNNER=exec), and the sanitizer cell runs nginx built with AddressSanitizer and UndefinedBehaviorSanitizer (see the next section). -
A manual run of
test.ymlwith the inputvalgrindruns every cell except the sanitizer cell under valgrind:$ gh workflow run test.yml --ref next -f valgrind=true -
The nightly run:
.github/workflows/valgrind-nightly.ymlstarts such a run oftest.ymlevery day at 18:23 UTC (03:23 JST), waits for it, and fails when it fails. GitHub runs a schedule only from the workflow file on the default branch,master, so the copy of the file onmaster(#674) starts the runs onmasterand onnext. Pull requests do not start it: a change oftest.ymlortest/valgrind.suppruns the cell under valgrind in its own run, and every cell under valgrind waits for the next night, or for “Run workflow” ofvalgrind-nightly.ymlor oftest.ymlwithvalgrindon the branch. The run does not give the status checkci-ok: its job is namedci-ok (valgrind)(Which jobs a run starts).
valgrind adds little time to a cell. The part of test.sh from the start
of nginx to test.sh ... successful took 67 to 94 seconds in the 16 cells
under valgrind of the run with the input valgrind
40824932917,
and 57 to 75 seconds in the 15 cells that ran nginx directly in the run of
the same commit for its pull request
40824899601
(2026-10-29). A cell takes 4 to 13 minutes in all, most of it for the
builds of OpenSSL, mruby and nginx.
A cell under valgrind runs nginx with these options:
valgrind --error-exitcode=99 --leak-check=full --show-leak-kinds=definite,indirect,possible --errors-for-leak-kinds=definite,indirect --suppressions=test/valgrind.supp --log-file=$RUNNER_TEMP/valgrind/valgrind.%p.logWhen test/ci/changes.sh gives build standard or full (a change
of the code or of the tests, as for the four standard cells), the author
runs the suite under valgrind with these options before marking the pull
request ready; the cell under valgrind repeats it on the x86-64 runner. Run it in a checkout
of its own (for example git archive HEAD | tar -x -C DIR), because a
sanitizer build in the same checkout leaves mruby built with other flags.
valgrind needs Linux; on macOS, run it in a Linux container, from the top
of that checkout:
$ docker run --rm --init -v "$PWD":/work -w /work ubuntu:22.04 sh -c 'apt-get -qq update && DEBIAN_FRONTEND=noninteractive apt-get -qq install -y build-essential rake ruby bison git gperf wget ca-certificates curl zlib1g-dev libpcre3-dev libssl-dev valgrind >/dev/null && mkdir -p build/valgrind && NGINX_RUNNER="valgrind --error-exitcode=99 --leak-check=full --show-leak-kinds=definite,indirect,possible --errors-for-leak-kinds=definite,indirect --suppressions=$PWD/test/valgrind.supp --log-file=$PWD/build/valgrind/valgrind.%p.log" sh test.sh && grep -h "ERROR SUMMARY" build/valgrind/valgrind.*.log | sed "s/^==[0-9]*== //" | sort | uniq -c'On Linux, run the part inside the quotes after the installation of the
packages. The last command prints each distinct ERROR SUMMARY line with
the number of logs that have it; every line has to count 0 errors, and
at least one log has to have the line, as the step “Check the valgrind
reports” below requires. --init makes a process of the container reap
the processes that the tests leave when they exit.
This form, and the form of the sanitizer section, build against the
OpenSSL of Ubuntu 22.04 (its libssl-dev, OpenSSL 3.0.2), while every
cell of CI removes libssl-dev and builds against OpenSSL 3.5.9 built
from source (Which jobs a run starts). The
cells cover that difference.
valgrind counts an error for each memory error that occurs (an invalid
read or write, a conditional jump or a system call on an uninitialised
value, an invalid or mismatched free) and for each loss record of the
kinds definite and indirect that test/valgrind.supp does not
suppress, and prints the count, with the number of distinct stacks
(contexts), in its ERROR SUMMARY line when nginx exits: an invalid read
in a handler that 20 requests ran gives 20 errors from 1 contexts. With
--error-exitcode=99, valgrind also exits with 99 when the count is not 0.
test.sh prints the exit status of nginx when it has stopped it
(test.sh: nginx (pid 374) exit status 99 after valgrind counted errors;
see Stopping nginx), but does not take it into its own,
so the step “Check the valgrind reports” reads the logs instead: it prints each log, and fails unless at least one
log has an ERROR SUMMARY and every ERROR SUMMARY counts 0 errors. A
cell without any ERROR SUMMARY fails as well: valgrind did not finish its
report.
Possibly lost blocks are printed with their stacks, but are not errors. At
the exit of the test nginx (master_process off), nginx has destroyed the
pool of the cycle, and the arrays of connections and events that
ngx_event_process_init() allocated, the table of ngx_crc32_table_init()
and the pools of the connections that are still open are left. valgrind
finds only interior pointers to them and reports them as possibly lost.
Their number depends on the connections that are open when nginx exits: in
the CI runs
40819115837,
40820233191
and
40821209339
of 2026-10-29, each of the 16 cells under valgrind reported from 5 blocks
(85,440 bytes) to 8 blocks (86,208 bytes), and the cell of nginx 1.31.6
with the static module gave 5, 7 and 7 blocks in these three runs. A leak
of ngx_mruby, mruby or a gem that drops the last pointer to a block is
definitely lost, and fails the cell. A request or connection that ngx_mruby
never finalizes keeps its pools until nginx exits, and valgrind reports
them as possibly lost, so it does not fail the cell. In the run
40824899601, the pool of an upstream connection that was still open at exit
is possibly lost, with ngx_mrb_post_fiber in the stack of its allocation.
The soak job, which compares the connections, timers and file descriptors
of the worker between samples (Soak test for
memory), is the check for such leaks.
test/valgrind.supp suppresses only allocations of nginx itself, each with
its stack and the reason in a comment: today the environment array of
ngx_set_environment() (8 bytes, definitely lost in every cell), which
nginx leaves allocated at exit on purpose (ngx_cleanup_environment() in
nginx.c), and the connection and event arrays that
ngx_event_process_init() allocates in a worker process (the nginx of
test/t/cases/_workers.rb, “Several workers and reloads” above). nginx
never frees these arrays; a worker that ends on SIGTERM destroys the cycle
pool that points at them and exits, and on the x86-64 runner valgrind
reports them as definitely and indirectly lost (14,848 and 2 x 6,144
bytes in each such worker, in the first CI run of #706), where the arm64
container reports them as possibly lost. That entry needs the frame
ngx_worker_process_init, so it does not cover the single process of the
test nginx. An entry is added only with the stack of the report and the
reason why the block is not ngx_mruby’s; a leak with a frame of ngx_mruby,
mruby or a gem is not suppressed.
Running the tests under AddressSanitizer and UndefinedBehaviorSanitizer
Section titled “Running the tests under AddressSanitizer and UndefinedBehaviorSanitizer”When test/ci/changes.sh gives build standard or full (a change
of the code or of the tests, as for the four standard cells), the author
runs the suite with -fsanitize=address,undefined before marking the
pull request ready; the sanitizer cell of the CI matrix builds with the
same flags and repeats it on the x86-64 runner (against OpenSSL 3.5.9,
as valgrind in CI says after its docker form). Run it in a checkout of its own (for example git archive HEAD | tar -x -C DIR): an existing mruby build is not rebuilt when only
the flags change. On Linux, with gcc or clang:
$ sudo sysctl -w vm.mmap_rnd_bits=28 # ASan and the kernel's high-entropy ASLR do not mix$ export NGINX_EXTRA_CC_OPT="-fsanitize=address,undefined -fno-omit-frame-pointer"$ export NGINX_EXTRA_LD_OPT="-fsanitize=address,undefined"$ export NGX_MRUBY_CFLAGS="-fsanitize=address,undefined -fno-omit-frame-pointer"$ export NGX_MRUBY_LDFLAGS="-fsanitize=address,undefined"$ export ASAN_OPTIONS="detect_leaks=1"$ export LSAN_OPTIONS="suppressions=$PWD/test/lsan.supp"$ export UBSAN_OPTIONS="print_stacktrace=1:suppressions=$PWD/test/ubsan.supp"$ NGINX_RUNNER=exec sh test.sh$ grep -HnE 'ERROR: (AddressSanitizer|LeakSanitizer)|runtime error:|SUMMARY: (AddressSanitizer|UndefinedBehaviorSanitizer)' $(find build/nginx/logs $(ls -d build/nginx/second_instance 2>/dev/null) -type f -name '*.log')On macOS, run it in a Linux container, from the top of that checkout.
The sysctl line is for the kernel that the container shares, which a
container cannot change. Read its value first:
$ docker run --rm ubuntu:22.04 cat /proc/sys/vm/mmap_rnd_bitsAt 28 or below, nothing is needed. Above 28, set it to 28 where that
kernel runs (the Linux VM of the container runtime), as the sanitizer
cell of CI does with sudo sysctl -w vm.mmap_rnd_bits=28 before it
builds: ASan’s shadow memory layout fails with a higher entropy.
$ docker run --rm --init -v "$PWD":/work -w /work ubuntu:22.04 sh -c 'apt-get -qq update && DEBIAN_FRONTEND=noninteractive apt-get -qq install -y build-essential rake ruby bison git gperf wget ca-certificates curl zlib1g-dev libpcre3-dev libssl-dev >/dev/null && export NGINX_EXTRA_CC_OPT="-fsanitize=address,undefined -fno-omit-frame-pointer" NGINX_EXTRA_LD_OPT="-fsanitize=address,undefined" NGX_MRUBY_CFLAGS="-fsanitize=address,undefined -fno-omit-frame-pointer" NGX_MRUBY_LDFLAGS="-fsanitize=address,undefined" ASAN_OPTIONS=detect_leaks=1 LSAN_OPTIONS=suppressions=$PWD/test/lsan.supp UBSAN_OPTIONS=print_stacktrace=1:suppressions=$PWD/test/ubsan.supp && NGINX_RUNNER=exec sh test.sh && ! grep -HnE "ERROR: (AddressSanitizer|LeakSanitizer)|runtime error:|SUMMARY: (AddressSanitizer|UndefinedBehaviorSanitizer)" $(find build/nginx/logs $(ls -d build/nginx/second_instance 2>/dev/null) -type f -name "*.log")'The command exits 0 when test.sh passed and the grep found no
report. The container uses gcc 11 of Ubuntu 22.04, the compiler of the
sanitizer cell.
The last line of the Linux commands reads the files that the “Check for sanitizer reports” step of the CI cell reads, with the same pattern; the step fails when a line matches. A report goes to the stderr of the process that wrote it, and that is one of these files:
| File | Whose stderr |
|---|---|
build/nginx/logs/error.log |
the test nginx, once it has opened its log (its leak report at exit included) |
build/nginx/logs/stderr.log |
the test client (test/t/ngx_mruby.rb and the case files; test.sh redirects it), and so the processes that the tests start without redirecting it: nginx -t, the nginx -s quit of the stream-only test, an nginx before it has opened its log |
the other build/nginx/logs/*.log |
the nginx processes that the tests start with a log of their own, after they have opened it (stream_only.error.log of the stream-only test, which runs nginx with daemon on; workers_MODE.error.log and workers_MODE.stderr.log of the master and the workers of test/t/cases/_workers.rb, the second from its start) |
build/nginx/second_instance/<mode>/*.log |
the second nginx of test/t/cases/_second_instance.rb (error.log, stderr.log, stdout.log) |
build/nginx/second_instance/ is missing when test.sh stopped before the
case files; the step then reads the other files and says so. The step does
not read the *.log.bak files, in which test.sh keeps the
logs of an earlier run in the same directory. A report of the test nginx
from before it opened error.log goes to the terminal (in CI, the output of
the script step), as the next paragraph says.
NGINX_EXTRA_CC_OPT and NGINX_EXTRA_LD_OPT are appended to nginx’s
--with-cc-opt and --with-ld-opt; NGX_MRUBY_CFLAGS and NGX_MRUBY_LDFLAGS
reach the mruby build. Do not combine the sanitizers with valgrind. nginx
sends its stderr to error.log once the configuration is loaded, so reports
from code that runs while nginx parses the configuration (directive handlers,
script compilation, mruby_init_code) appear on the terminal instead; read
both. test/ubsan.supp lists the reports that come from nginx itself, each
with the reason. test/lsan.supp does the same for LeakSanitizer: nginx
never frees cycle->connections and the two event arrays it allocates in
ngx_event_process_init, and whether LeakSanitizer reports them at exit
depends on whether anything outside them still points at them, so without
the suppression a run can fail on nginx’s own allocations. With the gcc 11
runtime that CI uses, a suppressed block hides nothing else; the LLVM
runtime and gcc 12 and later also treat a suppressed block as a root, so
there a block reachable only from those arrays is not reported either.
nginx’s own test suite (nginx-tests)
Section titled “nginx’s own test suite (nginx-tests)”sh test/nginx-tests/run.sh runs nginx’s own test suite,
nginx-tests, with prove on two
builds of the nginx version in nginx_version, and compares the results:
- nginx with ngx_mruby, built by
test.shwithTEST_SH_SKIP_TESTS=1: the static module, the configure options oftest.sh(see Testing) and-DMRB_GC_STRESS, installed tobuild/nginx. WithTEST_SH_SKIP_TESTS=1,test.shstops after the install: it writes no test configuration, kills no nginx and starts none.run.shunsetsBUILD_DYNAMIC_MODULE, which would build intobuild_dynamic/instead. - nginx without ngx_mruby: a copy of the same nginx source in
build_nginxtests/plain/src, configured with the options thattest.shgave nginx (NGX_CONFIG_OPTof the generatedMakefile) without the two--add-moduleoptions that theMakefileadds, for ngx_mruby and for ngx_devel_kit. Theconfigure argumentslines ofnginx -Vin the summary show both.
The test files write their own nginx configurations, and none has an mruby
directive. In the build with ngx_mruby, each start and reload of nginx
creates an mrb_state of ngx_mruby for each http {} and each stream {}
block while nginx reads the configuration (mrb_open, under
MRB_GC_STRESS), and mrb_open runs the Ruby code of mruby’s core and
of the bundled gems. Each HTTP request passes the handlers that ngx_mruby
adds to the post-read, server rewrite, rewrite, access and log phases, in
the phases that the request reaches (a return or a rewrite can end it
before the later ones), and the one of the content phase unless the
location has a content handler of its own (proxy_pass and the like),
which nginx runs instead of the handlers of that phase. These handlers decline when no
code is configured. Each stream session passes ngx_mruby’s handler of the
access phase. The header and body filters of ngx_mruby are added only when
a directive asks for them, and no handler, filter or init code of a
configuration runs. A test file that fails with ngx_mruby and passes
without it therefore shows that the module changes nginx’s behavior
although nothing is configured for it.
The pinned commit of nginx-tests
Section titled “The pinned commit of nginx-tests”nginx-tests has no releases or tags. run.sh fetches the commit in
NGINX_TESTS_COMMIT, ec44bc80c4bf (the head of master on 2026-10-19,
committed on 2026-10-16), so that a test added upstream does not change the
results without a change in this repository. A pull request of its own moves
the pin, and quotes the summaries before and after; a change of
nginx_version is a reason to move it, because new test files of
nginx-tests test the features of new nginx versions and skip themselves on
older ones.
Running nginx-tests
Section titled “Running nginx-tests”run.sh needs Linux, a user other than root (the test files write their
configurations to directories that only that user can read, and nginx
started by root runs its workers as nobody), and what test.sh needs.
The test files that need a perl module or a program that is missing are
skipped, not failed. On Ubuntu 22.04, the job installs:
$ sudo apt-get install rake bison git gperf zlib1g-dev libpcre3-dev libssl-dev ruby \ libio-socket-ssl-perl libnet-ssleay-perl libfcgi-perl libscgi-perl \ libcache-memcached-perl libcache-memcached-fast-perl libprotocol-websocket-perl memcached$ sh test/nginx-tests/self-test.sh$ sh test/nginx-tests/run.shOn macOS, run it in a Linux container on a copy of the tree, mounted at the same path:
D=$HOME/ngx_mruby-nginx-tests # any directory outside the checkoutrm -rf "$D" && mkdir -p "$D" && git archive HEAD | tar -x -C "$D"docker run --rm -v "$D:$D" -w "$D" ubuntu:22.04 sh -c 'apt-get -qq update && DEBIAN_FRONTEND=noninteractive apt-get -qq install -y build-essential perl rake ruby bison git gperf wget ca-certificates openssl zlib1g-dev libpcre3-dev libssl-dev libio-socket-ssl-perl libnet-ssleay-perl libfcgi-perl libscgi-perl libcache-memcached-perl libcache-memcached-fast-perl libprotocol-websocket-perl memcached >/dev/null && useradd -m nt && chown -R nt "$PWD" && su nt -c "NGX_MRUBY_COMMIT='"$(git rev-parse HEAD)"' sh test/nginx-tests/run.sh"'The test files start nginx on ports from 8000 to 8999 that they lock, so
they run in parallel (NGINX_TESTS_JOBS, by default the number of CPUs).
They do not use the ports of test.sh, and test.sh stops only the nginx
that it started (“Stopping nginx” above).
Environment variables: NGINX_TESTS_OUT (the output directory, by default
build_nginxtests), NGINX_TESTS_SRC (a checkout of nginx-tests at the
pinned commit, used instead of the fetch, for example when github.com is
unreachable), NGINX_TESTS_JOBS, NGINX_TESTS_FILES (test files to run
instead of all, for example NGINX_TESTS_FILES='access.t proxy.t') and
NGX_MRUBY_COMMIT (the ngx_mruby commit that the summary names; by
default GITHUB_SHA in GitHub Actions, else HEAD of the checkout). Extra
arguments of run.sh go to test.sh, and so to ./configure. The
variables of nginx-tests itself work as well: to read the error log of a
failed file, run it again alone with TEST_NGINX_LEAVE=1 (keeps its
temporary directory) or TEST_NGINX_CATLOG=1 (prints the log).
The output, in build_nginxtests/:
| File | Content |
|---|---|
summary.md |
The counts of both builds, the failed files by build, the reasons of the skips, the configure arguments of both builds |
only-mruby.txt, both.txt, only-plain.txt |
The test files that failed with ngx_mruby only, on both builds, and without ngx_mruby only |
mruby/, plain/ |
For each build: prove.log (the output of prove), results.tsv (a line for each test file: the file, passed, failed or skipped, and the reason of a skip or the failed subtests), nginx-V.txt |
test/nginx-tests/results.pl reads prove.log: a file fails when its
entry in prove’s “Test Summary Report” has failed subtests, a wait status
other than 0 or parse errors (a missing or wrong plan), and a file whose
entry has only TODO subtests that passed has passed. It stops when the log
is not a complete run of prove. test/nginx-tests/compare.pl writes the
lists and the summary. sh test/nginx-tests/self-test.sh checks both on
the output of prove for the small test files in
test/nginx-tests/self-test/, which pass, fail, skip or break their plan
on purpose, differently for the two builds (the table at the top of
self-test.sh).
run.sh exits with 0 when both builds were tested and their results read,
also when test files failed. It exits with another status when a build,
the fetch or prove itself failed, or when a log is not a complete run.
nginx-tests in CI
Section titled “nginx-tests in CI”.github/workflows/nginx-tests.yml runs the self-test and run.sh on
ubuntu-22.04, on pull requests to next that change an input of the
nginx binary (the inputs that “In CI” of the performance
comparison lists), test.sh, test/nginx-tests/ or the workflow, and on
“Run workflow”. A draft runs nothing, and marking it ready for review
starts the run. Pushes to next do not run it: the run of the pull request
on its merge commit is the record of the merge. The job summary is summary.md, each
file that fails with ngx_mruby only is a warning annotation, and the
artifact nginx-tests has the output above without the copy of the nginx
source.
The job records; it does not gate. It fails only when the self-test, a
build, the fetch or prove itself fails, or summary.md is missing, not
on the result of a test file, and ci-ok does not need it (see
Required status check). Pillar E of
docs/proposals/v3-plan.md makes it required once its results are stable.
Results of nginx-tests
Section titled “Results of nginx-tests”Measured on 2026-10-19, with nginx-tests at ec44bc80c4bf, nginx 1.31.6,
gcc 11.4.0, OpenSSL 3.0.2 and perl 5.34 on both machines:
- CI: the
nginx-testsjob of run 39241352263 onubuntu-22.04(x86_64, 4 parallel test files), on the merge commit401cf56of #602 (093e47d) intonextat7b5a0c4. - A Docker container of
ubuntu:22.04on macOS (aarch64, 11 parallel test files), with the command of Running nginx-tests, on093e47d.
| Machine | Build | Test files | Passed | Failed | Skipped | Subtests run |
|---|---|---|---|---|---|---|
| CI (x86_64) | nginx with ngx_mruby | 511 | 302 | 0 | 209 | 3994 |
| CI (x86_64) | nginx without ngx_mruby | 511 | 302 | 0 | 209 | 3994 |
| Container (aarch64) | nginx with ngx_mruby | 511 | 301 | 0 | 210 | 3991 |
| Container (aarch64) | nginx without ngx_mruby | 511 | 301 | 0 | 210 | 3991 |
- Test files that fail because ngx_mruby is in nginx (they fail with ngx_mruby and pass without it): none, on both machines.
- Test files that fail on nginx without ngx_mruby too: none, on both machines.
The two machines differ in one file, ssl_ktls.t: it passes on the runner
and skips in the container (“no Linux KTLS”), whose kernel has no kTLS. On
each machine, the same files were skipped on both builds, for the same
reasons. Of the 209 on the runner, 189 skip because nginx lacks a module
that test.sh does not build: http_v2 (59 files), stream_ssl (25),
http_v3 (23), mail (16), realip (11), sub (9), and 18 other modules
with 1 to 6 files each. The other 20 skip for conditions of the machine or
of nginx-tests: a program or perl module that is not installed (uwsgi for
4 files, GD, Win32API::File, a newer IO::Socket::SSL), OpenSSL 3.5,
root (3 files), a wildcard address (2), Windows (1), and 6 files that
nginx-tests runs only with TEST_NGINX_UNSAFE or not at all (an orphaned
process group, a long test, keys of an OpenSSL engine or provider). The
summary of each run lists every reason with its count. On the runner, the
job took about 8 minutes: 2 minutes 37 seconds for the two builds and 2
minutes 37 seconds for prove on each build.
These results cover nginx with ngx_mruby present and nothing configured for
it. Two kinds of runs would cover more and are not in place: a build with
the modules that the skipped files need (--with-http_v2_module,
--with-http_v3_module, --with-stream_ssl_module, --with-mail and the
others above), and runs in which ngx_mruby’s handlers or filters run Ruby
code on the requests of the suite (TEST_NGINX_GLOBALS_HTTP, as Pillar E
of docs/proposals/v3-plan.md names). docs/HANDOFF.md lists both in its
queue.
Mock LLM upstream
Section titled “Mock LLM upstream”test/soak/mock_llm.rb (CRuby 3.0 or later) is an HTTP/1.1 server that
answers POST /v1/messages the way the Anthropic Messages API does and
POST /v1/responses the way the OpenAI Responses API does. It stands for
the upstream of an agent proxy (nginx with ngx_mruby in front of an LLM
API) in three places:
- the soak test (
agent_*scenarios) and the performance comparison (proxy_*,responses_stream_plain_*,auth,route_json_*), which start it as a process of its own on the port base + 2 (12362 and 12372) for each scenario, with the options of the scenario (mockintest/soak/scenarios.rb); - the case
test/t/cases/agent_proxy.rb, whose helpertest/t/cases/_agent_proxy_client.rbruns it in the helper’s process on 12372 and 12373 while it sends its requests. The case checks recipes of an agent proxy that work with the current build, on the server oftest/conf/conf.d/60-agent-proxy.conf(port 18132): a server rewrite handler that rejects an unknown client key with 401 and a JSON body from a server-levelerror_page(and the request does not reach the upstream); an access handler that picks one of two upstream blocks by themodelof the JSON body and replaces the client’s credential;limit_connper client key, whose rejection becomes a JSON 429 withretry-afterthroughlimit_conn_status 460anderror_page, and whose slot is free again after the client closed its stream; an upstream 529 passed to the client unchanged; and a log handler that reads$upstream_status. The same recipes run onlocation = /v1/responseswith Responses requests, where the proxy’s own 401 and 429 bodies use the error format of the Responses API and the upstream errors are a 503 and a 429. The case also checks that a stream of 50 deltas arrives byte for byte: one Messages stream, and one Responses stream for each terminal event.
$ ruby test/soak/mock_llm.rb --port 12362 --events 50 --hold-after 10| Option | Default | Meaning |
|---|---|---|
--port PORT |
(required) | listen on 127.0.0.1:PORT; repeat it for more ports, which answer the same way |
--events N |
50 | text delta events in a stream: content_block_delta (Messages), response.output_text.delta (Responses) |
--delta-text TEXT |
hello |
the text of each text delta event |
--hold-after K |
off | write the first K events of a stream, then nothing more until the other side closes the connection |
--reset-after K |
off | write the first K events of a stream, then reset the connection (RST) |
--status CODE |
off | answer every request with this status and an error body of its API, before any event: on /v1/messages an Anthropic error (overloaded_error for 529, api_error for 503), on /v1/responses an OpenAI error (rate_limit_error for 429, service_unavailable_error for 503, invalid_request_error for the other 4xx, server_error for the other 5xx) |
--delay-ms MS |
0 | wait MS milliseconds between two events |
--one-write |
off | write the head and the whole stream with one write |
--terminal NAME |
completed |
the event that ends a Responses stream: completed, incomplete or failed |
A request can override the options for itself with the headers
x-mock-events, x-mock-hold-after, x-mock-reset-after, x-mock-status
and x-mock-delay-ms, each a number of 0 or more, and x-mock-terminal,
one of completed, incomplete and failed. x-mock-hold-after,
x-mock-reset-after and x-mock-status also take off, which turns the
hold, the reset or the error status off; x-mock-events and
x-mock-delay-ms do not. Any other value gets a 400 answer
(invalid_request_error) that names the header. nginx forwards the headers
unless the configuration drops them. ruby test/soak/mock_llm_test.rb checks
these rules and the answers below against the mock in its own process (no
nginx, about a second); test/soak/run.sh runs it before the soak.
What it answers on POST /v1/messages:
- A JSON body with
"stream": true: 200,text/event-stream, chunked. The events aremessage_start(withusage),content_block_start, Ncontent_block_deltaevents with the text,content_block_stop,message_delta(withusage) andmessage_stop: N + 5 events. - Any other JSON object: 200 with a message as JSON (
Content-Length).
On POST /v1/responses, with the event types and fields of openapi.yaml
in openai/openai-openapi (read
at commit 33248a6, 2026-10-19):
- A JSON body with
"stream": true: 200,text/event-stream, chunked. Each event has anevent:line with its type and adata:line whose JSON has the sametypeand asequence_numberthat counts from 0. The events areresponse.createdandresponse.in_progress(theResponseobject in statusin_progress,usagenull),response.output_item.added,response.content_part.addedand Nresponse.output_text.deltaevents with the text. For the terminal eventcompletedorincompletefollowresponse.output_text.done,response.content_part.done,response.output_item.doneandresponse.completedorresponse.incomplete, which carries the wholeResponseobject with the output text andusage(input_tokens,input_tokens_details,output_tokens,output_tokens_details,total_tokens): N + 8 events. Forfailed,response.failedfollows the deltas, with statusfailed, anerrorandusagenull: N + 5 events. - Any other JSON object: 200 with a
Responseobject as JSON (statuscompleted,Content-Length).
On both paths, without --one-write, each event is one chunk written with
one write; the response head goes with the first event and the last chunk
with the last event. A body that is not a JSON object gets a 400 with an
invalid_request_error in the error format of the API. Another path gets
a 404 (Anthropic format), and GET /_mock/stats its counters as JSON
(below).
The answer depends only on the request and the options: the model is the
request’s model, input_tokens is the request body’s size in bytes / 4
(rounded up), output_tokens is N (6 for an answer without a stream), the
ids and times of a Response are fixed, and there are no sleeps but
--delay-ms. The drivers and the case compute the expected response with
the same code (MockLLM.stream_body, MockLLM.answer_body)
and compare it byte for byte. Responses also carry x-mock-port (the port
that answered) and, when the request had them, x-mock-authorization and
x-mock-x-api-key with the values that arrived, so that a test can see
which credential the proxy sent.
The mock never closes an idle keep-alive connection. It closes one only when
the request has Connection: close (the answer then has it too), when the
other side closes it, and for --hold-after and --reset-after. The
counters of GET /_mock/stats:
| Counter | Meaning |
|---|---|
connections_open |
connections open now, without the one asking |
connections |
connections accepted |
requests |
POST /v1/messages and POST /v1/responses requests read |
completed |
answers written in full (error statuses included) |
held |
streams waiting in --hold-after now |
held_closed |
streams in --hold-after that the other side closed |
reset |
connections reset by --reset-after |
write_failed |
answers cut short because the other side had closed the connection |
Soak test for memory
Section titled “Soak test for memory”An nginx worker serves requests for days. Anything that a finished request
leaves behind in the worker (a GC root, a live mruby object, a timer, a file
descriptor, a connection, heap memory) adds up over the life of the worker.
The soak test sends many requests to one location per scenario and checks
that none of these grows. When test/ci/changes.sh gives soak true,
the author runs it locally before marking the pull request ready
(Running it), and the soak job of CI runs it again on
the x86-64 runner.
What it measures
Section titled “What it measures”test/soak/soak.rb starts nginx (master_process on, one worker) for each
scenario, checks one response, sends SOAK_WARMUP requests, and then sends
SOAK_N requests in three windows of SOAK_N / 3. It takes a sample after
the warmup and after each window, four samples in all. Before each sample it
closes all of its connections and waits (up to 5 seconds) until stub_status
shows that the only open connection is the one asking for it. A sample is:
| Column | Source | Pass condition |
|---|---|---|
gc_live |
Nginx::Debug.stats (below), read after Nginx::Debug.gc |
same in all samples |
gc_root |
same | same in all samples |
fibers |
gc_root_fibers of the same |
same in all samples |
arena |
gc_arena_idx of the same |
same in all samples |
timers |
timers of the same |
0 in all samples |
stream_timers, stream_fibers |
stream counters of the same | 0 in all samples |
sessions |
completed TCP data sessions in stream_sleep |
first check + warmup + completed measurement sessions |
fd |
entries in /proc/<worker>/fd |
fd minus up the same in all samples |
up |
connections_open of the mock LLM (0 in a scenario without one) |
at most SOAK_CONCURRENCY; 0 in the abort and cut scenarios |
active |
Active connections of stub_status |
1 in all samples |
rss_kb |
VmRSS in /proc/<worker>/status |
last sample minus the one before it at most SOAK_RSS_STEP_KB; last sample minus the first at most SOAK_RSS_TOTAL_KB |
The worker is in the same state at every sample: every request sent before it
has finished, and the only request running is /debug/stats itself. The
counters therefore do not depend on how many requests ran before:
gc_liveis the number of live mruby objects.Nginx::Debug.gcruns a full GC first, so the objects of finished requests are already freed.gc_rootis the number of objects that ngx_mruby keeps from the GC, andfiberscounts the fibers among them: the holder of each mrb_state (src/http/ngx_http_mruby_gc_roots.h), registered once when the state is opened, and the objects stored in it. The compiled code of each directive stays in a slot of the holder until the state is closed, and a handler file withoutcacheputs the code of each run in the same slot (src/common/ngx_mruby_code.h). A handler stores its fiber in the holder when it starts and frees the slot when it ends, so the holders, the compiled code and the fiber of/debug/statsare left. ngx_mruby counts its own calls (see “The soak build andNginx::Debug” below), so the check does not depend on how mruby stores the GC roots.arenais the depth of the mruby GC arena while/debug/statsruns. Each code path that saves the arena restores it.timerscounts theNginx::Async.sleeptimers that have neither fired nor been deleted. No request is waiting, so there are none.fdis read after the/debug/statsconnection closed, so it counts the listening sockets, the log files and nginx’s own descriptors.activecomes from the/statusrequest, the only open connection at that time. A socket or file that a request leaves open adds one to them.upis the number of connections open at the mock LLM, which nginx’s upstream keeps for the next requests. Their number is the number of upstream connections that were in use at the same time at the busiest moment so far: it can grow by one in a later window that happens to be busier, and each of them is also infd. The soak therefore comparesfdminusupbetween samples, and boundsupby the number of client connections. In theabortandcutscenarios no response ends normally, so nginx keeps no upstream connection: the soak waits (up to 5 seconds) untilupis 0 and fails the sample otherwise.
VmRSS is not exact, so it is bounded instead of compared. The soak build
calls malloc_trim(0) in the full GC before each sample (see below), which
returns the free pages of the C heap; without it, VmRSS moved by up to about
1 MB in both directions between samples of a correct build, depending on
where the allocator happened to leave free memory.
A scenario of test/soak/nginx.conf also checks Nginx::Debug itself, once,
after its first response and before the warmup. It requests /debug/check,
whose code runs Nginx::Debug.gc, reads Nginx::Debug.stats, makes two
Strings that nothing references in each of the 1000 rounds of a loop, and reads
gc_live again before and after a second Nginx::Debug.gc. The scenario
fails unless Nginx::Debug.stats has the keys listed in “The soak build and
Nginx::Debug” below (with gc_root_mruby and gc_root_fibers_mruby
either both present or both absent) and only Integer values,
Nginx::Debug.gc returns nil, gc_live grew by 1000 or more with the
Strings, and the second full GC brought it down by 1000 or more, to at most
10 above the first read. The code makes the Strings between GC.disable
and GC.enable: with about 1000 live objects after a full GC, mruby’s
incremental GC would otherwise start within the loop and free most of the
Strings before the second read. Without
this check, a Nginx::Debug.stats that left out a key, or a
Nginx::Debug.gc that freed nothing, would make every sample the same and
the soak would pass.
A scenario also fails when a response is not the expected one, when nginx does
not exit within 30 seconds of SIGQUIT, or when its error.log or stderr
contains open socket, [error], [alert], [crit], [emerg],
runtime error: or Sanitizer. ngx_mruby logs an exception raised in a
handler at the error level, and the disconnect client reads only its first
response, so the log is where an exception in its later requests shows. A
scenario can list lines that nginx logs at the error level on purpose
(allowed_log in test/soak/scenarios.rb); they are left out before the
check. The exit status is 1 when any scenario fails.
The default scenarios, each one location in test/soak/nginx.conf:
| Scenario | Handler | Client |
|---|---|---|
hello |
Nginx.rputs |
keep-alive |
headers |
reads 3 request headers, sets 2 response headers | keep-alive |
var |
reads 2 variables with Nginx::Var, sets 1 |
keep-alive |
filter |
mruby_output_body_filter_code rewrites a return 200 body |
keep-alive |
sleep |
Nginx::Async.sleep 1, then the body |
keep-alive |
sub_request |
Nginx::Async::HTTP.sub_request to a location that proxies to the static location of a second server (port base + 1) |
keep-alive |
file |
mruby_content_handler with a file and cache |
keep-alive |
disconnect |
Nginx::Async.sleep 50, then the body |
one request per connection, closed without reading the response |
and four agent proxy scenarios, each in /v1/route or
/v1/responses/route of test/soak/nginx.agent.conf (see “Agent proxy
scenarios” below) in front of the mock LLM:
| Scenario | Mock | Client |
|---|---|---|
agent_stream |
streams of 50 content_block_delta events (55 events) |
keep-alive; every stream is compared byte for byte |
agent_client_abort |
writes 10 events, then holds the stream until nginx closes the connection | one request per connection: reads the head and the first 3 events, checks them, closes |
agent_upstream_reset |
writes 10 events, then resets the connection | one request per connection: reads to EOF and checks that the body is exactly the 10 events, without the end of the chunked body |
agent_responses_stream |
Responses streams of 50 response.output_text.delta events and response.completed (58 events), through /v1/responses/route |
keep-alive; every stream is compared byte for byte |
Keep-alive clients are SOAK_CONCURRENCY connections, each sending its next
request after reading the response; every response is checked. The
disconnect client sends bursts of 16 requests per thread every 100 ms, which
keeps the number of requests open at once the same in every window. The
agent_client_abort (mode :abort) and agent_upstream_reset (mode
:cut) clients are SOAK_CONCURRENCY threads, each sending one request per
connection and checking what arrives.
Agent proxy scenarios
Section titled “Agent proxy scenarios”test/soak/nginx.agent.conf is a configuration of its own, for the soak
test and the performance comparison. It has no mruby filter directive: one
body filter directive anywhere in an http {} block installs ngx_mruby’s
body filter for every response of that block (as the filter scenario does
in test/soak/nginx.conf), and each chunked response, such as a stream,
would then pass it and log a notice. Its error_log is at warn. The
proxy settings are those of an agent proxy: proxy_buffering off,
proxy_buffer_size 4k, proxy_http_version 1.1, the client’s
Authorization replaced by the credential that a handler picked, and an
upstream block with keepalive 32, keepalive_requests 1000000 and
keepalive_timeout 1h, so that nginx does not close idle connections to
the mock during a scenario either. The handlers are in test/soak/handlers/:
agent_init.rb(mruby_init_worker): the client key, credential and model tables;agent_auth.rb(server rewrite handler of the serverauth.agent.test): looks up the client key fromAuthorizationorx-api-key(two Hash lookups) and sets three variables;agent_route.rb(access handler of/v1/routeand/v1/responses/route, withmruby_enable_read_request_body on):JSON.parse(r.body)["model"], one Hash lookup, and$agent_backendforproxy_pass http://$agent_backend.
/v1/plain and /v1/route proxy to the mock’s /v1/messages, and
/v1/responses/plain and /v1/responses/route to its /v1/responses.
The request bodies are Messages requests of exactly 2 KB or 64 KB, and
Responses requests of exactly 2 KB for the responses_* and
agent_responses_* scenarios (MockLLM.request_body, with api: :responses); "model" is the first key of each, and the bodies of the
streams have "stream": true. The abort and reset scenarios have no
Responses counterpart: nginx relays the bytes of either API the same way.
In each agent_* scenario the soak starts the mock (port base + 2) before
nginx, and after the last sample checks its counters: it read exactly the
requests that the soak sent (1 + SOAK_WARMUP + SOAK_N), and every one of
them ended as the scenario makes it end: written in full (agent_stream),
held until nginx closed the connection or found closed while the mock was
still writing (agent_client_abort), or reset (agent_upstream_reset).
The mock’s output goes to logs/mock.<scenario>.log.
agent_upstream_reset allows two lines at the error level, both about the
upstream connection of /v1/messages at the mock: recv() failed (104: Connection reset by peer) (nginx logs errors of upstream connections at the
error level, and those of client connections at info), and upstream prematurely closed connection, which nginx logs for an upstream that closes
in the middle of a response in unbuffered mode. In the runs so far only the
first appeared, with “while reading upstream” or “while sending to client”
as the action. agent_client_abort logs nothing at warn or above.
TCP stream sleep
Section titled “TCP stream sleep”stream_sleep is part of the default soak and performance comparisons.
It opens one TCP connection per session, waits 1 ms in
Nginx::Stream::Async.sleep, receives exactly stream slept, and reads EOF.
The shared client bounds the whole exchange at 30 seconds in the soak and
120 seconds under callgrind. It rejects short,
extra or reset responses. It does not use HTTP framing or a persistent session.
test/soak/nginx.stream.conf uses the existing three-port range: the base
port is HTTP control, base + 1 is the TCP scenario, and base + 2 is stream GC
control. This scenario has no HTTP backend or model API mock. The soak’s
sessions column counts completed data sessions, including the first check
and warmup; requests still counts HTTP control requests. Other scenarios
show - in sessions.
Before warmup, the soak observes one stream_timers and one stream_fibers
while a control session waits for 250 ms. That session runs GC.start in the
same stream VM and returns stream gc complete through EOF. Both counters
must return to zero. Before each memory sample, the driver waits for zero,
runs that GC control session and waits for zero again. Every sample requires
both counters to be nonnegative Integers and zero; missing values fail.
Their columns are also printed for the HTTP scenarios.
HTTP and stream have separate mruby states. Nginx::Debug.gc and gc_live
on the HTTP control endpoint describe the HTTP state; they do not collect or
count the stream VM’s live objects. The stream control session collects that
VM before the worker RSS sample. The existing RSS thresholds remain unchanged.
The performance driver never calls the stream GC control server. Its
stream_sleep row divides instructions by completed TCP sessions, and the
window must contain exactly PERF_N calls to ngx_stream_mrb_start_fiber.
HTTP control wakes do not count as stream sessions. The existing report/JSON
field names retain request, but their unit is a session for this row.
The report header names both units; each JSON result adds unit (session
or request).
ruby_calls includes HTTP and stream fiber starts. GC exclusion and the
3% warning / 5% failure thresholds are unchanged.
test/build_release.sh builds both measured checkouts with --with-stream.
After updating it, run the soak and comparison without ONLY_RUN: the build
stamp detects the changed configure options and rebuilds. Production builds
and the ordinary integration-suite configuration are unchanged. The scripts
run ruby test/soak/tcp_client_test.rb before building; its local peers check
response bytes, EOF, timeout, reset and required counter values without nginx.
The soak build and Nginx::Debug
Section titled “The soak build and Nginx::Debug”test/soak/run.sh builds with test/build_release.sh in build_soak/ (see
“Release builds” below): nginx and ngx_mruby with -O2 -g, without
MRB_GC_STRESS and without --with-debug, and with the gems of
build_config.rb at the commits of build_config.rb.lock. It adds two
defines. ngx_mruby is compiled with
-DNGX_MRUBY_DEBUG_STATS, and mruby with -DMRB_USE_MALLOC_TRIM, so
mrb_full_gc() also calls malloc_trim(0). mruby runs a full GC only when
it is asked to (GC.start, Nginx::Debug.gc) or in rare cases (an
incremental GC that finds too many objects, a failed allocation), so the
worker behaves as without it between samples.
-DNGX_MRUBY_DEBUG_STATS adds the class Nginx::Debug. The default build
of build.sh does not have it. test.sh defines it for the integration suite,
including the repeated HTTP sleeps described above.
Nginx::Debug.gcruns a full GC (mrb_full_gc) and returnsnil.Nginx::Debug.statsreturns a Hash with these Symbol keys:gc_live:mrb->gc.live.gc_root: the objects that ngx_mruby keeps from the GC, one per call, in the whole nginx process: the objects registered withmrb_gc_register()and not yet unregistered (the holder of each mrb_state, seesrc/http/ngx_http_mruby_gc_roots.h), and the objects thatngx_mrb_gc_roots_add()stored in a holder andngx_mrb_gc_roots_remove()has not released, among them the compiled code of each directive, which stays in its slot until the state is closed (src/common/ngx_mruby_code.h). Those of the http module and those of the stream module are counted together. Immediate values are not counted, because mruby does not register them. The value is negative when ngx_mruby released more than it kept.gc_root_fibers: the fibers among the objects counted ingc_root.gc_arena_idx:mrb->gc.arena_idx.timers: theNginx::Async.sleeptimers that have neither fired nor been deleted.stream_timers: the same forNginx::Stream::Async.sleep, in the stream module.stream_fibers: the fibers of stream handlers that are kept in the GC root holder of the stream module’s mrb_state: those that slept and have neither ended nor had their session end. They are also counted ingc_root_fibers. Thestream_sleepscenario observes them while a session waits and requires zero after it ends. Every soak sample checks both stream counters and prints them in its table.gc_root_mrubyandgc_root_fibers_mruby: the length of mruby’s own_gc_root_array in the mrb_state of the http module, and the fibers in it. mruby 3.x and 4.0 create that array at the firstmrb_gc_register()of the mrb_state; mruby 4.1 keeps the GC roots inmrb->gc.rootinstead, and the two keys are then left out. Unlikegc_root, they include the objects that mruby and the gems register, and they do not include the objects in the holders: the array has the holder of the http module, not the fibers in it. The soak test does not check them: they are there to compare withgc_rooton the mruby versions that have the array.
gc_root and gc_root_fibers do not read the GC roots from mruby, because
mruby 4.1 no longer keeps them in the _gc_root_ array: a count read from
the array would be 0 in every sample, and the soak would pass without
checking anything. ngx_mruby stores its fibers in the holder of the
mrb_state, which src/http/ngx_http_mruby_core.c counts in
ngx_mrb_gc_roots_add() and ngx_mrb_gc_roots_remove(). With
-DNGX_MRUBY_DEBUG_STATS, src/http/ngx_http_mruby_debug.h defines
mrb_gc_register and
mrb_gc_unregister as macros that call the mruby function and then count
the call, so every call in a source file that includes the header is
counted, including a call added later. A .c file in src/http/,
src/stream/ or src/common/ that calls either function must therefore include
ngx_http_mruby_debug.h; test/soak/run.sh checks this before it builds
and stops otherwise. Without the define, the header defines no macro and the
calls go to mruby directly.
Running it
Section titled “Running it”The driver reads /proc and needs Linux and CRuby 3.0 or later. On other
systems, run it in a container, for example:
$ docker run --rm -v "$PWD":/work -w /work ubuntu:22.04 sh -c 'apt-get -qq update && DEBIAN_FRONTEND=noninteractive apt-get -qq install -y build-essential rake ruby bison git gperf wget ca-certificates zlib1g-dev libpcre3-dev libssl-dev >/dev/null && sh test/soak/run.sh'On Linux:
$ sh test/soak/run.sh # build, then run the default scenarios$ ONLY_RUN=1 sh test/soak/run.sh # run without building$ ONLY_RUN=1 SOAK_SCENARIOS=sleep,disconnect SOAK_N=100000 sh test/soak/run.shThe first run takes a few minutes (it clones the gems and builds mruby and
nginx); the scenarios themselves take about a minute (the agent_* ones
about 18 seconds of it, on the aarch64 container). The table has one row
per scenario; when a value differs between samples, all four values are shown,
separated by /, and the reasons are listed below the table. The logs are in
build_soak/nginx/soak/logs/: error.<scenario>.log,
stderr.<scenario>.log, mock.<scenario>.log (the mock LLM’s output, for
the agent_* scenarios) and soak.log (the output of the driver).
test.sh stops only the nginx that it started (“Stopping nginx” above),
and its ports are not those of the soak.
| Variable | Default | Meaning |
|---|---|---|
ONLY_RUN |
unset | set to skip the build |
NUM_THREADS_ENV |
half of the CPUs | build parallelism (passed to build.sh) |
SOAK_SCENARIOS |
the default set | comma-separated scenario names |
SOAK_N |
20000 | requests per scenario after the warmup |
SOAK_WARMUP |
2000 | requests before the first sample |
SOAK_CONCURRENCY |
8 | connections (threads, for disconnect) |
SOAK_PORT_BASE |
12360 | port of the scenarios; the backend of sub_request uses the next one, the mock LLM the one after |
SOAK_RSS_STEP_KB |
512 | VmRSS limit for the last window, in kB |
SOAK_RSS_TOTAL_KB |
1024 | VmRSS limit from the first to the last sample, in kB |
Thresholds and calibration
Section titled “Thresholds and calibration”The two VmRSS limits were set from runs of the default scenarios on
2026-10-03 with nginx 1.31.6 on Ubuntu 22.04 (gcc 11, glibc 2.35): in a
container on aarch64, and in the CI soak job on x86_64. The values are the
largest over all scenarios and runs of a row:
| Runs | Growth in the last window | Growth from the first sample |
|---|---|---|
aarch64, 3 runs, SOAK_N=20000, without MRB_USE_MALLOC_TRIM |
+144 kB | +212 kB (drops down to -1076 kB) |
aarch64, 1 run, SOAK_N=100000, without MRB_USE_MALLOC_TRIM |
+288 kB | +3396 kB (disconnect) |
aarch64, 10 runs, SOAK_N=20000 |
+112 kB (sub_request) |
+204 kB (sleep) |
aarch64, 1 run, SOAK_N=100000 |
+68 kB (file) |
+160 kB (file) |
x86_64 (CI), 4 runs, SOAK_N=20000 |
+144 kB (filter) |
+236 kB (sleep) |
aarch64, 3 runs, SOAK_N=20000, the agent_* scenarios (2026-10-04) |
+28 kB (agent_stream) |
+32 kB (agent_upstream_reset) |
x86_64 (CI), runs 37152930938 and 37156005549, SOAK_N=20000, the agent_* scenarios (2026-10-04) |
+16 kB (agent_stream, agent_client_abort) |
+84 kB (agent_stream, agent_client_abort) |
Without malloc_trim(0), the disconnect scenario at SOAK_N=300000 moved
by +2832, -788 and +1088 kB from window to window, which is over a 512 kB
limit for the last window, although its counters stayed the same and it grew
less in all than at SOAK_N=100000. With it, the largest growth is 144 kB in
the last window and 236 kB in all. The default limits, 512 kB for the last
window and 1024 kB in all, are about 3.5 and 4.3 times these. The limit for
the last window is the looser one because one window of a shared CI runner
grew by 144 kB; the limit in all is the one that finds small leaks: a leak
below about 50 bytes per request stays under 1024 kB over the 20000 requests
of the default SOAK_N, so raise SOAK_N to look for smaller ones. For
comparison, a build that leaks 4 kB per request grows by about 26 MB per
window.
Adding a scenario
Section titled “Adding a scenario”- Add a location to
test/soak/nginx.conf(and a handler file totest/soak/handlers/if it uses one). Use__SOAK_HANDLERS__for the path of that directory and__SOAK_BACKEND_PORT__for the second server. A scenario behind the mock LLM goes totest/soak/nginx.agent.confinstead, with__SOAK_MOCK_PORT__for the mock. - Add a
ScenariotoSCENARIOSintest/soak/scenarios.rbwith the path, the request headers, the expected body and response headers, and the mode (:keepalive,:disconnect,:abortor:cut). A scenario of the agent template also setsconf: 'nginx.agent.conf', the request method and body, andmock:with the options of the mock ([]for the defaults); the comment at the top of that file lists the fields. - Add its name to
DEFAULT_SCENARIOSintest/soak/soak.rb, run the soak three times, and check that all counters stay the same and that VmRSS stays within the limits. If a counter keeps growing, the scenario has found a problem: do not raise a limit or drop the comparison to make it pass. Report it as described inSECURITY.mdwhen it may be a vulnerability. - A
:keepalivescenario can also be measured by the performance comparison (below): add its name toDEFAULT_SCENARIOSintest/perf/perf.rband run the null comparison of “Thresholds and calibration” there.
Release builds
Section titled “Release builds”sh test/build_release.sh SOURCE_DIR BUILD_DIR builds nginx with ngx_mruby
as a static module with release-like options: nginx and ngx_mruby with
-O2 -g, mruby with the flags of its gcc toolchain (-O3 -g), without
MRB_GC_STRESS (which test.sh adds) and without --with-debug, and with
the gems of build_config.rb at the commits of a gem lock (below). The soak
test and the performance comparison
build with it, each in directories of their own, so that their options do
not mix:
| Build | Directory | Options added |
|---|---|---|
soak (test/soak/run.sh) |
build_soak/ |
-DNGX_MRUBY_DEBUG_STATS for ngx_mruby (RELEASE_CC_OPT), -DMRB_USE_MALLOC_TRIM for mruby (NGX_MRUBY_CFLAGS) |
perf (test/perf/run.sh, test/perf/compare.sh) |
build_perf/head/, build_perf/base/ |
none: both defines add code to the process under measurement |
SOURCE_DIR is the checkout to build, and its own build files are used
(build.sh, configure, build_config.rb, …), so the base of a pull
request is built the way it builds itself. The script copies the sources to
BUILD_DIR/tree, runs build.sh there and installs nginx to
BUILD_DIR/nginx, so the build/ directory, mruby/build and the generated
Makefile, config and mrbgems_config of test.sh are not touched. The
build is always a static module built from its own nginx source with the
system OpenSSL, so the script ignores BUILD_DYNAMIC_MODULE, NGINX_SRC_ENV
and OPENSSL_SRC_VERSION. The nginx source is downloaded to
BUILD_DIR/tree/build/; when nginx.org is unreachable, put it there as
described in AGENTS.md.
The third-party gems are built at the commits of the build_config.rb.lock
committed in SOURCE_DIR (see “Gem commits” in docs/install/README.md), or
of the lock that RELEASE_GEM_LOCK names when it is set and not empty.
test/soak/run.sh and test/perf/run.sh pass RELEASE_GEM_LOCK on from
the environment, so RELEASE_GEM_LOCK=/path/to/other.lock sh test/soak/run.sh builds other commits without a change to the checkout. A
checkout that has no lock (one from before the lock was committed) gets the
head of each gem’s branch on its first build. rake writes the commits it
built to BUILD_DIR/tree/build_config.rb.lock.
Later runs update the copy and rebuild only what changed:
- A file is written to the copy only when its content differs, and it gets
the time of the copy, not that of
SOURCE_DIR. A checkout made withgit archivehas the time of the commit, which can be older than the objects of the last build, and make would not rebuild it. - Files that no longer exist in
SOURCE_DIRare removed from the copy. - make does not track the headers in
src/anddependence/, so when one of them changed, the.cfiles there are touched. - Files are overwritten in place, not removed and created again. With the
sources on a case-insensitive file system shared with a container (Docker
Desktop on macOS), rake loads mruby’s
Rakefileasrakefile, and after the file had been removed and created again, a second run in the same container failed withLoadError: cannot load such file -- .../rakefile.
make and rake do not notice every change, though: the objects of a removed
source or a dropped gem stay in libmruby.a, and nginx’s configure does not
run again. The script therefore records these inputs in
BUILD_DIR/build_stamp: the mruby tree (its git tree id when SOURCE_DIR is
the top of a git work tree, else a checksum of its files without
mruby/build and mruby/bin, the output of the test.sh build), the file
names under mrbgems/, checksums of build_config.rb, config.in,
configure, Makefile.in, build.sh and nginx_version,
NGX_MRUBY_CFLAGS (passed to mruby, as with test.sh), the
nginx configure options and the gem lock of the build. When the stamp
differs from that of the last build, mruby and nginx are built from scratch,
with new clones of the gems; the downloaded nginx source is kept. The new
clones matter when only the lock changed: rake checks an existing clone out
at the commit of the lock without fetching, and an older clone may not have
that commit. After a change that the stamp
does not cover, for example an uncommitted change under mruby/ (the git
tree id is that of the commit), reset the build by hand with
rm -rf build_soak (or build_perf).
Performance comparison with callgrind
Section titled “Performance comparison with callgrind”A change to the request path can make every request do more work without
failing any test. The performance comparison counts the instructions that
nginx executes per request (callgrind’s Ir) in the base and the head of a
pull request, both release builds, and reports the change per scenario.
Instruction counts do not depend on the speed or the load of the machine, so
the comparison works on shared CI runners, whose throughput varies by more
than 30% (section 3.7 of docs/proposals/v3-plan.md). They do not show cache misses, branch mispredictions
or the time spent in the kernel: they measure how much work the user-space
code of nginx, ngx_mruby, mruby and the C library does for a request.
What it measures
Section titled “What it measures”The HTTP scenarios are those of the soak test that use keep-alive clients:
hello, headers, var, filter, sleep, sub_request and file (see
the table in “Soak test for memory”), and the agent proxy scenarios of “Agent
proxy scenarios” below, plus the TCP stream_sleep scenario.
disconnect is not measured. For each scenario, and
for the base and the head in turn, test/perf/perf.rb:
- starts the mock LLM when the scenario has one, and nginx with
test/soak/nginx.conf(or the scenario’s template), changed tomaster_process off, undervalgrind --tool=callgrind --instr-atstart=no, so that the startup runs without instrumentation; - checks one response and sends
PERF_WARMUPrequests or TCP sessions; - completes every TCP response through EOF, or waits until
stub_statusshows no open HTTP connection of the warmup, and switches the instrumentation on withcallgrind_control --instr=on; - sends
PERF_Nrequests over keep-alive and checks every response. nginx closes a connection after 1000 requests (keepalive_requests), and the client then opens the next one. Forstream_sleep, it instead completesPERF_NTCP sessions, one per connection, checking exact bytes and EOF; - writes the profile with
callgrind_control --dump, and stops nginx.
The profile of step 5 holds the cost of the requests of step 4: not the
startup, the configuration, the warmup or the exit. Ir per request is
its total Ir divided by PERF_N; for stream_sleep this is per completed
TCP session. The alternative, the difference of two runs
with N and 2N requests divided by N, needs three times as many requests and
adds the variation of two runs; the window of one run has nothing but the
requests in it.
callgrind_control reaches the process through valgrind’s gdbserver (vgdb),
which does not serve a process that valgrind forked, such as an nginx
worker. With master_process off, the one nginx process runs the event loop
and the handlers as a worker does. When vgdb may not use ptrace to interrupt
a process that waits in epoll_wait, a command runs the next time the
process runs code, so perf.rb sends a request to /status (stub_status)
every 0.2 seconds until the command has run. Parts of these requests (two
per window in the runs so far) fall into the window.
Ir per request without GC leaves out the cost of mruby’s garbage
collector: the inclusive cost of the calls into mrb_incremental_gc and
mrb_full_gc made from outside the collector (from mrb_obj_alloc, the
malloc wrappers, GC.start). It is the inclusive cost that
callgrind_annotate --inclusive=yes prints for mrb_incremental_gc and
mrb_full_gc without a recursion suffix ('2), without counting twice a
full GC that mrb_incremental_gc runs; perf.rb reads it from the call
records of the profile (see the comment of parse_profile). The GC runs in
bursts: mrb_obj_alloc calls the collector when the number of live objects
passes a threshold, and one cycle (root scan and marking, then sweeping) is a
few calls. In hello, 20000 requests make 21 calls, about one cycle per
2000 requests, and the GC is about 6% of the total. A change that allocates a
few objects more or less per request moves the cycles against the window,
and the total moves by a share of one cycle. The number without GC does not
have that step.
Ir per request without GC and constant lookups (w/o GC/const in the
report) also leaves out the constant lookups that miss the constant cache of
mruby: the self cost of mrb_vm_const_get and the inclusive cost of its
calls into iv_get (the search of the table of each class), which is all
that it calls in the profiles of the CI runs (CONST_LOOKUP_CALLEES in
test/perf/perf.rb). A call into any other function is not subtracted,
and the line of the scenario prints it as `not subtracted:
OP_GETCONST of mruby’s VM (mruby/src/vm.c) calls mrb_vm_const_get
only when its constant cache misses. The cache has 64 entries
(MRB_CONST_CACHE_SIZE), chosen by a hash of the address of the irep and
the symbol, so a lookup misses for two reasons:
- Two lookups that a request makes take the same entry, and then both
miss on every request. Whether they do depends on where the ireps were
allocated, which can differ between two runs of the same binary. On
#766, the head of
stream_sleepmissed 2 lookups per TCP session in two runs of theperfjob and none in a third run on the same base and head commits (ea3c5996eande7140ba08):mrb_vm_const_getandiv_getmade 671 Ir more per session, +6.05% without GC, aFAILof a change that the code did not make. The handler looks upNginx, andNginx::Stream::Async.sleep(mrbgems/ngx_mruby_mrblib) looks upFiber; which two lookups shared an entry was not observed. In later runs onnext,authmissed 2 or 3 lookups per request in both builds, by run. - The whole cache was cleared, and every lookup after the clear misses
once.
mrb_const_set,mrb_const_removeandmrb_define_const_id(mruby/src/variable.c) andmrb_include_moduleandmrb_prepend_module(mruby/src/class.c) clear it, so a constant set in Ruby (OP_SETCONST,Module#const_set),include,prependandextenddo as well;Rack.runofmrbgems/rack-based-apicallsKernel.const_seton every request. Such misses are the cost of the code that clears, and do not depend on the layout.
The report counts both per request: const_misses, the calls of
mrb_vm_const_get, and const_clears, the calls of those functions and
of mrb_const_cache_clear from a function that is not one of them, so a
clear counts also where gcc inlines it into its caller. callgrind recorded
2 of them per call of Module#const_set on aarch64; only the counts of
the base and the head of one run are compared. A scenario
is judged on the number in the column judged of the report: w/o GC/const, or w/o GC (the constant lookups counted) when its head clears
the cache more often than its base, or clears it at all and misses more
lookups than its base: a lookup that a change adds after a clear that
both builds make misses on every request as well. So the misses that a
clear causes count against the change. A head that does not clear, or
that clears as often as the base and misses no more, is judged w/o GC/const; its misses come from the layout. The report prints the total, the number
without GC and the number without GC and constant lookups. A lookup misses
the cache once after its entry was taken, so a change that makes a request
look up more constants still pays for the lookups that hit (the
instructions of OP_GETCONST in mrb_vm_exec). mrb_vm_const_get
allocates only when it calls const_missing, and a GC that such an
allocation would run is left out twice.
The GC also changes the cost of the rest: after a sweep has freed many
blocks, the next malloc calls of the request code take longer. The window
therefore has to span many GC cycles. With PERF_N=20000, hello measured
11039 Ir per request without GC, and 11049 (+0.1%) with the window moved by
5000 requests (PERF_WARMUP=7000); with PERF_N=1000 and
PERF_WARMUP=200, a window shorter than one cycle, it measured 10726 and
10732 in two runs (-2.8%). That is why PERF_N is 20000.
Agent proxy scenarios in the comparison
Section titled “Agent proxy scenarios in the comparison”These scenarios measure what an agent proxy costs per request: nginx in
front of the mock LLM (see “Mock LLM upstream”), with test/soak/nginx.agent.conf
(see “Agent proxy scenarios” in the soak section). The scenarios without
Ruby are the baselines of the scenarios of the same shape, so that the
difference between the two is what the Ruby adds. The last row of the table
is the seven scenarios of test/soak/nginx.conf, for their request
function.
| Scenario | Request and answer | Ruby per request | Request function (calls per request) |
|---|---|---|---|
proxy_plain_2k, proxy_plain_64k |
POST of a Messages body of exactly 2 KB or 64 KB to /v1/plain (proxy_pass to the mock); a JSON message |
none | ngx_http_log_request (1) |
proxy_stream_plain_50, proxy_stream_plain_1000 |
the 2 KB body with "stream": true to /v1/plain; a stream of 50 or 1000 deltas (55 or 1005 events) |
none | ngx_http_log_request (1) |
responses_stream_plain_50, responses_stream_plain_1000 |
a 2 KB Responses body with "stream": true to /v1/responses/plain (proxy_pass to the mock’s /v1/responses); a Responses stream of 50 or 1000 deltas and response.completed (58 or 1008 events) |
none | ngx_http_log_request (1) |
auth |
proxy_plain_2k to the server auth.agent.test, whose server rewrite handler agent_auth.rb looks up the client key and sets the upstream credential |
1 handler | ngx_http_log_request (1) |
route_json_2k, route_json_64k |
proxy_plain_* to /v1/route, whose access handler agent_route.rb parses the body with JSON.parse and sets the upstream block for proxy_pass http://$agent_backend |
1 handler | ngx_http_log_request (1) |
ruby_call_1, ruby_call_10 |
GET of a location with 1 or 10 mruby_set_code variables and return 200; no upstream |
1 or 10 calls | ngx_http_mruby_set_inline_handler (1 or 10) |
hello, headers, var, filter, sleep, sub_request, file |
GET of the locations of test/soak/nginx.conf (see “Soak test for memory”); no mock |
1 handler or filter | ngx_mrb_start_fiber (1) |
The seven scenarios of test/soak/nginx.conf count ngx_mrb_start_fiber:
ngx_mrb_run, which runs the Ruby code of a handler or filter, starts one
fiber for each run, and nothing else calls ngx_mrb_start_fiber. They do
not count ngx_mrb_run, because callgrind does not record its calls the
same way in every build. In the aarch64 build of next at 45c9e52
(2026-10-04), it recorded a second call of ngx_mrb_run per request: in
filter as a call from ngx_mrb_run to ngx_mrb_run'2, which counts
once, and in the other six scenarios as a call from
ngx_mrb_http_get_module_ctx.part.0 to ngx_mrb_run'2, although
ngx_mrb_http_get_module_ctx does not call ngx_mrb_run, so their
windows held two calls per request and failed the check. It recorded one
call of ngx_mrb_start_fiber per request in all seven, as in the x86_64
builds of the CI runner.
The scenarios with an upstream count ngx_http_log_request, which nginx
calls once for each request it ends, with or without Ruby (for a
subrequest only with log_subrequest on): the scenarios without Ruby have
no call of ngx_mrb_start_fiber to count, and all the scenarios with an
upstream count the same function. The /status requests that wake nginx for
callgrind_control also end in the window when they come after the
instrumentation was switched on, so the window may hold up to that many
calls more (2 per window in the runs so far). mruby_set_code reaches
ngx_mrb_run through a tail call of its handler, which callgrind does not
always record as a call of ngx_mrb_run: in an aarch64 build, one of the
ten calls per request of ruby_call_10 showed as a call from the handler to
ngx_mrb_start_fiber, and ruby_call_1 showed none. The ruby_call_*
scenarios therefore count the handler, which nginx calls through a pointer.
Four counts per request are printed after the table and kept in
report.json, but not checked: ruby_calls, the calls of
ngx_mrb_start_fiber (ngx_mrb_run starts one fiber per run of Ruby code,
and nothing else calls it; in the seven scenarios of test/soak/nginx.conf
the window check counts the same calls), and non_buffered_calls, the
calls of ngx_http_upstream_process_non_buffered_request, which relays an
unbuffered response. nginx calls it when the upstream connection becomes
readable and when the client connection becomes writable. One call reads
from the upstream until recv() would block, at most proxy_buffer_size
(4k) at a time, and writes what it read to the client, so the count is not
a count of 4k reads: how many calls a response takes depends on how much of
it had arrived at each call. It also differs between builds and machines:
2.00 per request for proxy_plain_*, auth and route_json_* in the
aarch64 container, 1.00 on the CI runner (x86_64). The third and the
fourth are const_misses and const_clears, the misses and the clears
of mruby’s constant cache (“What it measures”). When a build made constant
lookups, the line also gives their Ir per request in both builds
(const lookups <base>/<head> Ir), and the calls of mrb_vm_const_get
that are not subtracted.
Each call costs Ir. On aarch64, proxy_stream_plain_50 (PERF_N=2000)
with the mock writing the stream in one write, one write per event, and one
write per event 1 ms apart made 3.00, 4.96 and 109.99 calls per request and
cost 52270, 54255 and 142740 Ir per request: about 1010 Ir per call from the
first to the second, and 850 from the first to the third. With two builds,
the line of a scenario whose calls differ gives the Ir that the difference
alone makes at 1000 Ir per call (NON_BUFFERED_CALL_IR), as a share of the
base without GC, and from half of PERF_WARN_PERCENT on it says that the
change of the scenario may come from when the response arrived. For
proxy_stream_plain_50, about 2 more calls per request reach WARN and
about 3 reach FAIL; between the base and the head of one run, its calls
differed by 0.01 to 0.25 per request in the null changes of “Thresholds and
calibration”.
The line of a scenario whose const_misses differ gives the Ir of those
lookups in each build (measured, not estimated), as a share of the base
without GC and constant lookups, with “not judged”, and from half of
PERF_WARN_PERCENT on it says that more (or fewer) lookups of the head
miss the constant cache of mruby, for example const_misses 0.00/2.00 (const_misses +2.00 per request: +671 Ir, +5.86% of base w/o GC/const, not judged; more constant lookups of head miss the constant cache of mruby)
for stream_sleep of the FAIL run on #766. When the head does not
clear the cache, the note shows that the addresses of the head took other
cache entries: it does not change the verdict, and a second run is not
needed for it. When the head clears the cache more often, or clears it
and misses more lookups, the line ends with head clears the constant cache of mruby more often: judged w/o GC or head clears the constant cache of mruby and misses more lookups: judged w/o GC, and the row is
judged with the misses counted.
First measurement, 2026-10-04: one run of test/perf/perf.rb on the head
build of this checkout (next at f62880c plus the harness), nginx 1.31.6,
Ubuntu 22.04 (gcc 11, valgrind 3.18.1), container on aarch64,
PERF_N=20000. H is hello in the same run, 11856 Ir per request (11158
without GC):
| Scenario | Ir/req (w/o GC) | H | non_buffered_calls/req |
|---|---|---|---|
proxy_plain_2k |
21952 (21952) | 1.85 | 2.00 |
proxy_plain_64k |
22264 (22264) | 1.88 | 2.00 |
proxy_stream_plain_50 |
54294 (54294) | 4.58 | 4.97 |
proxy_stream_plain_1000 |
432228 (432228) | 36.5 | 38.43 |
auth |
49132 (46977) | 4.14 | 2.00 |
route_json_2k |
109396 (106246) | 9.23 | 2.00 |
route_json_64k |
1603842 (1600216) | 135.3 | 2.00 |
ruby_call_1 |
11404 (10737) | 0.96 | 0 |
ruby_call_10 |
41541 (35237) | 3.50 | 0 |
The differences between scenarios of the same shape, from that run:
authminusproxy_plain_2k: 27180 Ir per request (25025 without GC), 2.3 H, for one server rewrite handler with two Hash lookups and three variable assignments.route_json_2kminusproxy_plain_2k: 87444 (84294), 7.4 H, for one access handler that parses a 2 KB body, plus theproxy_passwith a variable, which looks the upstream block up by name.route_json_64kminusproxy_plain_64k: 1581578 (1577952), 133 H. The 62 KB more of body cost about 24100 Ir per KB without GC.- One more Ruby call,
(ruby_call_10 - ruby_call_1) / 9: 3349 Ir (2722 without GC), 0.28 H. - One more stream event without Ruby,
(proxy_stream_plain_1000 - proxy_stream_plain_50) / 950: 398 Ir, with thenon_buffered_callsof that run (4.97 and 38.43 per request).
The Responses streams, 2026-10-19: one run of test/perf/compare.sh with
the release builds of next at a466688 (base) and of the head that
added the scenarios (the same binary), nginx 1.31.6, Ubuntu 22.04 (gcc 11,
valgrind 3.18.1) in a container on aarch64, PERF_N=20000. The base
build measured hello at 11618 Ir per request (11260 without GC), and:
| Scenario | Ir/req (w/o GC) | H | non_buffered_calls/req |
|---|---|---|---|
proxy_stream_plain_50 |
53674 (53674) | 4.62 | 4.27 |
responses_stream_plain_50 |
55710 (55710) | 4.80 | 5.65 |
proxy_stream_plain_1000 |
433190 (433190) | 37.3 | 37.20 |
responses_stream_plain_1000 |
478098 (478098) | 41.2 | 59.46 |
One more relayed Responses event,
(responses_stream_plain_1000 - responses_stream_plain_50) / 950, cost
445 Ir against 399 Ir for a Messages event in the same run. A delta event
of the Responses stream is 186 to 189 bytes (its sequence_number grows),
one of the Messages stream 121 bytes; the body of the 1000-delta stream is
215046 bytes against 121619, and it took 59.46 relay calls per request
against 37.20.
proxy_stream_plain_1000 and route_json_64k took 83 and 129 seconds per
build in that run, while all the scenarios of the default set together take
about 140 seconds, so they are not in DEFAULT_SCENARIOS, nor is
responses_stream_plain_1000; name them in PERF_SCENARIOS. The other
agent proxy scenarios are in the default set and add about one minute per
build.
Thresholds and calibration
Section titled “Thresholds and calibration”A scenario is WARN when its judged number (the column judged: without GC
and constant lookups, or without GC where the misses may come from a clear
of the constant cache in the head; “What it measures”) is 3% or more above
the base (PERF_WARN_PERCENT), and FAIL from 5% (PERF_FAIL_PERCENT).
The exit status is 1 when a scenario is FAIL, when a measurement fails
(ERROR: an unexpected response, a line at the error level or above in
error.log, a timeout, a missing dump, or a window that does not hold the
requests, see “In CI”), or when no scenario was compared. The one exception
is a base that answers with an unexpected response while the head passes,
which is what a scenario that needs a feature of the head gets: it is shown
as n/a and does not fail the run. That holds only when nothing else went
wrong in the base measurement. Its error.log may have the lines that
ngx_mruby logs when the Ruby code of a request raises (mrb_run failed: return 500 HTTP status code to client: error: ...), which is what a base
that lacks a Ruby method or class of the head logs (NoMethodError,
NameError). Any other line at the error level or above makes the
scenario ERROR, for example the connect() or recv() error that nginx
logs when the mock LLM failed and nginx answered 502. So do a problem of
the mock, and more than five lines at those levels in one log: the report
shows five lines per log and does not read the rest. A directive that only
the head has stops the base at startup, which is ERROR as well.
The thresholds were checked with a null change on 2026-10-03: three runs of
compare.sh on two checkouts with the same build inputs (next at
f713d4e, and a branch that changes only test/ and the documentation),
with nginx 1.31.6 on Ubuntu 22.04 (gcc 11, valgrind 3.18.1) in a container
on aarch64. The first run built both trees; the other two measured the same
builds again (ONLY_RUN=1). Change of head against base, total / without
GC:
| Scenario | Base Ir/req (w/o GC) | Run 1 | Run 2 | Run 3 |
|---|---|---|---|---|
hello |
11736 (11039) | +0.002% / +0.003% | -0.000% / -0.000% | +0.008% / +0.008% |
headers |
36002 (34025) | +0.002% / +0.002% | -0.002% / -0.002% | +0.000% / +0.000% |
var |
21199 (19700) | -0.000% / -0.000% | +0.000% / +0.000% | +0.000% / +0.000% |
filter |
14010 (13089) | -0.002% / -0.002% | -0.002% / -0.002% | +0.002% / +0.002% |
sleep |
15427 (14654) | -0.006% / -0.007% | -0.002% / -0.002% | +0.000% / +0.000% |
sub_request |
48710 (47538) | -0.003% / -0.004% | -0.001% / -0.001% | -0.001% / -0.001% |
file |
14009 (13075) | +0.002% / +0.002% | +0.000% / +0.000% | -0.002% / -0.002% |
The largest change is 0.008%, about 1 instruction per request, more than
300 times below the 3% of WARN; N does not need to be raised. A later
run on 2026-10-04, on new builds of the same two checkouts, measured
sub_request at +0.076%
(47501 and 47537 Ir per request without GC) and every other scenario
within 0.011%: sub_request, which proxies to a second server, varies the
most, on the CI runner as well (below). A request
to /status that wakes callgrind_control costs about 12500 instructions
(the connection included), and parts of two of them are in each window, at
most about 0.01% of the smallest window (hello, 235 million
instructions).
The agent proxy scenarios were checked with null changes on 2026-10-04:
two in the same kind of container on aarch64, with all nine of them (the
two left out of the default set included), and the perf job of the pull
request that added them on the CI runner (x86_64, run 37152930938: base
next at f62880c, head the merge commit, no build input changed), with
the seven of the default set. Change of head against base, total / without
GC:
| Scenario | aarch64, same build twice (next at 4f29b06) |
aarch64, compare.sh, next at f62880c and the pull request |
x86_64, CI run 37152930938 |
|---|---|---|---|
proxy_plain_2k |
+0.000% / +0.000% | +0.005% / +0.005% | +0.000% / +0.000% |
proxy_plain_64k |
+0.005% / +0.005% | +0.009% / +0.009% | -0.110% / -0.110% |
proxy_stream_plain_50 |
+0.042% / +0.042% | +0.023% / +0.023% | -0.063% / -0.063% |
proxy_stream_plain_1000 |
+0.004% / +0.004% | +0.007% / +0.007% | not run |
auth |
+0.000% / +0.000% | +0.000% / +0.000% | +0.000% / +0.000% |
route_json_2k |
+0.000% / +0.000% | +0.000% / +0.000% | -0.000% / -0.000% |
route_json_64k |
+0.000% / +0.000% | +0.000% / +0.000% | not run |
ruby_call_1 |
+0.000% / +0.000% | +0.000% / +0.000% | +0.000% / +0.000% |
ruby_call_10 |
+0.003% / +0.004% | +0.000% / +0.000% | +0.000% / +0.000% |
The scenarios with an upstream vary the most, and the non_buffered_calls
do not explain all of it. On aarch64, proxy_stream_plain_50 moved by up
to 0.042% with its calls at 5.01 and 5.02 per request in base and head. On
the CI runner, proxy_plain_64k moved by -0.110% (22530 and 22506 Ir per
request) with its calls at 1.00 in both builds, and proxy_stream_plain_50
by -0.063% while its calls went from 4.86 to 4.94, which alone would add
about 0.14%. A second run of that job (37156005549, on the commit that
applied the review of that pull request, again with no build input
changed) measured proxy_plain_64k at +0.30% (22564 and 22630 Ir per
request, calls 1.00 in both builds) and proxy_stream_plain_50 at +0.39%
while its calls went from 4.92 to 5.17, which the comparison estimated at
+0.43%; every other scenario stayed within 0.02%. Other work whose amount
depends on timing, such as how many reads the 64 KB request body takes, is
not counted. Between runs of the same build on aarch64, auth measured
46797 and 46977 Ir per request without GC (0.38%) and route_json_2k
106066 and 106246 (0.17%). The largest change between the base and the
head of one run, which the thresholds compare, is 0.39% in these runs,
about 8 times below the 3% of WARN. Later runs on the runner can move the
scenarios with an upstream by a few tenths of a percent as well; a change
of these scenarios near WARN needs a second run, and the
non_buffered_calls note, before it is taken as a regression.
In the compare.sh run, the seven scenarios of test/soak/nginx.conf
failed their window check in both builds on aarch64: they counted
ngx_mrb_run then, and callgrind recorded two calls of it per request (see
“Agent proxy scenarios in the comparison”). They count
ngx_mrb_start_fiber now. A null change on 2026-10-04 in the same kind of
container (compare.sh with next at 45c9e52 as the base and a branch
that changes only test/perf/ and the documentation as the head) found
exactly 20000 calls of it in each window of the seven, in both builds, and
measured every scenario of the default set within 0.017%
(proxy_stream_plain_50), the seven within 0.009%.
To check that the thresholds fire, the head was a copy of the base with an
empty loop of 100 iterations (volatile counter) at the start of
ngx_mrb_run, which each of the seven scenarios of that time runs once per
request (the agent proxy scenarios came later; the proxy_* ones run no
Ruby, and ruby_call_10 runs it ten times). The self cost of
ngx_mrb_run grew by 604 instructions per request in every scenario, and
the report showed hello +5.47% (FAIL), filter and file +4.62%,
sleep +4.10%, var +3.07% (WARN), headers +1.79% and sub_request
+1.26% (ok), the 604 instructions over the Ir per request without GC of
each scenario, and compare.sh exited with 1. In instructions, 3% is
about 330 per request in hello and 1430 in sub_request.
On the CI runner (ubuntu-22.04, x86_64, the same versions), the perf job
of the pull request that added it is the same null change: the base was
next at f713d4e, the head the merge commit. Four runs of that job, each
building both trees: runs 1 to 3 on the first commit of that pull request
(the run and two re-runs of the job), run 4 on its second commit, which
changed only the documentation:
| Scenario | Base Ir/req (w/o GC) | Run 1 | Run 2 | Run 3 | Run 4 |
|---|---|---|---|---|---|
hello |
11669 (11006) | -0.003% / -0.004% | -0.003% / -0.003% | +0.000% / +0.000% | +0.000% / +0.000% |
headers |
39424 (37267) | -0.002% / -0.002% | +0.000% / +0.000% | +0.000% / +0.000% | +0.000% / +0.000% |
var |
22383 (20706) | +0.000% / +0.000% | +0.000% / +0.000% | +0.000% / +0.000% | +0.000% / +0.000% |
filter |
14569 (13636) | +0.000% / +0.001% | +0.003% / +0.004% | +0.000% / +0.000% | +0.003% / +0.002% |
sleep |
15833 (15068) | +0.006% / +0.006% | +0.000% / +0.000% | +0.001% / +0.001% | +0.002% / +0.002% |
sub_request |
49996 (48854) | +0.001% / +0.001% | +0.016% / +0.016% | -0.001% / -0.001% | +0.020% / +0.021% |
file |
14309 (13360) | +0.002% / +0.002% | +0.000% / +0.000% | -0.002% / -0.003% | +0.000% / +0.001% |
Two more runs of the job, on the commits that applied the reviews of that
pull request (still with no change to a build input), gave every scenario
within 0.007% except sub_request, at +0.005% and -0.047% (48706 and 48683
Ir per request without GC). Over the six runs up to 2026-10-04:
- Between base and head of one run, the largest change was about 0.05%
(
sub_request, about 23 instructions); every other scenario stayed within 0.007%. - Between runs,
sub_requestmeasured from 48683 to 48857 Ir per request without GC (0.36%); every other scenario stayed within 0.015%.
sub_request, which proxies to a second server, is the scenario that
varies; even its largest change within a run is about 60 times below the 3%
of WARN. Base and head are measured in the same job, one after the other,
so it is the change within a run that the thresholds see. The numbers differ from those of
aarch64 because the instruction set differs; on x86_64, callgrind shows no
recursion suffix on the GC entry points, and the number without GC again
equals the inclusive cost of mrb_incremental_gc that callgrind_annotate
prints. On next, ci-ok needs the job; see “In CI” below.
Known differences
Section titled “Known differences”These differences of the bundled mruby are accepted, so a later comparison
measures against them; the base of a pull request is next with mruby
4.1.0-rc2 since #678.
- mruby 4.1.0-rc2 against 4.0.0, both with the GC root holder of #682
(the comparison on #678):
auth42599 → 45621 Ir per request without GC (+7.10%), and the GC share of the scenarios with Ruby grows, for exampleauth3.7% → 13.5% andheaders6.6% → 20.9%; the other scenarios stay belowWARNwithout GC.- The number without GC: the VM and the core of mruby 4.1 run the same
bytecode with more instructions in
auth(about +2260 Ir per request in mruby’s own functions): a two-way method cache that finds 22 methods per request by a search where 4.0.0 found 17,Hashlookups of String keys throughobj_eql, and another split of the argument parsing of C methods. These are how mruby 4.1 is built; none is known as a defect. - The GC share: when the heap has no free slot, mruby 4.1 runs a full GC
instead of adding a heap page if its last mark left room (mruby commit
c009f1633, “gc: try to reclaim before growing the object heap”, made to
bound the heap of small fixed-arena targets). A worker of ngx_mruby
keeps about 1100 live objects in two pages of 1024, so the heap runs out
before an incremental GC starts, and each time a full GC runs.
GC.interval_ratiodoes not change it, and a larger heap (MRB_HEAP_PAGE_SIZE4096, or pages thatmrb_gc_add_region()adds at the start) takes the GC share to 0-3% but raises the number without GC in most scenarios, by up to 17%; the measurements are on #678. ngx_mruby keeps mruby’s defaults.
- The number without GC: the VM and the core of mruby 4.1 run the same
bytecode with more instructions in
- The owner accepted these differences on 2026-10-31 (the record on #678
and in
docs/HANDOFF.md).
A comparison whose base bundles an older mruby than the head measures such
a difference again: one with a commit of next from before #678 as the
base meets the differences of mruby 4.0.0 above. The comparison with the
previous tag before a tag (see “Design principles” in AGENTS.md) has an
older base as well: before the first pre-release of v3 the base is
v2.7.2, which bundles mruby 3.3.0. No line of the file covers mruby 3.3
against 4.1, so the WARN and FAIL rows of that comparison go to the
owner as the rule says. test/perf/accepted.txt
lists the accepted differences, one line each: a condition, the scenario,
and an allowance on the change of the judged number, in percent.
There are two conditions. mruby-version compares the MAJOR.MINOR of
MRUBY_RELEASE_MAJOR and MRUBY_RELEASE_MINOR in
mruby/include/mruby/version.h of each build’s tree; the comment at the
top of the file has the format. mruby-version 4.0 4.1 auth 8 applies
when the base bundles mruby 4.0.x and the head 4.1.x; such a line applies
only when both builds match it. pull-request is described below. When
a line applies, a WARN or FAIL row of its scenario
within the allowance shows as ok (accepted: mruby 4.0.0→4.1.0, +8%),
does not fail the run, and is a notice in the annotations of the job; a
change above the allowance keeps its WARN or FAIL. A header line of
report.txt, and a line below the table of the job summary, print the
mruby version of both builds and the differences that applied, for
example perf: mruby: base 4.0.0, head 4.1.0; accepted differences (accepted.txt): auth up to +8% (mruby 4.0.0→4.1.0), or none apply. perf.rb reads the file before it
measures and stops at a line that is not in the format, with the file and
the line number; --self-test checks the file too. The GC share is
printed and not judged, so the file has no allowance on it.
The other condition, pull-request, is for a FAIL that the owner
accepted on a pull request (see “In CI”). Its line names the base branch
and the number of the pull request, for example pull-request next 812 stream_sleep 7, and applies only in the perf job of that pull
request, which passes them to perf.rb as PERF_BASE_REF and
PERF_PULL_REQUEST; a local run without them, the comparison with the
previous tag among them, applies no such line. The row shows as ok (accepted: pull request #812 to next, +7%), and the header line adds
pull request #812 to next. When a pull-request line and a
mruby-version line apply to one scenario, the pull-request line
does. The line stays in the file after the merge as the record, and
applies to no other pull request.
Running it
Section titled “Running it”When the pull request changes an input of the measured binary or the
measurement (perf true), its author runs the comparison with the base
before marking it ready and quotes the report (“Design principles” in
AGENTS.md); the perf job (In CI) then runs
it again on the runner.
The driver needs CRuby 3.0 or later and valgrind with callgrind_control
and its helper vgdb, which work on Linux; perf.rb stops at once
without them. On macOS, run it in a Linux container. To compare this
checkout with origin/next in a container, in one command:
$ rm -rf build_perf/base-src && git archive --prefix=build_perf/base-src/ origin/next | tar -x && docker run --rm -v "$PWD":/work -w /work ubuntu:22.04 sh -c 'apt-get -qq update && DEBIAN_FRONTEND=noninteractive apt-get -qq install -y build-essential rake ruby bison git gperf wget ca-certificates zlib1g-dev libpcre3-dev libssl-dev valgrind >/dev/null && sh test/perf/compare.sh build_perf/base-src'On Linux:
$ sh test/perf/compare.sh build_perf/base-src # build both, then compare$ ONLY_RUN=1 sh test/perf/compare.sh build_perf/base-src$ ONLY_RUN=1 PERF_SCENARIOS=hello,var sh test/perf/compare.sh build_perf/base-src$ sh test/perf/run.sh # measure this checkout onlycompare.sh BASE_DIR [HEAD_DIR] builds BASE_DIR into build_perf/base and
HEAD_DIR (default: this checkout) into build_perf/head, each with the
third-party gems at the commits of the build_config.rb.lock committed in
it. The gems therefore differ only where the head changes the lock, and a
pull request that moves a gem to another commit is measured with that
commit, as it would be merged. A checkout that has no lock (one from before
the lock was committed) takes the lock of the other side: the base the
head’s, the head the one that the base build wrote. The report says how many
gems are at different commits and lists them. The measurement (test/perf/, the scenarios in test/soak/) comes
from this checkout for both builds. The first run builds two trees (a few
minutes each); the measurement of the default HTTP scenarios takes about 140
seconds per build (139 seconds on the CI runner, 28 of them in sleep), and
proxy_stream_plain_1000 and route_json_64k add about 80 and 130 seconds
when PERF_SCENARIOS names them.
The report goes to build_perf/report.txt and
build_perf/report.json. test.sh starts its mock LLM upstream on 12372
too, so do not run it at the same time.
| Variable | Default | Meaning |
|---|---|---|
ONLY_RUN |
unset | set to skip the builds |
NUM_THREADS_ENV |
half of the CPUs | build parallelism (passed to build.sh) |
PERF_SCENARIOS |
DEFAULT_SCENARIOS of test/perf/perf.rb |
comma-separated scenario names |
PERF_N |
20000 | requests or TCP sessions in the measured window |
PERF_WARMUP |
2000 | requests or TCP sessions before the window |
PERF_WARN_PERCENT |
3 | WARN from this change of the judged number (the column judged) |
PERF_FAIL_PERCENT |
5 | FAIL from this change |
PERF_PORT_BASE |
12370 | HTTP port; the next is the HTTP backend or TCP data port, the third is the mock LLM or unused stream GC control port |
PERF_REPORT_DIR |
build_perf |
where report.txt and report.json go |
PERF_ACCEPTED_FILE |
test/perf/accepted.txt |
the accepted differences (see “Known differences”); an empty file shows the verdicts without them |
PERF_BASE_REF |
unset | the base branch of the pull request, matched against the pull-request lines of the accepted file; set with PERF_PULL_REQUEST or not at all |
PERF_PULL_REQUEST |
unset | the number of the pull request; the perf job sets both variables |
PERF_REQUEST_FUNCTIONS |
ngx_mrb_start_fiber |
names of the function counted once per request in the seven scenarios of test/soak/nginx.conf (see “In CI”) |
The perf job repeats on the runner the comparison that the author ran
locally before marking the pull request ready (Running
it), so that the result does not rest on one machine
alone. The perf job of .github/workflows/test.yml starts on ready pull
requests whose run builds (build is standard or full in Which jobs a run
starts), and compares when test/ci/changes.sh
gives perf true, that is when the pull request changes an input of the
measured binary or the measurement, or when it has the label perf (read
when the job runs, so adding the label and re-running the job is enough). A
pull request that changes only test.yml among these files compares only
with the label, and a pull request whose files are all outside the code and
the tests skips the job, with the label too. The inputs of the binary are what
test/build_release.sh copies: src/, mrbgems/, mruby/, dependence/
(ngx_Makefile.in adds to nginx), the top-level build
files build_config.rb, configure, config.in, Makefile.in,
build.sh and nginx_version, and the gem lock build_config.rb.lock. The
measurement is test/perf/,
test/build_release.sh and the scenario files in test/soak/
(nginx.conf, nginx.agent.conf, nginx.stream.conf, scenarios.rb, http_client.rb, tcp_client.rb,
mock_llm.rb, handlers/). The job of master is the same, with two
differences: master commits no gem lock, so the lock is not an input
there, and master has no soak job, so its perf job runs the check of
the mock LLM (ruby test/soak/mock_llm_test.rb), which test/soak/run.sh
runs on next, before it compares. On next, a change of
test/soak/mock_llm_test.rb gives soak true and so runs that check in
the soak job; on master, where soak starts no job, such a change
alone runs it only when the pull request has the label perf.
A change of nginx_version builds the head with another nginx, so its
WARN or FAIL measures nginx’s own cost as well as ngx_mruby’s; read the
functions of the profiles (below) before taking it as a regression of
ngx_mruby.
The checkout of a pull request is the merge commit (refs/pull/N/merge);
the job compares its first parent, the base branch, with the merge commit,
so the difference is the change of the pull request as it would be merged.
The table is in the job summary, with the reasons of the failed
measurements below it; WARN, FAIL, ERROR and n/a rows and a run
that compared no scenario are annotations; the job fails on FAIL and
ERROR. A measurement fails (ERROR) when nginx does not start or answer,
callgrind_control fails or times out, the dump is missing, error.log has
a line at the error level or above, or the window does not hold the
requests: 0 Ir, or not the expected calls of the scenario’s request
function: exactly one call of ngx_mrb_start_fiber per request in the
seven scenarios of test/soak/nginx.conf, and in the agent proxy scenarios
the functions and counts of the table in “Agent proxy scenarios in the
comparison” (callgrind_control prints “OK.” even when vgdb did not reach
the process, so an empty window is caught here). Only a base that answers a
scenario with an unexpected response and has no other problem than
ngx_mruby’s mrb_run failed lines (a scenario that needs a feature of the
head; see Thresholds and calibration of
the performance comparison) gives n/a, and a run in which no scenario was
compared fails.
The calls of the request function are counted by name. A copy that GCC
makes of a function at -O2 (ngx_mrb_start_fiber.part.0, .isra.0,
.constprop.0, .cold) counts as the function, and a call from one copy to
another counts once, so a change that makes GCC split or clone it does not
fail the window. A pull request that renames ngx_mrb_start_fiber, or
moves the start of the fiber of a Ruby run to another function, has to add
the new name to REQUEST_FUNCTIONS in test/perf/perf.rb and keep the old
one: the perf.rb of the head measures the base too, and a call between
two listed names counts once. GCC cannot inline ngx_mrb_start_fiber now,
because it is defined in ngx_http_mruby_async.c and called from
ngx_http_mruby_module.c. A change that lets the compiler inline it (for
example making it static in the file of its single caller, or a build
with LTO) would leave no call of it in the profile, and every window of the
seven scenarios would hold 0 calls, so such a change has to list a function
that keeps a real call.
PERF_REQUEST_FUNCTIONS (comma-separated) overrides the list for a local
run, REQUEST_FUNCTION_CALLS sets another number of calls per request
for a scenario, and SCENARIO_REQUEST_FUNCTIONS another function. ruby test/perf/perf.rb --self-test checks the name rules
and the reading of a profile on a made-up profile, without a build or
valgrind; compare.sh and run.sh run it first.
On next, ci-ok needs the job, so a FAIL or an ERROR row fails the
required check, and a WARN row does not. On master, ci-ok does not
need it yet; a later pull request to master adds it. Instruction counts do not vary with the
runner’s speed (calibration above). In the 15 runs of the job that
compared on the pull requests to next from #779 to #791 (2026-11-09 and
10, 16 scenarios each), every row was ok, the largest change 0.60%
without GC (responses_stream_plain_50). Judged again without the
constant lookups as well (the profiles of the runs of #788, #791 and #794),
every row stays ok. “Design principles” in
AGENTS.md says what each verdict asks of the pull
request: a WARN goes to the owner through the review, and a FAIL
holds the merge until the owner accepts it on the pull request. An
ERROR is a measurement that failed: fix the run or run it again.
To pass a FAIL that the owner accepted, the pull request adds a line to
test/perf/accepted.txt after the owner’s acceptance is in a comment on
it (the comment names the run, the scenario and its change): the
condition pull-request, the base branch, the number of the pull
request, the scenario, and an allowance in percent at or above the
measured change of the judged number, for example pull-request next 812 stream_sleep 7 for +6.05% (“Known differences”). The push of the line
runs the job again, which shows the row as ok (accepted: ...) and
passes; a later change above the allowance is a FAIL again. A FAIL
that no line covers, or an ERROR, keeps ci-ok failed.
On #766, two runs of the job measured stream_sleep at +6.05% (FAIL)
and a third run on the same base and head commits at -0.01%; the
difference was the misses of mruby’s constant cache, which the judged
number now leaves out (“What it measures”). Judged again from the
profiles of those runs, both give ok (+0.19% and -0.01%), and the first
has the note on const_misses.
The artifact perf-callgrind has the report, and for base and head the
profiles of the windows (callgrind/callgrind.out.<scenario>), the output
of callgrind_control and valgrind, and the nginx logs. To see where a
change comes from, print the functions of the same scenario in both builds
and compare them:
$ callgrind_annotate --inclusive=yes build_perf/head/callgrind/callgrind.out.var | head -40$ callgrind_annotate build_perf/base/callgrind/callgrind.out.var > base.txt$ callgrind_annotate build_perf/head/callgrind/callgrind.out.var > head.txt$ diff base.txt head.txtThe counts are for the whole window; divide by PERF_N for one request.
KCachegrind and QCachegrind open the profiles as well.
Site CI
Section titled “Site CI”The Markdown link check recognizes heading anchors and explicit anchors
written as <a id="name"></a>. Use an explicit anchor to preserve an old
link when renaming a heading. Code examples do not define anchors. Run
python3 site/scripts/test_check_links.py from the repository root to
check this compatibility behavior; npm run check:ci also runs it.
.github/workflows/site.yml runs on every push to next and on
workflow_dispatch (site-dispatch.yml of master starts it after every
push to master), not on pull requests. Local site checks cover the PR
head before it is ready; deployment validates the merged tree.
| Job | What it does |
|---|---|
build |
Builds a fresh clone (check-fresh-clone.sh), builds it again with network access blocked (check-offline-build.sh), builds the 2.x pages from master (build-v2.mjs), and runs npm test, check_links.py, the checks of npm run check, contrast.py, the Ruby blocks with the mrbc of the bundled mruby and check-v2.mjs --no-browser. It uploads the built site as the artifact site, and as github-pages only when every step passed |
deploy |
Publishes the artifact github-pages to GitHub Pages, on next only, after build passed |
verify-live |
Reads the deployment back: the artifact’s commit and the deployed commit, the branch policy of the environment, the published bytes, the redirect to HTTPS, the kept verification file, Lighthouse on the live home page, and screenshots of the published pages in the two patterns (below) |
The screenshots come from site/scripts/screenshots-live.mjs. It opens
the published pages in one read-only browser context for each engine and
language, and fails on a response of 400 or more, a page exception, a
failed or blocked request, or HTML that differs from the artifact. Right
after a deploy, GitHub Pages answers some files with 503 for a short time.
From 2026-10-29 to 2026-11-16, 8 of the 14 runs of site.yml on next
that failed or passed only on a re-run failed on that alone, 1.5 to 2
minutes after the deploy; the other 6 failed in the build, the deploy,
the setup of the verification job or its check of the deployment through
the GitHub API. The re-runs of 3 of the 8 started the check 126, 144
and 289 seconds after the failure and got 200, so the 503 ended within 153
seconds in the quickest case. When the only errors of a context are 5xx
answers of the site’s own origin and failed requests of the same URLs, the
script fetches each of those URLs with Node every 5 seconds until it
answers 200, then checks that context again from its first page, once. The
waits of a run share a limit of 240 seconds, which keeps the job within its
timeout-minutes: 10. A URL that does not answer 200 within the limit, a
second failure of the same context, and every other error fail the step at
once, as before. The script prints each retry (the context, the URL, its
status, the seconds until it answered 200, and the errors of the first
attempt), and its evidence.json lists them under retries. --local never retries.
site/scripts/test-screenshots-live.mjs, which npm run check:ci runs,
puts a proxy in front of the built site and checks four cases: a 503
that ends is checked again and passes, a 503 that stays fails at the
limit, and a 404, alone or with a 503 in the same context, fails at
once.
No job runs a browser on the built site before the deploy. Every browser
check runs locally instead, in npm run check:ci in site/
(site/scripts/check-ci.sh), which the author of a pull request that
changes docs/, site/ or examples/ runs before marking it ready, and
quotes under “How was this tested”. It runs:
- the checks of the
buildjob:npm run build, the 2.x pages (fromNGX_MRUBY_V2_TREE, or from a clone oforigin/masterinsite/generated/v2-tree),npm test,check_links.pyon the Markdown files that git tracks,npm run check,contrast.py, the Ruby blocks withmrbc(fromMRBCormruby/build/host/bin/mrbc; without one it says that it skipped the check) andcheck-v2.mjs --no-browser; - every browser check on every page, which
npm run check:browserruns alone on an earlier build:check-styles.mjsandcheck-japanese-lines.mjswith--ja-face notoin Chromium and WebKit,check-japanese-lines.mjson/v2/ja/,check-copy-idle.mjs,check-home-interaction.mjs,check-cta-edges.mjs(painted CTA rims at rest, hover, keyboard focus and active, including native button/submit probes, in both languages and themes),check-code-layout.mjson the 3.x and 2.x pages,check-focus.mjs,check-language-links.mjs,check-lighthouse.mjs,check-project-badges.mjs --browser,check-search-identifiers.mjs,check-v2.mjswith the search in Chromium,test-read-only-browser.mjs,screenshots.mjs,screenshots-live.mjs --localandtest-screenshots-live.mjs.
The fresh-clone and offline builds stay in the build job; a pull request
that changes site/package.json, site/package-lock.json or the build
scripts runs them locally as well (sh scripts/check-fresh-clone.sh,
npm run check:offline). node scripts/test-site-workflow.mjs, which
npm test runs, fails when site.yml gains a pull request trigger or a
job other than these three, or when a check of the build job or a
browser check is missing from check-ci.sh. SITE_CI_PAGES (a JSON array
of routes) limits check-styles.mjs, check-focus.mjs and
check-code-layout.mjs to some pages, and SITE_CI_VERSION (2 or 3)
check-code-layout.mjs to one version, for a quicker local loop; such a
run is not the evidence of a pull request.
A change of code or tests in the base can break a check of the site
that reads src/, test/ or build_config.rb (the directive tables, the
cases, the gems) after the pull request’s local run. The run of site.yml
on the push of the merge catches it before it deploys, and the fix is a
pull request of its own (condition 2 of “Review and merge” in
AGENTS.md).
A pull request that merges master into next changes docs/ as well,
so its author runs npm run check:ci in site/ of its head before
marking it ready. The check of “The 2.x pages of the site” in
docs/test/README.md on master does not stand for this run: it builds
the 3.x pages from a checkout of next without the merge, not from the
merged tree. When the merge brings a change to an English page that has a
Japanese page in docs/ja/ (docs/releases/README.md, for example), the
same pull request updates the Japanese page, because check-japanese.mjs
compares the code blocks of the two pages. This keeps the push of the
merge from stopping the deploy of the site on the checks of the merged
tree.
The layout and wrapping checks of the site (check-styles.mjs,
check-japanese-lines.mjs) measure two patterns only, PC at full screen
(1440px, Chromium) and iPhone Safari (402px, WebKit); no other widths,
browsers or font combinations are added to the checks.
site/scripts/lib/patterns.mjs holds them.
The badges of the strip of the four index pages (the workflows test,
nginx-tests, valgrind-nightly, CodeQL and zizmor, from GitHub, and
OpenSSF Best Practices, from bestpractices.dev) are the only images of the
site from other origins; the license badge is a file of the site, and the
release badge of the home pages and the star count of the header are made at
build time (site.yml reads the star count from the GitHub API before the
builds and passes it in NGX_MRUBY_SITE_STARS and
NGX_MRUBY_SITE_STARS_DATE; a build without them shows no count). The read-only browser context
(scripts/lib/read-only-browser.mjs) still allows only same-origin GET
requests; a caller may name exact image URLs whose GET it answers from a local
fixture instead of the network. The local browser checks of npm run check:ci (check-project-badges.mjs, check-home-interaction.mjs,
screenshots-live.mjs --local) and check-layout-parity.mjs serve those
URLs from the stand-ins of scripts/lib/project-badges.mjs, so a slow or
failing service cannot hold them. check-project-badges.mjs also loads each index
page with the external images refused and checks that the strip keeps its
box. Only check-live.mjs, after the deploy, fetches the real images, and
retries each three times before it reports a problem.