Add test
2.xのドキュメントは、リリースノートを除いて英語で載せています。このページにも日本語訳はありません。
ngx_mruby test uses mruby test. ngx_mruby test is very earlier and experimental version.
Welcome pull-request!
Add test
Section titled “Add test”Add location config to test/conf/nginx.conf
Section titled “Add location config to test/conf/nginx.conf”# Nginx.hello testlocation /mruby { mruby_content_handler build/nginx/html/unified_hello.rb cache;}Add hook script into test/html/ if you need the script for location config
Section titled “Add hook script into test/html/ if you need the script for location config”# test/htdocs/unified_hello.rbif server_name == "NGINX" Server = Nginxelsif server_name == "Apache" Server = Apacheend
Server::rputs "Hello #{Server::module_name}/#{Server::module_version} world!"Add test code to test/t/ngx_mruby.rb
Section titled “Add test code to test/t/ngx_mruby.rb”assert('ngx_mruby', 'location /mruby') do res = HttpRequest.new.get base + '/mruby' assert_equal 'Hello ngx_mruby/0.0.1 world!', res["body"]endAdd a test as a fragment and a case file
Section titled “Add a test as a fragment and a case file”Instead of editing test/conf/nginx.conf and test/t/ngx_mruby.rb, a test
can live in files of its own:
test/conf/conf.d/<name>.conf: a fragment included at the end of thehttp {}block. It holds its ownserver {}on a port of its own (18080 to 18199; 18116 is taken by the second nginx thattest/t/cases/_second_instance.rbstarts). Fragments forstream {}go totest/conf/conf.d/stream/(ports 12345 to 12399; 12357 and 12385 to 12388 are taken by the second nginx; 12393-12396 belong to the independent UDP session checks in_udp_wait_session.rb, including its echo upstream). A case ofmasterreachesnextby the merge, whereTEST_SH_PORT_OFFSETmoves only the ports in these ranges, so a new case takes a port that is free on both branches:git grepthe number onorigin/masterand onorigin/next. Ports below 18110 are allowed.test/t/cases/<name>.rb: a test file run aftertest/t/ngx_mruby.rb, withtest/t/cases/_prelude.rbprepended. The prelude definesbase,base_ssl,http_host,html_path,nginx_t(runsnginx -ton a generated configuration),tcp_clientandsecond_instance. It starts withSimpleTest.newand ends witht.report. Files whose name starts with_are helpers and are not run as cases;_tcp_client.rb,_filter_connection_client.rb,_request_body_client.rb,_second_instance.rb,_udp_wait_session.rband_redis_client.rbare CRuby scripts that the prelude helpers and the cases run withruby, and_wait_for_listen.rbis the CRuby script that test.sh runs to wait for nginx (see “Waiting for nginx to listen” below).- Hook scripts go to
test/html/as before; the fragment refers to them asbuild/nginx/html/<name>.rb.
test.sh copies the fragments into the built nginx’s conf/conf.d/ with the
same substitutions as nginx.conf, and runs every case file in turn. A
failing case makes test.sh exit non-zero, like a failing assertion in
test/t/ngx_mruby.rb.
A case that starts nginx itself (nginx_t, second_instance) also runs in
the sanitizer cell of CI, where LeakSanitizer appends its report to the
output of a process that leaked and changes that process’s exit status. Run
such a case once with the sanitizer build (see “Running the tests under
AddressSanitizer and UndefinedBehaviorSanitizer” below) before opening the
PR. nginx_t turns leak detection off for its nginx -t run, because
nginx -t returns without freeing the configuration it read, so every
allocation of nginx itself would be reported. The stream-only test at the end
of test/t/ngx_mruby.rb does the same for the nginx -s quit that stops the
nginx it started: that process returns from nginx’s main() without freeing
the pool that main() creates first, and its report would go to the test
runner’s stderr.log. second_instance returns the first sanitizer line of
the second nginx’s error.log or stderr.log as sanitizer, and the cases
print the second nginx’s error.log when the line is set.
Required status check
Section titled “Required status check”.github/workflows/test.yml runs on pull requests to master, next and
v*.x, on pushes to those branches and to v* tags, and on “Run workflow”.
Its ci-ok job is the status check that rulesets require: it fails unless
every job that it lists in needs passed or was skipped because the
changes job decided so. These are the changes job, every job of the
build matrix and the files job, and on next also the soak job. The
perf job (see “In CI” under “Performance comparison with callgrind”) is
not among them yet.
A run of test.yml for a pull request starts when the pull request is
opened or reopened, when a commit is pushed to its head, and when it is
marked ready for review (pull_request with the types opened,
synchronize, reopened and ready_for_review). A draft runs no job but
changes and ci-ok, and the job ci-ok is then named ci-ok (draft):
the run tests nothing, GitHub does not merge a draft (condition 7 of
“Review and merge” in AGENTS.md), and under its own name
the pass of that job leaves no ci-ok on the head commit, so the head has
no passing ci-ok until the run that marking the pull request ready
starts has passed. The other workflows that run on pull requests skip
their jobs for a draft in the same way. A new push to the head of a ready
pull request cancels the run of the previous head and starts a new one. A
move of the base alone starts no run; updating the branch
(gh api -X PUT repos/<owner>/<repo>/pulls/<number>/update-branch) pushes
a merge commit to the head and so starts one. The result stands for the
head commit and the base commit of the merge ref at that time, which is
what condition 2 of “Review and merge” asks the pull request to record;
that condition also says when a move of the base makes a new run needed.
A cancelled run of a draft reports no result. Marking a draft ready for
review starts the run that counts, and cancels the run of the draft when
that is still going (cancel-in-progress). In that cancelled run ci-ok
does not run: GitHub marks it cancelled, as it marks every job of the run
that had not finished, and a cancelled check is not a pass. The draft run
of #692 shows this
(run 41193944190);
before, ci-ok ran there on always() and failed, and the draft run of
#678 left a failed
ci-ok (draft) beside the passing ci-ok of the run that counted
(run 41191594780).
A job that GitHub cancels before it starts keeps the expression of its
name as its name, and gh pr checks lists a cancelled check as fail;
neither is the ci-ok of the run that counts.
In every other run ci-ok keeps always(): when the run is cancelled (by
a newer push, which leaves the failure on the head that the push
replaced, or by “Cancel workflow”) before a needed job has finished,
ci-ok runs and fails, because the run did not test the commit to the
end. GitHub’s documentation defines
cancelled() only as “the workflow was canceled” and does not say
whether a needed job that reaches its timeout-minutes makes it true; if
it did, ci-ok on !cancelled() would be skipped, and the skip would
pass the required check.
CI is the final check in a clean environment, not the place where a
change is first tested (“Branches and pull requests” in
AGENTS.md). The checks that CI runs for a change run
locally first: the check of the added files (Files that do not belong
in the repository), the
suite with the static and the dynamic module, the suite under the
sanitizers and under valgrind, the callgrind comparison with the check
of the mock LLM, the report of the heads of the GitHub gems when the
change touches build_config.rb, test/gem_heads.sh or
.github/workflows/gem-heads.yml (Heads of the GitHub
gems), the checks of the 2.x pages and
zizmor. The sanitizers,
valgrind and the comparison need Linux; on macOS they run in a Linux
container, as the sections of this file show. CI keeps for itself only
what one machine does not hold, the other cells of the build matrix
(with OpenSSL 1.1.1w built from source), the x86-64 runner and a clean
environment, to which it spreads the local checks, and the checks that
“Branches and pull requests” keeps in CI with their reason. These are
two: CodeQL (CodeQL), and the auto-ssl case for a change that
touches none of mrbgems/auto-ssl/, the Nginx::SSL code and
test/auto_ssl/: the case needs the services that
test/auto_ssl/deps.sh starts and skips without them, and every cell of
CI starts them (Testing the auto-ssl mrbgem). A failure is then found before the push that
would show it in CI, and when a run fails for a reason of the runner,
the local result of the same check shows whether the code is at fault.
Before marking a pull request ready, its author runs the local commands
of the checks that the change touches: sh test/check_added_files.sh
for every pull request, and when test/ci/changes.sh gives build
standard or full, the suite with the static and the dynamic module,
under the sanitizers and under valgrind, as the four standard cells run
it, and the suite with the static module on the oldest nginx of the build
matrix, which those cells do not run (Which jobs a run
starts); this file describes them, and “Build
and test” in AGENTS.md lists them.
The author quotes them under “How was this tested”, and the reviewer
checks that they are there. A check that can run locally is not left to
CI. The run of a ready pull request runs only the jobs that the
change needs (the tables below) and finishes in about fifteen minutes. On
2026-10-31, on next, a cell of the build matrix took 6 to 14 minutes, and
the run of test.yml with the whole matrix for #678 took 15 minutes from
its start to its end
(run 41191594773).
A proposal to add a check to CI says whether it can run locally, why CI
needs it, and how many minutes it adds to the run.
Which jobs a run starts
Section titled “Which jobs a run starts”test/ci/changes.sh BASE HEAD classifies the files that a change touches,
and the workflows that decide by the kind of change read it, so that the
policy is in that one file (sh test/ci/changes_test.sh checks it). The
script is the same on master and next. It prints one line per key: the
categories code, tests, docs, site, examples and workflows,
then soak and perf, each true or false, and build, the cells of
the build matrix: standard when code or tests is true, and
none otherwise. A renamed file counts under its old and its new path
(git diff --no-renames), so that moving a source file to docs/ still
builds. The job changes of test.yml also passes the script the run (a
draft, the labels of a ready pull request, a push, a tag, “Run workflow”
and its input valgrind), and the script gives build full, every
cell, or none for the runs of the second table below.
master has no soak job, site/ or examples/, so soak, site and
examples decide nothing here.
The jobs of the tables, except codeql, run on the runner the checks
that the author has already run locally (Required status
check); codeql runs only in CI
(CodeQL), and so does the auto-ssl case of the build cells
for a change outside its paths. gem-heads (.github/workflows/gem-heads.yml)
decides by its own path filter, not by test/ci/changes.sh: it runs on a
ready pull request that changes build_config.rb, test/gem_heads.sh or
the workflow, and its local counterpart is bash test/gem_heads.sh build_config.rb "" .claude/tmp/gem-heads.tsv (Heads of the GitHub
gems), which the author runs for such a
change.
| Category | Paths | What a ready pull request to master runs |
|---|---|---|
code |
src/, mrbgems/, mruby/, dependence/, build_config.rb, build_config.rb.lock, configure, configure.in, config.in, Makefile.in, build.sh, nginx_version, and every path that no other category names |
test.yml: the four standard cells of the build matrix (below) and perf. codeql for src/ and mrbgems/; gem-heads for build_config.rb |
tests |
test/, test.sh, .github/workflows/test.yml |
test.yml: the four standard cells; perf compares when perf is true. gem-heads for test/gem_heads.sh |
docs |
docs/, the Markdown files at the top, AUTHORS, LEGAL, LICENSE, MITL |
No job: the author checks the 2.x pages locally (The 2.x pages of the site) |
workflows |
.github/ except test.yml and site.yml |
zizmor (all the workflows, test.yml included); codeql for .github/workflows/ and .github/codeql/; gem-heads for .github/workflows/gem-heads.yml |
The four standard cells build each link mode and each line of nginx (mainline 1.31 and stable 1.30) at least once, and run both memory checks:
| Cell | nginx | Module | OpenSSL | nginx runs |
|---|---|---|---|---|
| nginx 1.31.8, static module | 1.31.8 | static | --with-openssl-src |
directly |
| nginx 1.30.6, dynamic module | 1.30.6 | dynamic | the installed OpenSSL | directly |
| the sanitizer cell | 1.31.8 | static | the installed OpenSSL | built with AddressSanitizer and UndefinedBehaviorSanitizer (Running the tests under AddressSanitizer and UndefinedBehaviorSanitizer) |
| the cell under valgrind | 1.31.8 | static | the installed OpenSSL | under valgrind (valgrind in CI) |
The CI policy adopted on 2026-11-01 runs these four cells on pull requests
instead of every cell. This covers both module modes, mainline and stable
nginx, and both memory checks while keeping pull request runs short.
Every cell builds OpenSSL 1.1.1w and installs it before it builds nginx;
the first cell builds it into nginx as well (--with-openssl-src), the path of
configure.in and Makefile.in that the cell under valgrind, the same
nginx and module against the installed OpenSSL, does not take. Every cell
(the 16 combinations of nginx 1.31.8, 1.30.6, 1.28.3 and 1.26.3, the
static and the dynamic module, and --with-openssl-src or not, among them
the cell under valgrind, and the sanitizer cell) still runs in the nightly
run under valgrind, on the push of a tag, before a tag on the tip of the
base, in the run of the check pull request, which has the label
full-matrix, or with “Run workflow” on the base when the check needs no
pull request (“Before the tag” in
docs/releases/README.md), and on “Run workflow”
without the input base (the second table). A change that only one of the
other cells can show (for example a change for nginx 1.26.3) is checked
locally with that version (nginx_version, or the NGINX_VERSION of the
cell, with BUILD_DYNAMIC_MODULE), and the pull request can have the
label full-matrix to run every cell.
A pull request for which test/ci/changes.sh gives build standard or
full also runs, locally and at its head, the suite
with the static module on the oldest nginx of the build matrix, the oldest
version of the list NGINX_VERSION of full_matrix in
.github/workflows/test.yml (1.26.3 now). The four standard cells do not
include that version; only the nightly run and the runs of every cell do.
nginx accepts different configurations and behaves differently in
different versions (before 1.27.1, nginx refuses a stream server without
a content handler, which 1.30.6 and 1.31.8 accept). A test configuration
under test/conf/, a case that writes its own configuration (under
test/t/cases/, for example) and a change of the code can each depend on
the version, so the run is not limited to some paths: without it, such a
difference would first fail in the run of every cell before a tag. The
cells of CI switch the version by writing nginx_version from their
NGINX_VERSION; locally, write it the same way in a checkout of its own
(a git archive of the head), so that the change of nginx_version is
not committed, and run test.sh there without ONLY_BUILD_NGX_MRUBY
(Testing says why):
$ V=1.26.3$ printf 'NGINX_SRC_MAJOR=%s\nNGINX_SRC_MINOR=%s\nNGINX_SRC_PATCH=%s\nNGINX_SRC_VER=nginx-%s\n' $(echo $V | tr . ' ') $V > nginx_version$ sh test.shperf is true when code is, or when the measurement of the performance
comparison changed (test/perf/, test/build_release.sh and the scenario
files of test/soak/). A file of no category counts as code, so a file
of a new kind runs everything until a pull request names it in the script,
with the reason. A pull request that changes several categories runs what
each of them runs.
| Run | build and perf |
|---|---|
| Draft pull request | Skipped, and files too; the job is named ci-ok (draft) |
| Ready pull request | By the categories of the files that its merge commit (refs/pull/N/merge) changes against its first parent, the base: the table above. perf starts when build is standard or full, and compares when perf is true or the pull request has the label perf |
Ready pull request with the label full-matrix |
Every cell, whatever the files; perf as without the label |
Push to master, next or a v*.x branch |
Skipped |
Push of a v* tag |
Every cell (perf runs on pull requests only) |
“Run workflow” without the input base, or with the input valgrind (the nightly run) |
Every cell (perf runs on pull requests only) |
“Run workflow” with the input base alone |
As for a pull request against base, without labels (perf runs on pull requests only) |
The job changes reads the labels from the event of the run. A label
added to a ready pull request counts from its next run, which a push to
its head starts, or marking it a draft and ready again (gh pr ready --undo, then gh pr ready); a re-run of the jobs keeps the event, and so
the labels it had. The label full-matrix goes on the pull request when it
is opened (gh pr create --draft --label full-matrix), so that marking it
ready starts the run with every cell; gh label create full-matrix adds
the label to a repository that has none.
A change of test.yml alone is tests, so its run starts the four
standard cells, but not every cell or the comparison of perf. A pull
request that changes those jobs in test.yml runs them with the label
full-matrix, with “Run workflow” on its branch (gh workflow run test.yml --ref <branch>), whose job is named ci-ok (manual), or the
comparison with the label perf.
The pull request quotes that run as evidence; it is not the run that
condition 2 of “Review and merge” counts, which is the run of the pull
request.
For “Run workflow” with the input base (a branch or a commit id), the
files are those that the branch changes since its merge base with base,
so that a branch can show what its pull request would run:
$ gh workflow run test.yml --ref claude/topic -f base=masterThe result of such a run is the job ci-ok (dry run), not ci-ok: the
person who starts the run chooses base, so its result is not the check
that the rulesets require of the commit. In the same way, a run with the
input valgrind names it ci-ok (valgrind): such a run builds the tip of
the branch when it starts, so a run that is cancelled would otherwise
leave a failed ci-ok on that commit. Every other “Run workflow” names it
ci-ok (manual): it tests the head of the branch, not the merge commit of
a pull request, and as ci-ok on the head it would pass the required
check of a pull request before the pull request’s own run has passed.
valgrind-nightly.yml reads the result of the run, not the name of the
job.
A push to master or next runs only changes, files and ci-ok. Both
branches take changes only through pull requests, and condition 2 of
“Review and merge” in AGENTS.md has the session merge a
pull request only while the run of the pull request still stands for the
merge: when the base has moved since that run, CI runs again only when both
the pull request and the base’s new commits change code or tests. The rulesets require ci-ok, but not that the
branch is up to date with its base. To build and test the tip of a branch
by itself, use “Run workflow” (gh workflow run test.yml --ref master), as
after a move of the head of a GitHub gem (Heads of the GitHub
gems); the nightly run under valgrind is such
a run on master and next (valgrind in CI). The other
workflows decide on their own when they run: site-dispatch starts the
build and deploy of the site on next after every push to master,
codeql scans pushes to master and next that change src/,
mrbgems/, the workflows or its configuration (CodeQL), and
zizmor and gem-heads run on ready pull requests (and gem-heads
weekly), and site-v2 only on “Run workflow”.
ci-ok passes a skipped build or files job only when changes passed
and decided that the run does not need it. A job that was skipped for
another reason (a failed changes job) fails ci-ok, as does a needed job
that did not pass. A pull request whose run skipped jobs that its files do
not need meets condition 2 of “Review and merge” with that run.
Files that do not belong in the repository
Section titled “Files that do not belong in the repository”The files job runs test/check_added_files.sh BASE HEAD TIP. The script
checks two lists of files:
- the files that the change adds: the paths that
HEADhas andBASEdoes not have (git diff --no-renames --diff-filter=A, so a file that is renamed or moved counts as added at its new path); - the files that its commits add: each commit from
TIPback along first parents toBASE(git log --first-parent BASE..TIP), compared with its first parent. A file that one commit adds and a later commit removes stays in the history after the merge, so it is checked as well.
A file of either list fails with exit status 1 when it matches a pattern of
test/forbidden_paths.txt: a session directory or the local settings of an AI
coding agent under .claude/ (everything there except
.claude/settings.json, .claude/commands/ and .claude/skills/, which also
fail below another .claude/), a build directory at the top (build/,
build_dynamic/, build_soak/, build_perf/) or mruby/build/, a file
that ./configure or the Makefile writes at the top (Makefile, config,
config.status, autom4te.cache/, mrbgems_config, .mruby_version, the
.openssl_*_done files), .vagrant/, a tag file of GNU Global or ctags, a
.log file, .env or .env.*, private key material, an editor or OS
leftover, node_modules/, __pycache__/ or a .pyc file, or a core dump. A
file of the first list also fails when it is empty (0 bytes), unless its name
is .gitkeep or .keep. Both lists include submodule entries (gitlinks),
whatever the git configuration or .gitmodules says about ignoring
submodules.
The script prints each failing path with the pattern and its line in the pattern file, or with “empty file”, and with the commit that added it. When every file passes, it prints how many files it checked and exits 0. It exits 2 when the pattern file is missing, a pattern is invalid, a revision is not in the repository, the repository is a shallow clone or a git command fails. It reads only the pattern file and the git repository.
The commits follow first parents only. A merge commit among them is compared
with its first parent, so the files that its second parent brings in are
checked at the paths they have in the repository, and the commits of the
second parent are not checked one by one. This keeps out the squashed commit
of a subtree merge (./update-mruby-subtree), whose paths lack the mruby/
prefix. It also means that the check does not see a file that a commit of a
merged side branch adds and a later commit of that side branch removes.
For a pull request, the job checks out the merge commit of the pull request
with the whole history (fetch-depth: 0). It compares the merge commit with
its first parent, the base branch that the pull request was merged into, and
checks the commits from the head of the pull request (the second parent) back
to that base. So a pull request whose branch is behind its base is not blamed
for the files that the base removed in the meantime. For a push to a branch,
the job compares the previous tip of the branch (github.event.before) with
the pushed commit, and checks the commits between them. A tag push, a push
that creates a branch and a manual run have nothing to compare, and the job
passes. A pull request that changes the script or the pattern file is checked
with its own version of them, so the review of the pull request covers that
change.
test/forbidden_paths.txt holds one POSIX extended regular expression (as
grep -E reads it) per line, matched against the path from the top of the
repository. A line that starts with ! is an exception, and the last line
that a path matches decides, as in .gitignore: .claude/skills/a/SKILL.md
matches (^|/)\.claude/ and then the exception !(^|/)\.claude/skills/, so
it passes, while .claude/skills/a/.DS_Store also matches (^|/)\.DS_Store$
further down and fails. Empty lines and lines that start with # are
comments. The patterns must be generic: no names of other projects, people or
hosts.
To run the check before opening a pull request, commit the change and compare
it with the base branch (origin/next for a change to next). With two
arguments, the script also checks the commits from HEAD back to the base.
It needs the whole history, so run it in a clone that is not shallow:
$ git fetch origin master$ sh test/check_added_files.sh origin/master HEADcheck_added_files.sh: 2 added file(s) checked between origin/master and HEAD and in the 2 commit(s) of origin/master..HEAD, none forbidden or emptyWhen a file fails, remove it from the change (git rm --cached <path> keeps
it on disk), or, if it belongs in the repository, change
test/forbidden_paths.txt (for an empty file, the script) in the same pull
request and say why in its description. A file that a commit adds and a later
commit removes stays in the history: remove it from the commit that added it
(for example with git rebase -i), or squash the commits.
The 2.x pages of the site
Section titled “The 2.x pages of the site”The site ngx.mruby.org is built on next (site/ there, and
docs/proposals/site.md there for the plan). Its 2.x pages under /v2/
are built from docs/ of this branch. A pull request to master runs no
job for them. The CI policy adopted on 2026-11-02 keeps checks that run
locally out of pull request CI; the site’s build and deployment run on
next after pushes to next or master. The author of a pull request
that changes docs/, the Markdown files at the top, or a file that they
link to, runs these checks locally before marking it ready, and quotes
them under “How was this tested”:
-
The relative links and anchors of the Markdown files that git tracks, with
check_links.pyofnext, whichmasterdoes not have. From the top of the checkout of the pull request, withTa directory of your own outside the repository or under.claude/tmp/:$ git fetch origin next$ git show origin/next:site/scripts/check_links.py > "$T/check_links.py"$ git ls-files -z -- '*.md' ':!:mruby/' ':!:dependence/' | xargs -0 python3 "$T/check_links.py" . -
The 2.x pages built from the pull request: in a checkout of
next, with the Node.js ofsite/.node-versionin that checkout (on a machine with another version, a version manager ornpx -p node@<that version>gives it), afternpm ci --ignore-scriptsandnpx playwright install chromium webkitin itssite/,NGX_MRUBY_V2_TREE=DIR npm run check:ciwithDIRthe top of the checkout of the pull request (“Site CI” indocs/test/README.mdonnext). It builds the 3.x pages ofnextand the 2.x pages fromDIR, and runs the checks of the site on both, the browser checks among them; it compiles the Ruby blocks of the pages only whenMRBCnames anmrbcof the bundled mruby, and otherwise says that it skipped them. It prints the commit ofnext; quote it with the head of the pull request. The generator stops on a link ofdocs/to a file that git does not track, and the checks fail on a broken link of the built site, on a 2.x page without its banner, and when the search of a version shows pages of the other.The release notes in
docs/releases/(vX.Y.Z.mdandvX.Y.Z.ja.md) are also 3.x pages (/releases/...and/ja/releases/...) oncemasteris merged intonext. This check builds the 3.x pages fromdocs/of the checkout ofnext, so it does not check a release note that the pull request adds or changes as a 3.x page, and the checks of the Japanese pages ofnext(the spaces between Japanese and Latin text, the code blocks, the line breaks) do not read that file. A pull request that adds or changes a release note therefore copies that file intodocs/releases/of the checkout ofnextbefore the run, and commits it there in a local commit that is never pushed:$ git add docs/releases/vX.Y.Z.md docs/releases/vX.Y.Z.ja.md$ git commit -m 'Release notes to check (local only, not pushed)'The generator reads only the files that git tracks, and
site/scripts/export-docs.mjsstops withsource must exist at HEAD before exporton a page whose source is not in the commitHEAD, so staging alone stops the build for a new file, while a change of a file thatHEADalready has would go through; the commit serves both. One run then checks the file as a 2.x page and as a 3.x page.check:ciprints the local commit: quote it with its parent, the commit ofnext. Afterwards, remove that checkout or reset it toorigin/next. This keeps the push tonextafter the merge ofmasterfrom stopping the deploy of the site on those checks.
Two workflows here serve the 2.x pages:
.github/workflows/site-dispatch.ymlruns on every push tomaster, and startssite.ymlonnextwithgh workflow run site.yml --ref next, so that the site is built again with the new 2.x pages, checked, and deployed only when the checks pass. It fails when GitHub refuses the dispatch or it cannot name the run that the dispatch started..github/workflows/site-v2.ymlruns only on “Run workflow” (gh workflow run site-v2.yml --ref <branch>). It checks outnextbeside the branch, builds the site withsite/and the 3.x pages ofnextand the 2.x pages from the branch’sdocs/(node site/scripts/build-v2.mjs --tree DIRofnext), and runs the link, banner and search checks above. The built site is the workflow artifactsite-v2. It does not run on pull requests or pushes: on a push, the run ofsite.ymlthatsite-dispatch.ymlstarts builds and checks the same 2.x pages. That run checks the links of the Markdown files ofnext, not ofmaster: the links of the Markdown files ofmasterthat the site does not render (AGENTS.md, for example, whose links the generator turns into GitHub URLs) are checked only by the author’s run ofcheck_links.pyabove.
Neither is in the needs of ci-ok.
Supply chain checks
Section titled “Supply chain checks”The badge above is the project’s OpenSSF Best Practices badge (passing level), which is separate from the Scorecard results below.
Four more workflows check the supply chain of the build and the code. None
of them is in the needs of ci-ok, so none of them blocks a merge through
the ruleset; “Review and merge” in AGENTS.md asks for every CI check on
the head commit to pass, which includes the zizmor and codeql jobs of a
pull request that runs them. Like test.yml, the ones that run on pull
requests skip their jobs for a draft, and marking it ready for review
starts them.
Static analysis of the workflows
Section titled “Static analysis of the workflows”.github/workflows/zizmor.yml runs zizmor on
ready pull requests that change .github/workflows/,
.github/dependabot.yml or .github/zizmor.yml, and on “Run workflow”;
pushes do not run it, because the run of the pull request checked the same
files. It
audits the files under .github/ with the default persona, including the
audits that query GitHub (for example known vulnerable actions and commits
that are not on the action’s repository), and fails on any finding. The
vendored mruby/ tree has workflows of its own, which ngx_mruby does not
run; they are not audited. The version of zizmor is pinned in the workflow
(version:): Dependabot updates the pin of the action but not that input.
A finding is fixed in the workflow, or accepted with its reason: by a
# zizmor: ignore[<audit>] comment on a line of the finding, with the
reason in a comment above it, or by an ignore rule in .github/zizmor.yml
with the reason next to it. To run the same check before opening a pull
request (the online audits need a GitHub token):
$ python3 -m venv .claude/tmp/zizmor && .claude/tmp/zizmor/bin/pip install zizmor==1.31.1$ GH_TOKEN=$(gh auth token) .claude/tmp/zizmor/bin/zizmor --config .github/zizmor.yml .githubOpenSSF Scorecard
Section titled “OpenSSF Scorecard”.github/workflows/scorecard.yml runs
OpenSSF Scorecard on pushes to
master, weekly, and on “Run workflow”. Scorecard supports only the default
branch, so pull requests do not run it. It publishes the results to the Scorecard API
(https://api.scorecard.dev/projects/github.com/matsumotory/ngx_mruby),
uploads them to code scanning (the repository’s Security tab), and keeps the
SARIF file as an artifact of the run for 5 days. Publishing makes the
Scorecard API check the workflow itself, so the workflow has no run steps and
no defaults; the comment at its top lists the restrictions.
GitHub disables the scheduled workflows of a public repository after 60 days
without activity in the repository. Then neither this workflow nor the next
one runs on its schedule, and nothing fails. “Run workflow”, or enabling the
workflow again (gh workflow enable scorecard.yml, gh workflow enable gem-heads.yml), starts them again.
CodeQL
Section titled “CodeQL”.github/workflows/codeql.yml runs CodeQL
code scanning for three languages, c-cpp, ruby and actions, and
uploads the results to code scanning (the repository’s Security tab). It
runs on ready pull requests to master and next and on pushes to these
branches that change src/, mrbgems/, .github/workflows/ or
.github/codeql/, every Wednesday at 04:37 UTC (GitHub runs a schedule
only from this file on master, so the weekly run scans master), and on
“Run workflow”. It
replaces GitHub’s default setup of code scanning, which ran on every pull
request and every push; the default setup is turned off in the repository
settings, which the owner manages.
All three languages use build-mode: none: for C and C++ CodeQL reads the
sources without a build of nginx and mruby, so the job needs no build.
CodeQL’s documentation says that a database made without a build can give
less accurate results than one made from a build
(“Build options for compiled languages”).
The actions are pinned by commit id like those of the other workflows, and
zizmor checks the file.
CodeQL is one of the two checks that “Branches and pull requests” in AGENTS.md keeps in CI (the other is the auto-ssl case for a change outside its paths): an author does not run it before marking a pull request ready. The job uploads its results to code scanning, and GitHub shows the new alerts on the lines that the pull request changes as annotations in the pull request (“Triaging code scanning alerts in pull requests”). A local run would need the CodeQL bundle, about 550 MB for Linux on arm64 in the version that the pinned action uses by default (2.27.3), and the system requirements of CodeQL list Linux on arm64 and Apple Silicon as beta (System requirements). CodeQL has not been run locally for this repository.
CodeQL does not analyze the vendored trees, mruby/ (mruby, with the code
that mruby itself vendors) and dependence/ (ngx_.github/codeql/codeql-config.yml, which the init step of the workflow
reads (config-file), lists them under paths-ignore. They are upstream
code that the repository takes with git subtree and does not edit by hand
(“Do not” in AGENTS.md), so an alert in them would not
be fixed here; a change of them comes with the next update of the subtree.
The owner decided on 2026-10-31 that CodeQL leaves them out, that the
alerts that code scanning had on them are dismissed as vendored upstream
code, and that nothing is reported upstream for them. GitHub’s
documentation says that paths-ignore applies to the interpreted
languages, Ruby among them, and to C and C++ analyzed without a build
(“Workflow configuration options for code scanning”).
Heads of the GitHub gems
Section titled “Heads of the GitHub gems”.github/workflows/gem-heads.yml runs test/gem_heads.sh every Monday, on
“Run workflow”, and on ready pull requests that change build_config.rb,
the script or the workflow. For each conf.gem github: line of
build_config.rb, it reports the head of the branch that a build clones
(the line’s branch:, or master, mruby’s default when the line has
none), whether the gem’s default branch is another branch, whether the
repository was renamed or archived, and whether the head moved since the
previous report. The conf.gem mgem: line (mruby-secure-random) is not
checked: mruby resolves it through the mgem-list repository.
The previous report is the artifact gem-heads of the newest successful
scheduled or manual run of the workflow on master; artifacts are kept for
90 days. master commits no build_config.rb.lock, so this report is the
only record of the heads that earlier builds took. A run with no previous
report reports every gem as new. A moved head is a warning annotation and
a row in the job summary, with a link that compares the two commits; the job
still passes. A gem whose repository or branch cannot be read fails the job,
and so does a failure to list the earlier runs. A previous report that cannot
be downloaded is a warning, and every gem is then reported as new.
To run the script on a checkout (it needs gh with a token):
$ bash test/gem_heads.sh build_config.rb "" .claude/tmp/gem-heads.tsvWhen a run reports a move, a session does the following. This is the sessions’ procedure, not a decision of the owner.
- Read the commits behind the comparison link. Every build of
master, CI included, and every source build of a released 2.x tag already takes the new head; CI does not run for a move by itself, so run thetestworkflow onmaster(gh workflow run test.yml --ref master) and read its result. - When the new head breaks the build or the suite, or changes what a 2.x
configuration, script or build observes, report it to the owner with the
run, and propose a pull request to
master: a fix, or achecksum_hash:that pins the gem to the last good commit, with its release notes entry (seedocs/releases/README.md). For the owner’s gems (matsumotory/mruby-*), “Other repositories” inAGENTS.mdapplies. - On
next,build_config.rb.lockpins each gem, so the move reachesnextonly through a pull request that moves the pin; theperfjob measures that pull request like any change of the lock.
Testing
Section titled “Testing”build nginx into ./build/nginx and test on ./build/nginx
Section titled “build nginx into ./build/nginx and test on ./build/nginx”sh test.shtest.sh configures nginx with --with-debug, --with-http_stub_status_module,
--with-http_ssl_module, --with-http_auth_request_module and --with-stream
(see NGINX_DEFAULT_OPT and NGINX_CONFIG_OPT in test.sh), so the test
configuration can use the directives of these modules. CI runs the same
test.sh on a fresh checkout. A local build directory keeps the options that
nginx was first configured with: the Makefile configures nginx again only when
build/nginx-X.Y.Z/objs/Makefile does not exist (build_dynamic/... for
BUILD_DYNAMIC_MODULE). After these options change, remove that file and run
test.sh once without ONLY_BUILD_NGX_MRUBY; otherwise nginx stops at the
test configuration with an “unknown directive” error. After changing
nginx_version or BUILD_DYNAMIC_MODULE, run test.sh once without
ONLY_BUILD_NGX_MRUBY as well: it then uses another build directory
(build/nginx-X.Y.Z of the new version, or build_dynamic/), and configures
nginx there when that file is missing in it.
To run nginx under valgrind, set the environment variable NGINX_RUNNER:
$ NGINX_RUNNER=valgrind sh test.shvalgrind writes its report when nginx exits at the end of test.sh, and
test.sh does not take valgrind’s errors into its exit status. To check a
run as CI does, give valgrind the options of CI and read the ERROR SUMMARY
line of the log (see valgrind in CI):
$ rm -rf build/valgrind && mkdir -p build/valgrind$ NGINX_RUNNER="valgrind --error-exitcode=99 --leak-check=full --show-leak-kinds=definite,indirect,possible --errors-for-leak-kinds=definite,indirect --suppressions=$PWD/test/valgrind.supp --log-file=$PWD/build/valgrind/valgrind.%p.log" sh test.sh$ grep 'ERROR SUMMARY' build/valgrind/valgrind.*.logWaiting for nginx to listen
Section titled “Waiting for nginx to listen”test.sh starts the test nginx in the background ($NGINX_RUNNER build/nginx/sbin/nginx &) and sends the first request only after nginx
accepts TCP connections. It runs test/t/cases/_wait_for_listen.rb with
CRuby twice:
-
Before it starts nginx, the script waits until no process accepts a connection on the ports below. A port that another process listens on would make the wait after the start succeed without nginx, so the run fails when one still accepts after
NGINX_HEATTIMEseconds, with the ports:wait_for_listen: before nginx starts, 1 of 33 ports still accept connections after 60.05 seconds (limit 60.00): 18080; another process listens on them -
After it starts nginx, the script connects to each port every 0.1 seconds until every port has accepted one connection, and prints the time that took, counted from the start of the script, right after nginx was started:
wait_for_listen: nginx (pid 1234) accepted connections on all 33 ports after 0.21 seconds (limit 60.00)It fails when the nginx process ends first, or when
NGINX_HEATTIMEseconds pass, and names the ports that still refused connections:wait_for_listen: nginx (pid 1234) ended after 0.47 seconds, before it accepted connections on 33 of 33 ports: 18080, 18081, ...; see its output above and logs/error.logwait_for_listen: nginx (pid 1234) did not accept connections within 60.00 seconds (NGINX_HEATTIME) on 33 of 33 ports: 18080, 18081, ...
The ports are those of every listen directive in test/conf/nginx.conf
and test/conf/conf.d/*.conf, the http {} servers (36 ports as of
2026-10). nginx opens all its listening sockets in one loop
(ngx_open_listening_sockets()) after it has read the whole configuration,
so one port would almost always show that the others are open. Each port
costs one connection of well under a millisecond, though, and checking every
one leaves no gap between the first socket and the last, and also catches a
fragment of conf.d that did not reach the installed configuration. The
stream {} ports (test/conf/nginx.stream.conf and
test/conf/conf.d/stream/) are left out: a connection to a stream server
runs that server’s session code, such as the Nginx::Stream::Async.sleep 3000 of the 12352 server, and the cases read what that code writes to
error.log. nginx opens the stream sockets in the same loop, right after
those of http {}, and the first test that connects to a stream port
runs after many HTTP tests of test/t/ngx_mruby.rb have passed. The ports that
Nginx::Server#add_listener and Nginx::Stream.add_listener add are opened
in the same loop as well, and are not checked.
NGINX_HEATTIME is the upper bound, in seconds, of each of the two waits.
The default is 60, and CI does not set it. A command that sets it, such as
NGINX_RUNNER=valgrind NGINX_HEATTIME=10 sh test.sh, waits at most that
long: the run goes on as soon as nginx listens, and fails at 10 seconds only
when nginx is not listening by then, where a first request sent at that time
would have been refused (Errno::ECONNREFUSED).
The default rests on these measurements. On 2026-10-19, the test nginx
built from master at 03aad7031 (nginx 1.31.6, static module) was started
the way test.sh starts it, in a container of Ubuntu 22.04 on an aarch64
host with 11 CPUs whose load average was about 15 from other builds, and
the script above measured the time until all ports accepted connections:
NGINX_RUNNER=exec, 10 starts: 0.10 to 0.11 seconds each, that is, every port accepted on the second round of attempts (the script tries every 0.1 seconds);NGINX_RUNNER=valgrind(valgrind 3.18.1), 5 starts: 2.27 to 2.61 seconds.
In CI, a cell under valgrind (nginx 1.28.3 on an x86_64 runner) was still
reading the configuration 10 seconds after the start: valgrind’s report at
the SIGTERM of test.sh shows nginx inside ngx_conf_parse()
(the run,
reported in
#564).
The time is longer on a loaded CI runner, under valgrind (here more than 20
times the time without it), with the sanitizer build (not measured here) and
with more configuration to read. 60 seconds is 6 times the 10 seconds of
that CI run and about 23 times the slowest start under valgrind above.
The upper bound can create these races:
- A start slower than the bound fails the run although nginx would have
started. Raise
NGINX_HEATTIMEfor slower setups; the wait still ends as soon as nginx listens, so a higher bound costs time only when nginx hangs without ending (an nginx that exits is reported at once). - A listening socket is not yet a running handler. The kernel accepts a
connection as soon as nginx listens, and the test nginx is a single
process (
master_process off) that runs theinit_workerhooks before it reads any connection. The first request then waits in the socket’s backlog until nginx reads it; the test client (mruby-simplehttp) reads without a timeout, so it waits too. - nginx later reads each probe connection of the wait, finds it closed
without a request, and closes it. No handler runs for it, and nginx writes
only
debuglines about it toerror.log.
valgrind in CI
Section titled “valgrind in CI”The build matrix of .github/workflows/test.yml runs test.sh in each
cell. Which cells run nginx under valgrind depends on the run:
-
In a run that starts the
buildmatrix, every cell or the four standard cells (Which jobs a run starts), one cell runs nginx under valgrind: nginx 1.31.8, static module, without--with-openssl-src(the matrix gives itVALGRIND_ON_EVERY_RUN; in the whole matrix it is the jobbuild (1.31.8)). The other cells without a sanitizer run nginx directly (NGINX_RUNNER=exec), and the sanitizer cell runs nginx built with AddressSanitizer and UndefinedBehaviorSanitizer (see Running the tests under AddressSanitizer and UndefinedBehaviorSanitizer). -
A manual run of
test.ymlwith the inputvalgrindruns every cell except the sanitizer cell under valgrind:$ gh workflow run test.yml --ref master -f valgrind=true -
The nightly run:
.github/workflows/valgrind-nightly.ymlstarts such a run oftest.ymlevery day at 18:23 UTC (03:23 JST) for each branch of its matrix (masterandnext), waits for each, and fails when one fails. GitHub runs a schedule only from the workflow file on the default branch,master, so the file onmasterstarts the runs of both branches. Pull requests do not start it: a change oftest.ymlortest/valgrind.suppruns the cell under valgrind in its own run, and every cell under valgrind waits for the next night, or for “Run workflow” ofvalgrind-nightly.ymlor oftest.ymlwithvalgrindon the branch. The run does not give the status checkci-ok: its job is namedci-ok (valgrind)(Which jobs a run starts).
A cell under valgrind runs nginx with these options:
valgrind --error-exitcode=99 --leak-check=full --show-leak-kinds=definite,indirect,possible --errors-for-leak-kinds=definite,indirect --suppressions=test/valgrind.supp --log-file=$RUNNER_TEMP/valgrind/valgrind.%p.logWhen test/ci/changes.sh gives build standard or full (a change
of the code or of the tests, as for the four standard cells), the author
runs the suite under valgrind with these options before marking the pull
request ready (Testing has the commands on Linux); the cell
under valgrind repeats it on the x86-64 runner. Run it in a checkout of its own (for
example git archive HEAD | tar -x -C DIR), because a sanitizer build in
the same checkout leaves mruby built with other flags. valgrind needs
Linux; on macOS, run it in a Linux container, from the top of that
checkout:
$ docker run --rm --init -v "$PWD":/work -w /work ubuntu:22.04 sh -c 'apt-get -qq update && DEBIAN_FRONTEND=noninteractive apt-get -qq install -y build-essential rake ruby bison git gperf wget ca-certificates curl zlib1g-dev libpcre3-dev libssl-dev valgrind >/dev/null && mkdir -p build/valgrind && NGINX_RUNNER="valgrind --error-exitcode=99 --leak-check=full --show-leak-kinds=definite,indirect,possible --errors-for-leak-kinds=definite,indirect --suppressions=$PWD/test/valgrind.supp --log-file=$PWD/build/valgrind/valgrind.%p.log" sh test.sh && grep -h "ERROR SUMMARY" build/valgrind/valgrind.*.log | sed "s/^==[0-9]*== //" | sort | uniq -c'On Linux, run the part inside the quotes after the installation of the
packages. The last command prints each distinct ERROR SUMMARY line with
the number of logs that have it; every line has to count 0 errors, and
at least one log has to have the line, as the step “Check the valgrind
reports” below requires. The SSL cases of the suite request with curl,
so the container installs it. --init makes a process of the container
reap the processes that the tests leave when they exit.
This form, and the form of the sanitizer section, differ from the cells
of CI in two ways, and the cells cover the difference. They build
against the OpenSSL of Ubuntu 22.04 (its libssl-dev, OpenSSL 3.0.2),
while every cell removes libssl-dev and builds against OpenSSL 1.1.1w
built from source (Which jobs a run starts).
They do not start the services of the auto-ssl case, so
test/t/cases/auto_ssl.rb skips, while every cell runs it. A change of
mrbgems/auto-ssl/, of the Nginx::SSL code
(src/http/ngx_http_mruby_ssl.c and .h) or of test/auto_ssl/ also
runs that case locally with its services, as Testing the auto-ssl
mrbgem says: in a container, start the
container first with --init and a name and with the packages that
section names, start the services with
AUTO_SSL_DOCKER_NETWORK=container:NAME, and set
NGX_MRUBY_TEST_AUTO_SSL to their directory as the container sees it.
valgrind counts an error for each memory error that occurs (an invalid
read or write, a conditional jump or a system call on an uninitialised
value, an invalid or mismatched free) and for each loss record of the
kinds definite and indirect that test/valgrind.supp does not
suppress, and prints the count, with the number of distinct stacks
(contexts), in its ERROR SUMMARY line when nginx exits. With
--error-exitcode=99, valgrind also exits with 99 when the count is not 0,
but test.sh does not keep the exit status of nginx in its own, so the
step “Check the valgrind reports” reads the logs instead: it prints each
log, and fails unless at least one log has an ERROR SUMMARY and every
ERROR SUMMARY counts 0 errors. A cell without any ERROR SUMMARY fails
as well: valgrind did not finish its report.
Possibly lost blocks are printed with their stacks, but are not errors. At
the exit of the test nginx (master_process off), nginx has destroyed the
pool of the cycle, and the arrays of connections and events that
ngx_event_process_init() allocated, the table of ngx_crc32_table_init()
and the pools of the connections that are still open are left. valgrind
finds only interior pointers to them and reports them as possibly lost.
Their number depends on the connections that are open when nginx exits,
so counting them would make a cell fail by timing. A leak of ngx_mruby,
mruby or a gem that drops the last pointer to a block is definitely lost,
and fails the cell. A request or connection that ngx_mruby never finalizes
keeps its pools until nginx exits, and valgrind reports them as possibly
lost, so this check does not fail on it; master has no check for such
leaks yet (on next, the soak job compares the connections, timers and
file descriptors of the worker between samples).
test/valgrind.supp suppresses only allocations of nginx itself, each with
its stack and the reason in a comment: today the environment array of
ngx_set_environment() (8 bytes, definitely lost in every cell), which
nginx leaves allocated at exit on purpose (ngx_cleanup_environment() in
nginx.c). An entry is added only with the stack of the report and the
reason why the block is not ngx_mruby’s; a leak with a frame of ngx_mruby,
mruby or a gem is not suppressed.
Testing the auto-ssl mrbgem
Section titled “Testing the auto-ssl mrbgem”test/t/cases/auto_ssl.rb tests Nginx::SSL::ACME::Dehydrated and
Nginx::SSL::ACME::Client (mrbgems/auto-ssl) with real services: Redis, the
ACME test server Pebble and the
dehydrated client.
test/auto_ssl/deps.sh provides them:
- Redis 7.4 on
127.0.0.1:6379; - Pebble with its ACME API on
127.0.0.1:14000and its management API on127.0.0.1:15000. It validates the http-01 challenges on port 18174; - a second Pebble that requires external account binding, with its ACME
API on
127.0.0.1:14001and its management API on127.0.0.1:15001, so that every account registration there fails; pebble-challtestsrvas Pebble’s DNS server on127.0.0.1:8053(its management API on127.0.0.1:8055), which answers every A query with127.0.0.1and no AAAA query, so that Pebble connects to the test nginx;- dehydrated 0.7.2, downloaded from its GitHub release and checked against its SHA-256.
The services run in Docker containers whose images the script pins by
digest. They use the host network by default, because Pebble has to connect
to the nginx that the case starts on 127.0.0.1, and test.sh has to reach
Redis and Pebble there. The case also needs curl, the openssl command,
hexdump (for dehydrated; package bsdextrautils on Ubuntu 22.04) and CRuby.
It runs only when NGX_MRUBY_TEST_AUTO_SSL names the directory that
deps.sh start prepared:
$ sh test/auto_ssl/deps.sh start /tmp/auto-ssl$ NGX_MRUBY_TEST_AUTO_SSL=/tmp/auto-ssl sh test.sh$ sh test/auto_ssl/deps.sh stopWithout NGX_MRUBY_TEST_AUTO_SSL, or in a build without the gem, the case
prints why it skips and checks nothing. CI starts the services in every cell
of the build matrix before test.sh (“Start the services of the auto-ssl
tests” in .github/workflows/test.yml), so the case runs there.
To run test.sh in a container, start that container first and let the
services share its network namespace: AUTO_SSL_DOCKER_NETWORK=container:NAME
makes deps.sh start them with --network container:NAME.
AUTO_SSL_CONTAINER_PREFIX changes the names of the containers
(ngx-mruby-auto-ssl-redis, -challtestsrv, -pebble and -pebble-eab
by default), so
that several runs on one machine do not take the same names; deps.sh stop
removes only the containers with that prefix. Start the test container with
--init: the case and test/t/cases/config_hooks.rb start an nginx as a
daemon and wait until it has exited, and without an init process its exited
process stays a zombie that kill -0 still finds.
The test nginx gets its certificate in the TLS handshake on 18172
(test/conf/conf.d/78-auto-ssl.conf), as
test/conf/auto-ssl/nginx.conf.dehydrated_example does:
Nginx::SSL::ACME::Dehydrated#auto_cert_deploy runs dehydrated and waits for
it. The test nginx runs as a single process, so it cannot answer the requests
that dehydrated and Pebble make in that time. The case therefore starts a
second nginx (test/auto_ssl/peer.conf, prefix
NGINX_INSTALL_DIR/auto_ssl/peer/, error.log in
NGINX_INSTALL_DIR/logs/auto_ssl_peer.error.log) with the hook server of
test/auto_ssl/hook.sh on 18173 and the http-01 challenge server on 18174.
The account of dehydrated is new in each run, so the first run of dehydrated
stops at the terms of service of Pebble, and the case checks that
#auto_accept_terms registers the account and that the second run gets the
certificate. For eab.autossl.test, dehydrated uses the second Pebble: the
registration fails, dehydrated removes the account key it made, and the
case checks that #auto_cert_deploy accepts the terms only once and returns
after two runs of dehydrated. The case reads Redis with test/t/cases/_redis_client.rb, a
client of its own in CRuby, not with the mruby-redis of the nginx under
test.
Nginx::SSL::ACME::Client#register and #auto_cert_deploy are not tested.
They speak ACME v1 (new-reg, new-authz, new-cert), which Let’s Encrypt
turned off in 2021 and which Pebble does not implement, so no ACME server
that the tests could use answers them. The other methods of the Client are
tested without an ACME server.
Running the tests under AddressSanitizer and UndefinedBehaviorSanitizer
Section titled “Running the tests under AddressSanitizer and UndefinedBehaviorSanitizer”When test/ci/changes.sh gives build standard or full (a change
of the code or of the tests, as for the four standard cells), the author
runs the suite with -fsanitize=address,undefined before marking the
pull request ready; the sanitizer cell of the CI matrix builds with the
same flags and repeats it on the x86-64 runner (against OpenSSL 1.1.1w
and with the auto-ssl case, as valgrind in CI says
after its docker form). Run it in a checkout of its own (for example git archive HEAD | tar -x -C DIR): an existing mruby build is not rebuilt when only
the flags change. On Linux, with gcc or clang:
$ sudo sysctl -w vm.mmap_rnd_bits=28 # ASan and the kernel's high-entropy ASLR do not mix$ export NGINX_EXTRA_CC_OPT="-fsanitize=address,undefined -fno-omit-frame-pointer"$ export NGINX_EXTRA_LD_OPT="-fsanitize=address,undefined"$ export NGX_MRUBY_CFLAGS="-fsanitize=address,undefined -fno-omit-frame-pointer"$ export NGX_MRUBY_LDFLAGS="-fsanitize=address,undefined"$ export ASAN_OPTIONS="detect_leaks=1"$ export LSAN_OPTIONS="suppressions=$PWD/test/lsan.supp"$ export UBSAN_OPTIONS="print_stacktrace=1:suppressions=$PWD/test/ubsan.supp"$ NGINX_RUNNER=exec sh test.sh$ grep -HnE 'ERROR: (AddressSanitizer|LeakSanitizer)|runtime error:|SUMMARY: (AddressSanitizer|UndefinedBehaviorSanitizer)' $(find build/nginx/logs $(ls -d build/nginx/second_instance 2>/dev/null) -type f -name '*.log')On macOS, run it in a Linux container, from the top of that checkout.
The sysctl line is for the kernel that the container shares, which a
container cannot change. Read its value first:
$ docker run --rm ubuntu:22.04 cat /proc/sys/vm/mmap_rnd_bitsAt 28 or below, nothing is needed. Above 28, set it to 28 where that
kernel runs (the Linux VM of the container runtime), as the sanitizer
cell of CI does with sudo sysctl -w vm.mmap_rnd_bits=28 before it
builds: ASan’s shadow memory layout fails with a higher entropy.
$ docker run --rm --init -v "$PWD":/work -w /work ubuntu:22.04 sh -c 'apt-get -qq update && DEBIAN_FRONTEND=noninteractive apt-get -qq install -y build-essential rake ruby bison git gperf wget ca-certificates curl zlib1g-dev libpcre3-dev libssl-dev >/dev/null && export NGINX_EXTRA_CC_OPT="-fsanitize=address,undefined -fno-omit-frame-pointer" NGINX_EXTRA_LD_OPT="-fsanitize=address,undefined" NGX_MRUBY_CFLAGS="-fsanitize=address,undefined -fno-omit-frame-pointer" NGX_MRUBY_LDFLAGS="-fsanitize=address,undefined" ASAN_OPTIONS=detect_leaks=1 LSAN_OPTIONS=suppressions=$PWD/test/lsan.supp UBSAN_OPTIONS=print_stacktrace=1:suppressions=$PWD/test/ubsan.supp && NGINX_RUNNER=exec sh test.sh && ! grep -HnE "ERROR: (AddressSanitizer|LeakSanitizer)|runtime error:|SUMMARY: (AddressSanitizer|UndefinedBehaviorSanitizer)" $(find build/nginx/logs $(ls -d build/nginx/second_instance 2>/dev/null) -type f -name "*.log")'The command exits 0 when test.sh passed and the grep found no
report. The container uses gcc 11 of Ubuntu 22.04, the compiler of the
sanitizer cell.
The last line of the Linux commands reads the files that the “Check for sanitizer reports” step of the CI cell reads, with the same pattern; the step fails when a line matches. A report goes to the stderr of the process that wrote it, and that is one of these files:
| File | Whose stderr |
|---|---|
build/nginx/logs/error.log |
the test nginx, once it has opened its log (its leak report at exit included) |
build/nginx/logs/stderr.log |
the test client (test/t/ngx_mruby.rb and the case files; test.sh redirects it), and so the processes that the tests start without redirecting it: nginx -t, the nginx -s quit of the stream-only test, an nginx before it has opened its log |
the other build/nginx/logs/*.log |
the nginx processes that the tests start with a log of their own, after they have opened it (stream_only.error.log of the stream-only test, which runs nginx with daemon on) |
build/nginx/second_instance/<mode>/*.log |
the second nginx of test/t/cases/_second_instance.rb (error.log, stderr.log, stdout.log) |
build/nginx/second_instance/ is missing when test.sh stopped before the
case files; the step then reads the other files and says so. The step does
not read the *.log.bak files, in which test.sh keeps the
logs of an earlier run in the same directory. A report of the test nginx
from before it opened error.log goes to the terminal (in CI, the output of
the script step), as the next paragraph says.
NGINX_EXTRA_CC_OPT and NGINX_EXTRA_LD_OPT are appended to nginx’s
--with-cc-opt and --with-ld-opt; NGX_MRUBY_CFLAGS and NGX_MRUBY_LDFLAGS
reach the mruby build. Do not combine the sanitizers with valgrind. nginx
sends its stderr to error.log once the configuration is loaded, so reports
from code that runs while nginx parses the configuration (directive handlers,
script compilation, mruby_init_code) appear on the terminal instead; read
both. test/ubsan.supp lists the reports that come from nginx itself, each
with the reason. test/lsan.supp does the same for LeakSanitizer: nginx
never frees cycle->connections and the two event arrays it allocates in
ngx_event_process_init, and whether LeakSanitizer reports them at exit
depends on whether anything outside them still points at them, so without
the suppression a run can fail on nginx’s own allocations. With the gcc 11
runtime that CI uses, a suppressed block hides nothing else; the LLVM
runtime and gcc 12 and later also treat a suppressed block as a root, so
there a block reachable only from those arrays is not reported either.
Mock LLM upstream
Section titled “Mock LLM upstream”test/soak/mock_llm.rb (CRuby 3.0 or later) is an HTTP/1.1 server that
answers POST /v1/messages the way the Anthropic Messages API does. On
master, it is the upstream of the agent proxy scenarios of the
performance comparison (proxy_*, auth, route_json_*; see
“Performance comparison with callgrind”), which test/perf/perf.rb starts
as a process of its own on the port base + 2 (12372) for each scenario, with
the options of the scenario (mock in test/soak/scenarios.rb). On next,
the memory soak test and a functional case use it as well.
$ ruby test/soak/mock_llm.rb --port 12372 --events 50| Option | Default | Meaning |
|---|---|---|
--port PORT |
(required) | listen on 127.0.0.1:PORT; repeat it for more ports, which answer the same way |
--events N |
50 | content_block_delta events in a stream |
--delta-text TEXT |
hello |
the text of each content_block_delta |
--hold-after K |
off | write the first K events of a stream, then nothing more until the other side closes the connection |
--reset-after K |
off | write the first K events of a stream, then reset the connection (RST) |
--status CODE |
off | answer every request with this status and an Anthropic error body (overloaded_error for 529, api_error for 503), before any event |
--delay-ms MS |
0 | wait MS milliseconds between two events |
--one-write |
off | write the head and the whole stream with one write |
A request can override the options for itself with the headers
x-mock-events, x-mock-hold-after, x-mock-reset-after, x-mock-status
and x-mock-delay-ms, each a number of 0 or more. x-mock-hold-after,
x-mock-reset-after and x-mock-status also take off, which turns the
hold, the reset or the error status off; x-mock-events and
x-mock-delay-ms do not. Any other value gets a 400 answer
(invalid_request_error) that names the header. nginx forwards the headers
unless the configuration drops them. ruby test/soak/mock_llm_test.rb checks
these rules and the answers below against the mock in its own process (no
nginx, about a second); the perf job runs it before the comparison.
What it answers:
- A JSON body with
"stream": true: 200,text/event-stream, chunked. The events aremessage_start(withusage),content_block_start, Ncontent_block_deltaevents with the text,content_block_stop,message_delta(withusage) andmessage_stop: N + 5 events. Without--one-write, each event is one chunk written with one write; the response head goes with the first event and the last chunk withmessage_stop. - Any other JSON object: 200 with a message as JSON (
Content-Length). - A body that is not a JSON object: 400; another path: 404.
GET /_mock/stats: its counters as JSON (below).
The answer depends only on the request and the options: the model is the
request’s model, input_tokens is the request body’s size in bytes / 4
(rounded up), output_tokens is N (6 for a message), and there are no
sleeps but --delay-ms. perf.rb computes the expected response with the
same code (MockLLM.stream_body, MockLLM.message_body) and compares it
byte for byte. Responses also carry x-mock-port (the port that answered)
and, when the request had them, x-mock-authorization and
x-mock-x-api-key with the values that arrived, so that a test can see
which credential the proxy sent.
The mock never closes an idle keep-alive connection. It closes one only when
the request has Connection: close (the answer then has it too), when the
other side closes it, and for --hold-after and --reset-after. The
counters of GET /_mock/stats:
| Counter | Meaning |
|---|---|
connections_open |
connections open now, without the one asking |
connections |
connections accepted |
requests |
POST /v1/messages requests read |
completed |
answers written in full (error statuses included) |
held |
streams waiting in --hold-after now |
held_closed |
streams in --hold-after that the other side closed |
reset |
connections reset by --reset-after |
write_failed |
answers cut short because the other side had closed the connection |
Release builds
Section titled “Release builds”sh test/build_release.sh SOURCE_DIR BUILD_DIR builds nginx with ngx_mruby
as a static module with release-like options: nginx and ngx_mruby with
-O2 -g, mruby with the flags of its gcc toolchain (-O3 -g), without
MRB_GC_STRESS (which test.sh adds) and without --with-debug. The script
is the one of next, so its header comment also names the soak build of
next (test/soak/run.sh, -DNGX_MRUBY_DEBUG_STATS) and
NGX_MRUBY_AUTO_SSL, which on next adds the auto-ssl mrbgem; on master,
build_config.rb always builds that gem, and the variable changes only the
build stamp. The performance comparison builds with it (test/perf/run.sh,
test/perf/compare.sh), in build_perf/head/ and build_perf/base/, with
no option added: the defines of the soak build of next
(-DNGX_MRUBY_DEBUG_STATS, -DMRB_USE_MALLOC_TRIM) would add code to the
process under measurement.
SOURCE_DIR is the checkout to build, and its own build files are used
(build.sh, configure, build_config.rb, …), so the base of a pull
request is built the way it builds itself. The script copies the sources to
BUILD_DIR/tree, runs build.sh there and installs nginx to
BUILD_DIR/nginx, so the build/ directory, mruby/build and the generated
Makefile, config and mrbgems_config of test.sh are not touched. The
build is always a static module built from its own nginx source with the
system OpenSSL, so the script ignores BUILD_DYNAMIC_MODULE, NGINX_SRC_ENV
and OPENSSL_SRC_VERSION. The nginx source is downloaded to
BUILD_DIR/tree/build/; when nginx.org is unreachable, put it there as
described in AGENTS.md.
The third-party gems are built at the commits of a gem lock: the one that
RELEASE_GEM_LOCK names when it is set and not empty, else the
build_config.rb.lock in SOURCE_DIR. master commits no lock (its
.gitignore lists the file), so a fresh checkout of master has none and
gets the head of each gem’s branch on its first build; a checkout in which
sh test.sh has run has the lock that rake wrote there, with the commits of
that build. rake writes the commits it built to
BUILD_DIR/tree/build_config.rb.lock.
Later runs update the copy and rebuild only what changed:
- A file is written to the copy only when its content differs, and it gets
the time of the copy, not that of
SOURCE_DIR. A checkout made withgit archivehas the time of the commit, which can be older than the objects of the last build, and make would not rebuild it. - Files that no longer exist in
SOURCE_DIRare removed from the copy. - make does not track the headers in
src/anddependence/, so when one of them changed, the.cfiles there are touched. - Files are overwritten in place, not removed and created again. With the
sources on a case-insensitive file system shared with a container (Docker
Desktop on macOS), rake loads mruby’s
Rakefileasrakefile, and after the file had been removed and created again, a second run in the same container failed withLoadError: cannot load such file -- .../rakefile.
make and rake do not notice every change, though: the objects of a removed
source or a dropped gem stay in libmruby.a, and nginx’s configure does not
run again. The script therefore records these inputs in
BUILD_DIR/build_stamp: the mruby tree (its git tree id when SOURCE_DIR is
the top of a git work tree, else a checksum of its files without
mruby/build and mruby/bin, the output of the test.sh build), the file
names under mrbgems/, checksums of build_config.rb, config.in,
configure, Makefile.in, build.sh and nginx_version,
NGX_MRUBY_CFLAGS (passed to mruby, as with test.sh),
NGX_MRUBY_AUTO_SSL (see above), the nginx configure options and the gem lock of the build. When the stamp
differs from that of the last build, mruby and nginx are built from scratch,
with new clones of the gems; the downloaded nginx source is kept. The new
clones matter when only the lock changed: rake checks an existing clone out
at the commit of the lock without fetching, and an older clone may not have
that commit. After a change that the stamp does not cover, for example an
uncommitted change under mruby/ (the git tree id is that of the commit),
reset the build by hand with rm -rf build_perf.
Performance comparison with callgrind
Section titled “Performance comparison with callgrind”A change to the request path can make every request do more work without
failing any test. The performance comparison counts the instructions that
nginx executes per request (callgrind’s Ir) in the base and the head of a
pull request, both release builds, and reports the change per scenario.
Instruction counts do not depend on the speed or the load of the machine, so
the comparison works on shared CI runners, whose throughput varies by more
than 30% (section 3.7 of docs/proposals/v3-plan.md on next). They do not
show cache misses, branch mispredictions or the time spent in the kernel:
they measure how much work the user-space code of nginx, ngx_mruby, mruby
and the C library does for a request.
The comparison is the one of next. master has its files (test/perf/,
test/build_release.sh, and in test/soak/ the configurations, scenarios,
HTTP client, handlers and mock LLM that it reads), but not the memory soak
test of next (test/soak/soak.rb, test/soak/run.sh) and not
Nginx::Debug, the Ruby class of the soak build that the soak reads: the
copies of test/soak/nginx.conf and test/soak/nginx.agent.conf on
master have no /debug/stats location, which only the soak requests. The
comparison reads none of the soak’s counters, so it measures the same on
both branches.
What it measures
Section titled “What it measures”The scenarios are the locations of test/soak/nginx.conf, each with a
keep-alive client:
| Scenario | Handler |
|---|---|
hello |
Nginx.rputs |
headers |
reads 3 request headers, sets 2 response headers |
var |
reads 2 variables with Nginx::Var, sets 1 |
filter |
mruby_output_body_filter_code rewrites a return 200 body |
sleep |
Nginx::Async.sleep 1, then the body |
sub_request |
Nginx::Async::HTTP.sub_request to a location that proxies to the static location of a second server (port base + 1) |
file |
mruby_content_handler with a file and cache |
and the agent proxy scenarios of “Agent proxy scenarios in the comparison”
below. (test/soak/scenarios.rb also lists the scenarios that only the soak
of next runs: disconnect and agent_*, which are not keep-alive and
are not measured.) For each scenario, and for the base and the head in
turn, test/perf/perf.rb:
- starts the mock LLM when the scenario has one, and nginx with
test/soak/nginx.conf(or the scenario’s template), changed tomaster_process off, undervalgrind --tool=callgrind --instr-atstart=no, so that the startup runs without instrumentation; - checks one response and sends
PERF_WARMUPrequests; - waits until
stub_statusshows no open connection of the warmup, and switches the instrumentation on withcallgrind_control --instr=on; - sends
PERF_Nrequests over keep-alive and checks every response. nginx closes a connection after 1000 requests (keepalive_requests), and the client then opens the next one; - writes the profile with
callgrind_control --dump, and stops nginx.
The profile of step 5 holds the cost of the requests of step 4: not the
startup, the configuration, the warmup or the exit. Ir per request is
its total Ir divided by PERF_N. The alternative, the difference of two runs
with N and 2N requests divided by N, needs three times as many requests and
adds the variation of two runs; the window of one run has nothing but the
requests in it.
callgrind_control reaches the process through valgrind’s gdbserver (vgdb),
which does not serve a process that valgrind forked, such as an nginx
worker. With master_process off, the one nginx process runs the event loop
and the handlers as a worker does. When vgdb may not use ptrace to interrupt
a process that waits in epoll_wait, a command runs the next time the
process runs code, so perf.rb sends a request to /status (stub_status)
every 0.2 seconds until the command has run. Parts of these requests (two
per window in the runs so far) fall into the window.
Ir per request without GC leaves out the cost of mruby’s garbage
collector: the inclusive cost of the calls into mrb_incremental_gc and
mrb_full_gc made from outside the collector (from mrb_obj_alloc, the
malloc wrappers, GC.start). It is the inclusive cost that
callgrind_annotate --inclusive=yes prints for mrb_incremental_gc and
mrb_full_gc without a recursion suffix ('2), without counting twice a
full GC that mrb_incremental_gc runs; perf.rb reads it from the call
records of the profile (see the comment of parse_profile). The GC runs in
bursts: mrb_obj_alloc calls the collector when the number of live objects
passes a threshold, and one cycle (root scan and marking, then sweeping) is a
few calls. A change that allocates a few objects more or less per request
moves the cycles against the window, and the total moves by a share of one
cycle. The number without GC does not have that step, so the thresholds
apply to it; the total is reported as well. The GC also changes the cost of
the rest (after a sweep has freed many blocks, the next malloc calls take
longer), so the window has to span many GC cycles, which is why PERF_N is
20000 (the runs that showed it are in “Performance comparison with
callgrind” of docs/test/README.md on next).
Agent proxy scenarios in the comparison
Section titled “Agent proxy scenarios in the comparison”These scenarios measure what an agent proxy costs per request: nginx in
front of the mock LLM (see “Mock LLM upstream”), with
test/soak/nginx.agent.conf. That configuration has no mruby filter
directive: one body filter directive anywhere in an http {} block
installs ngx_mruby’s body filter for every response of that block (as the
filter scenario does in test/soak/nginx.conf), and each chunked
response, such as a stream, would then pass it and log a notice. Its
error_log is at warn. The proxy settings are those of an agent proxy:
proxy_buffering off, proxy_buffer_size 4k, proxy_http_version 1.1,
the client’s Authorization replaced by the credential that a handler
picked, and an upstream block with keepalive 32,
keepalive_requests 1000000 and keepalive_timeout 1h. The handlers are in
test/soak/handlers/:
agent_init.rb(mruby_init_worker): the client key, credential and model tables;agent_auth.rb(server rewrite handler of the serverauth.agent.test): looks up the client key fromAuthorizationorx-api-key(two Hash lookups) and sets three variables;agent_route.rb(access handler of/v1/route, withmruby_enable_read_request_body on):JSON.parse(r.body)["model"], one Hash lookup, and$agent_backendforproxy_pass http://$agent_backend.
The request bodies are Messages requests of exactly 2 KB or 64 KB
(MockLLM.request_body). The scenarios without Ruby are the baselines of
the scenarios of the same shape, so that the difference between the two is
what the Ruby adds. The last row of the table is the seven scenarios of
test/soak/nginx.conf, for their request function.
| Scenario | Request and answer | Ruby per request | Request function (calls per request) |
|---|---|---|---|
proxy_plain_2k, proxy_plain_64k |
POST of a Messages body of exactly 2 KB or 64 KB to /v1/plain (proxy_pass to the mock); a JSON message |
none | ngx_http_log_request (1) |
proxy_stream_plain_50, proxy_stream_plain_1000 |
the 2 KB body with "stream": true to /v1/plain; a stream of 50 or 1000 deltas (55 or 1005 events) |
none | ngx_http_log_request (1) |
auth |
proxy_plain_2k to the server auth.agent.test, whose server rewrite handler agent_auth.rb looks up the client key and sets the upstream credential |
1 handler | ngx_http_log_request (1) |
route_json_2k, route_json_64k |
proxy_plain_* to /v1/route, whose access handler agent_route.rb parses the body with JSON.parse and sets the upstream block for proxy_pass http://$agent_backend |
1 handler | ngx_http_log_request (1) |
ruby_call_1, ruby_call_10 |
GET of a location with 1 or 10 mruby_set_code variables and return 200; no upstream |
1 or 10 calls | ngx_http_mruby_set_inline_handler (1 or 10) |
hello, headers, var, filter, sleep, sub_request, file |
GET of the locations of test/soak/nginx.conf; no mock |
1 handler or filter | ngx_mrb_start_fiber (1) |
The seven scenarios of test/soak/nginx.conf count ngx_mrb_start_fiber:
ngx_mrb_run, which runs the Ruby code of a handler or filter, starts one
fiber for each run, and nothing else calls ngx_mrb_start_fiber. They do
not count ngx_mrb_run, because callgrind does not record its calls the
same way in every build: in an aarch64 build of next (2026-10-04), it
recorded a second call of ngx_mrb_run per request in six of the seven,
and one call of ngx_mrb_start_fiber per request in all seven, as in the
x86_64 builds of the CI runner.
The scenarios with an upstream count ngx_http_log_request, which nginx
calls once for each request it ends, with or without Ruby (for a
subrequest only with log_subrequest on): the scenarios without Ruby have
no call of ngx_mrb_start_fiber to count, and all the scenarios with an
upstream count the same function. The /status requests that wake nginx for
callgrind_control also end in the window when they come after the
instrumentation was switched on, so the window may hold up to that many
calls more (2 per window in the runs so far). mruby_set_code reaches
ngx_mrb_run through a tail call of its handler, which callgrind does not
always record as a call of ngx_mrb_run, so the ruby_call_* scenarios
count the handler, which nginx calls through a pointer.
Two counts per request are printed after the table and kept in
report.json, but not checked: ruby_calls, the calls of
ngx_mrb_start_fiber, and non_buffered_calls, the calls of
ngx_http_upstream_process_non_buffered_request, which relays an
unbuffered response. nginx calls it when the upstream connection becomes
readable and when the client connection becomes writable. One call reads
from the upstream until recv() would block, at most proxy_buffer_size
(4k) at a time, and writes what it read to the client, so how many calls a
response takes depends on how much of it had arrived at each call. It also
differs between builds and machines (2.00 per request for proxy_plain_*,
auth and route_json_* in an aarch64 container, 1.00 on the x86_64 CI
runner, in the runs of next).
Each call costs Ir: about 1000 per call in the runs of next
(NON_BUFFERED_CALL_IR in perf.rb). With two builds, the line of a
scenario whose calls differ gives the Ir that the difference alone makes, as
a share of the base without GC, and from half of PERF_WARN_PERCENT on it
says that the change of the scenario may come from when the response
arrived. For proxy_stream_plain_50, about 2 more calls per request reach
WARN and about 3 reach FAIL.
proxy_stream_plain_1000 and route_json_64k take more than a minute per
build each, so they are not in DEFAULT_SCENARIOS; name them in
PERF_SCENARIOS.
Thresholds and calibration
Section titled “Thresholds and calibration”A scenario is WARN when its Ir per request without GC is 3% or more above
the base (PERF_WARN_PERCENT), and FAIL from 5% (PERF_FAIL_PERCENT).
The exit status is 1 when a scenario is FAIL, when a measurement fails
(ERROR: an unexpected response, a line at the error level or above in
error.log, a timeout, a missing dump, or a window that does not hold the
requests, see “In CI”), or when no scenario was compared. The one exception
is a base that answers with an unexpected response while the head passes,
which is what a scenario that needs a feature of the head gets: it is shown
as n/a and does not fail the run. That holds only when nothing else went
wrong in the base measurement. Its error.log may have the lines that
ngx_mruby logs when the Ruby code of a request raises (mrb_run failed: return 500 HTTP status code to client: error: ...), which is what a base
that lacks a Ruby method or class of the head logs (NoMethodError,
NameError). Any other line at the error level or above makes the
scenario ERROR, for example the connect() or recv() error that nginx
logs when the mock LLM failed and nginx answered 502. So do a problem of
the mock, and more than five lines at those levels in one log: the report
shows five lines per log and does not read the rest. A directive that only
the head has stops the base at startup, which is ERROR as well.
The thresholds were calibrated on next in October 2026, with null changes
(base and head with the same build inputs) and with a planted cost (an
empty loop of 100 iterations in ngx_mrb_run, which the report showed as
WARN and FAIL rows); the runs and their numbers are in “Thresholds and
calibration” under “Performance comparison with callgrind” in
docs/test/README.md on next. In those null changes, the scenarios of
test/soak/nginx.conf moved by at most about 0.08% between base and head
(sub_request, which proxies to a second server, the most), and the
scenarios with an upstream by at most 0.39% on the CI runner, more than 7
times below the 3% of WARN.
On master, two null changes on 2026-10-19 (the base master, the head
the pull request that added the comparison to master, which changed no
build input) measured:
| Scenario | aarch64: base Ir/req (w/o GC) | aarch64 container, compare.sh |
x86_64, CI run 39240312315 |
|---|---|---|---|
hello |
11853 (11173) | -0.003% | +0.000% |
headers |
36364 (34388) | +0.000% | -0.001% |
var |
21342 (19973) | +0.000% | +0.002% |
filter |
14203 (13364) | +0.002% | +0.000% |
sleep |
15661 (14888) | +0.004% | +0.000% |
sub_request |
48731 (47568) | +0.002% | +0.002% |
file |
14069 (13231) | +0.000% | +0.000% |
proxy_plain_2k |
22102 (22102) | -0.004% | +0.001% |
proxy_plain_64k |
22426 (22426) | -0.001% | -0.229% |
proxy_stream_plain_50 |
53135 (53135) | -0.018% | -0.119% |
auth |
48206 (46038) | +0.003% | -0.001% |
route_json_2k |
109332 (106191) | -0.005% | +0.000% |
ruby_call_1 |
11360 (10677) | +0.009% | +0.000% |
ruby_call_10 |
41988 (35598) | +0.001% | +0.000% |
The last two columns are the change of head against base without GC. The
first run was compare.sh with nginx 1.31.6 on Ubuntu 22.04 (gcc 11.4,
valgrind 3.18.1) in a container on aarch64, PERF_N=20000, base master at
a6c32a4; the .text sections of the two nginx binaries were byte for byte
the same. The second is the perf job of that pull request on the CI runner
(ubuntu-22.04, x86_64), base master at a500846. Both builds of each run
had the same 21 third-party gems at the same commits. The scenarios of
test/soak/nginx.conf and those without an upstream moved by at most
0.009%. The scenarios with an upstream moved the most: on the CI runner
proxy_plain_64k by -0.229% (22777 and 22725 Ir per request, its
non_buffered_calls 1.00 in both builds) and proxy_stream_plain_50 by
-0.119% (its non_buffered_calls 4.62 and 4.54, which the report estimated
at -0.15%). The largest, 0.229%, is about 13 times below the 3% of WARN.
In the aarch64 container, the measurement of the default scenarios took
about 4 minutes per build, sleep 45 seconds of it; the perf job took
13 minutes, both builds included.
On master the absolute numbers of a scenario can differ between runs on
different days: master commits no gem lock, so each run builds the gems
at the heads of their branches at that time (both builds of one run at the
same commits). The change between base and head of one run is what the
thresholds compare.
Running it
Section titled “Running it”When the pull request changes an input of the measured binary or the
measurement (perf true), its author runs the comparison with the base
before marking it ready and quotes the report (“Design principles” in
AGENTS.md), and runs ruby test/soak/mock_llm_test.rb,
which the perf job runs before its comparison (Mock LLM
upstream); the perf job (In CI) then
runs both again on the runner.
The driver needs CRuby 3.0 or later and valgrind with callgrind_control
and its helper vgdb, which work on Linux; perf.rb stops at once
without them. On macOS, run it in a Linux container. To compare this
checkout with origin/master in a container, in one command:
$ rm -rf build_perf/base-src && git archive --prefix=build_perf/base-src/ origin/master | tar -x && docker run --rm -v "$PWD":/work -w /work ubuntu:22.04 sh -c 'apt-get -qq update && DEBIAN_FRONTEND=noninteractive apt-get -qq install -y build-essential rake ruby bison git gperf wget ca-certificates zlib1g-dev libpcre3-dev libssl-dev valgrind >/dev/null && sh test/perf/compare.sh build_perf/base-src'On Linux:
$ sh test/perf/compare.sh build_perf/base-src # build both, then compare$ ONLY_RUN=1 sh test/perf/compare.sh build_perf/base-src$ ONLY_RUN=1 PERF_SCENARIOS=hello,var sh test/perf/compare.sh build_perf/base-src$ sh test/perf/run.sh # measure this checkout onlycompare.sh BASE_DIR [HEAD_DIR] builds BASE_DIR into build_perf/base and
HEAD_DIR (default: this checkout) into build_perf/head. Both builds take
the third-party gems at the same commits: a checkout without a gem lock
takes the lock of the other side (the base the head’s, the head the one
that the base build wrote). On master, which commits no lock, the base
(made with git archive) has none, so it takes the lock that sh test.sh
left in this checkout, and when this checkout has none either (a fresh
clone, as in CI), the base is built at the head of each gem’s branch and
the head takes the lock that the base build wrote. The report says how many
gems are at different commits and lists them. The measurement (test/perf/,
the scenarios in test/soak/) comes from this checkout for both builds, so
a base from before the comparison was added (a tag, for example) is
measured as well. The first run builds two trees (a few minutes each); the
measurement of the default scenarios takes about 4 minutes per build. The
report goes to build_perf/report.txt and build_perf/report.json.
test.sh kills every nginx on the machine, so do not run it at the same
time.
| Variable | Default | Meaning |
|---|---|---|
ONLY_RUN |
unset | set to skip the builds |
NUM_THREADS_ENV |
half of the CPUs | build parallelism (passed to build.sh) |
PERF_SCENARIOS |
DEFAULT_SCENARIOS of test/perf/perf.rb |
comma-separated scenario names |
PERF_N |
20000 | requests in the measured window |
PERF_WARMUP |
2000 | requests before the window |
PERF_WARN_PERCENT |
3 | WARN from this change of Ir per request without GC |
PERF_FAIL_PERCENT |
5 | FAIL from this change |
PERF_PORT_BASE |
12370 | port of the scenarios; the backend of sub_request uses the next one, the mock LLM the one after |
PERF_REPORT_DIR |
build_perf |
where report.txt and report.json go |
PERF_REQUEST_FUNCTIONS |
ngx_mrb_start_fiber |
names of the function counted once per request in the seven scenarios of test/soak/nginx.conf (see “In CI”) |
The perf job repeats on the runner the comparison that the author ran
locally before marking the pull request ready (Running
it), so that the result does not rest on one machine
alone. The perf job of .github/workflows/test.yml starts on ready pull
requests whose run builds (build is standard or full in Which jobs a run
starts), and compares when test/ci/changes.sh
gives perf true, that is when the pull request changes an input of the
measured binary or the measurement, or when it has the label perf (read
when the job runs, so adding the label and re-running the job is enough). A
pull request that changes only test.yml among these files compares only
with the label, and a pull request whose files are all outside the code and
the tests skips the job, with the label too. The inputs of the binary are what
test/build_release.sh copies: src/, mrbgems/, mruby/, dependence/
(ngx_Makefile.in adds to nginx), and the top-level build
files build_config.rb, configure, config.in, Makefile.in,
build.sh and nginx_version; master commits no gem lock, so on
master the lock is not an input (on next, build_config.rb.lock is).
The measurement is test/perf/, test/build_release.sh and the scenario
files of test/soak/ (nginx.conf, nginx.agent.conf, scenarios.rb,
http_client.rb, mock_llm.rb, handlers/), as test/ci/changes.sh lists
them on both branches. The job first runs ruby test/soak/mock_llm_test.rb,
the check of the mock LLM.
A change of nginx_version builds the head with another nginx, so its
WARN or FAIL measures nginx’s own cost as well as ngx_mruby’s; read the
functions of the profiles (below) before taking it as a regression of
ngx_mruby.
The checkout of a pull request is the merge commit (refs/pull/N/merge);
the job compares its first parent, the base branch, with the merge commit,
so the difference is the change of the pull request as it would be merged.
The table is in the job summary, with the reasons of the failed
measurements below it; WARN, FAIL, ERROR and n/a rows and a run
that compared no scenario are annotations; the job fails on FAIL and
ERROR. A measurement fails (ERROR) when nginx does not start or answer,
callgrind_control fails or times out, the dump is missing, error.log has
a line at the error level or above, or the window does not hold the
requests: 0 Ir, or not the expected calls of the scenario’s request
function: exactly one call of ngx_mrb_start_fiber per request in the
seven scenarios of test/soak/nginx.conf, and in the agent proxy scenarios
the functions and counts of the table in “Agent proxy scenarios in the
comparison” (callgrind_control prints “OK.” even when vgdb did not reach
the process, so an empty window is caught here). Only a base that answers a
scenario with an unexpected response and has no other problem than
ngx_mruby’s mrb_run failed lines (a scenario that needs a feature of the
head; see “Thresholds and calibration” above) gives n/a, and a run in
which no scenario was compared fails.
The calls of the request function are counted by name. A copy that GCC
makes of a function at -O2 (ngx_mrb_start_fiber.part.0, .isra.0,
.constprop.0, .cold) counts as the function, and a call from one copy to
another counts once, so a change that makes GCC split or clone it does not
fail the window. A pull request that renames ngx_mrb_start_fiber, or
moves the start of the fiber of a Ruby run to another function, has to add
the new name to REQUEST_FUNCTIONS in test/perf/perf.rb and keep the old
one: the perf.rb of the head measures the base too, and a call between
two listed names counts once. GCC cannot inline ngx_mrb_start_fiber now,
because it is defined in ngx_http_mruby_async.c and called from
ngx_http_mruby_module.c. A change that lets the compiler inline it (for
example making it static in the file of its single caller, or a build
with LTO) would leave no call of it in the profile, and every window of the
seven scenarios would hold 0 calls, so such a change has to list a function
that keeps a real call. PERF_REQUEST_FUNCTIONS (comma-separated) overrides
the list for a local run, REQUEST_FUNCTION_CALLS sets another number of
calls per request for a scenario, and SCENARIO_REQUEST_FUNCTIONS another
function. ruby test/perf/perf.rb --self-test checks the name rules and the
reading of a profile on a made-up profile, without a build or valgrind;
compare.sh and run.sh run it first.
ci-ok does not need the job yet, so a FAIL does not fail ci-ok.
“Design principles” in AGENTS.md still holds the merge of a pull request
whose latest perf run reports FAIL or ERROR (unless the owner
accepted the FAIL on the pull request), and a WARN until the owner
accepts it. The plan (Pillar E of docs/proposals/v3-plan.md on next) is
to add the job to the needs of ci-ok once it has run on pull requests
that change src/ or mrbgems/ without false alarms.
The artifact perf-callgrind has the report, and for base and head the
profiles of the windows (callgrind/callgrind.out.<scenario>), the output
of callgrind_control and valgrind, and the nginx logs. To see where a
change comes from, print the functions of the same scenario in both builds
and compare them:
$ callgrind_annotate --inclusive=yes build_perf/head/callgrind/callgrind.out.var | head -40$ callgrind_annotate build_perf/base/callgrind/callgrind.out.var > base.txt$ callgrind_annotate build_perf/head/callgrind/callgrind.out.var > head.txt$ diff base.txt head.txtThe counts are for the whole window; divide by PERF_N for one request.
KCachegrind and QCachegrind open the profiles as well.