Benchmark Setup
This section covers installing comparison tools, setting up the Juliet test suite, and configuring real-world codebases for benchmarking.
Benchmark Host Layout
Every machine that runs benchmarks needs one directory –
$SQC_BENCH_ROOT (default ~/toolchain) – holding the Juliet suite and every real-world codebase
checkout. bench/config.py and bench/realworld_runner.py both
read this from the SQC_BENCH_ROOT environment variable, or from a
.env file at the repo root (copy .env.example – it’s gitignored, so
each machine sets its own).
To provision a fresh node, run the Ansible playbook (clones and pins every real-world codebases; does not fetch Juliet – see below):
ansible-playbook playbooks/setup-benchmark-repos.yml -i "localhost," -c local
# Or a different root:
ansible-playbook playbooks/setup-benchmark-repos.yml -i "localhost," -c local \
-e bench_root=/data/toolchain
Keep bench_root and SQC_BENCH_ROOT/.env pointed at the same
directory – the playbook provisions the checkouts the code actually reads.
A mismatch is easy to miss: if SQC_BENCH_ROOT is unset and the corpora
do not live in ~/toolchain, BENCH_ROOT silently resolves to a
directory that does not exist. python -m bench corpus-check reports the
resolved root and says so outright.
A second, separate root names where the source clones are:
$AURORA_LINT_SRC_ROOT is the parent directory of this checkout
($AURORA_LINT_SRC_ROOT/aurora-lint), and, on the maintainer’s machine,
of the sibling benchmarking_db and sqc_paper clones too. It defaults to the
checkout’s own parent, so nothing here requires it; the copy-pasteable
commands below use it so they hold on any machine regardless of where the
clone lives (~/data is the conventional value, and the .env.example
default):
export AURORA_LINT_SRC_ROOT=~/data # parent of your aurora-lint clone
Installing Comparison Tools
cppcheck
sudo apt update
sudo apt install -y cppcheck
# Verify
cppcheck --version
# Expected: Cppcheck 2.x.x
For the latest release, build from source:
sudo apt install -y cmake libpcre3-dev
git clone https://github.com/danmar/cppcheck.git
cd cppcheck
cmake -DCMAKE_BUILD_TYPE=Release -DUSE_MATCHCOMPILER=ON .
make -j$(nproc)
sudo make install
clang-tidy
Pin LLVM 21 — do not take the distro default. Ubuntu 24.04 ships
clang-tidy 18.1.3 and tops out at 20, while the published comparison in
docs/tool-comparison.rst is LLVM 21.1. Taking the distro package makes a
freshly-provisioned node measure an older competitor than the table it will
be compared against, and clang-tidy is the tool beating aurora-lint on the
Juliet overlap in that comparison — so that regression would flatter aurora-lint by
understating a rival. playbooks/install-static-analyzers.yml does this
automatically; by hand:
sudo install -d -m 0755 /etc/apt/keyrings
wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key \
| sudo tee /etc/apt/keyrings/apt.llvm.org.asc > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/apt.llvm.org.asc] \
http://apt.llvm.org/noble/ llvm-toolchain-noble-21 main" \
| sudo tee /etc/apt/sources.list.d/llvm-21.list
sudo apt update && sudo apt install -y clang-tidy-21
sudo update-alternatives --install /usr/bin/clang-tidy clang-tidy \
/usr/bin/clang-tidy-21 100
# Verify
clang-tidy --version
# Expected: Ubuntu LLVM version 21.1.x
The distro package is fine for anything that is not a published comparison:
sudo apt install -y clang clang-tidy
bear (for compile_commands.json)
Required for clang-tidy on projects with build systems:
sudo apt install -y bear
Facebook Infer
Infer v1.2.0 is installed from prebuilt Linux binaries. The playbook at
playbooks/install-static-analyzers.yml automates this:
ansible-playbook playbooks/install-static-analyzers.yml \
-i "localhost," -c local --ask-become-pass
Manual install:
# libtinfo5 required (Infer links against libtinfo.so.5)
sudo apt install -y libtinfo5 || \
sudo ln -s /usr/lib/x86_64-linux-gnu/libtinfo.so.6 \
/usr/lib/x86_64-linux-gnu/libtinfo.so.5
VERSION=1.2.0
curl -sSL "https://github.com/facebook/infer/releases/download/v$VERSION/infer-linux-x86_64-v$VERSION.tar.xz" \
| sudo tar -C /opt -xJ
sudo ln -s "/opt/infer-linux-x86_64-v$VERSION/bin/infer" /usr/local/bin/infer
# Verify
infer --version
# Expected: Infer version v1.2.0
Frama-C
Frama-C is installed via opam (the Debian 12 apt package is version 25/Manganese, too old for benchmarking). The playbook handles this automatically.
Manual install:
sudo apt install -y opam gcc g++ make m4 pkg-config autoconf \
libgmp-dev zlib1g-dev libgtk-3-dev libgtksourceview-3.0-dev graphviz
opam init --bare --no-setup --disable-sandboxing
opam switch create default 4.14.2
eval $(opam env)
opam install -y frama-c
# Verify (requires eval $(opam env) in each shell session)
frama-c -version
# Expected: 33.0 (Arsenic) as of 2026-09-04; 32.0 (Germanium) is what the
# competitor Juliet runs in data/competitor_results/ were taken with
Note
Add eval $(opam env) to your shell profile (~/.bashrc) to make
frama-c available without manual activation. The benchmark runner
does not need it: bench/realworld_runner.py wraps every Frama-C
invocation in eval $(opam env) itself, because it is not a login
shell.
Warning
Frama-C renamed the compile-database option between the two versions
above — -json-compilation-database through 32.0, -compilation-db
from 33.0. The runner probes frama-c -kernel-h and picks whichever
the installed binary accepts, so both work; a hand-written command line
copied from an older doc will abort with “option is unknown”.
Build-based tools need a compile database
Infer and Frama-C cannot be pointed at a source tree the way aurora-lint and cppcheck
can — both need a real preprocess. On the real-world suite they are driven
from each checkout’s compile_commands.json, so
playbooks/setup-compile-commands.yml must have run for a codebase before
either tool can scan it. python -m bench realworld-run --tool infer fails
fast with the exact command to fix it if the database is missing.
Expect a partial result from both, and check the coverage block each run
writes:
Infer captures whatever preprocesses.
setup-compile-commands.ymlrestores each checkout to pristine after building it, which deletes the build’s generated headers; the playbook now stashes the ones the compile database still references (pure-ftpd’sconfig.h, sqlite’ssqlite_cfg.h, valkey’sjemalloc/jemalloc.h, libcrc’stab/gentab*.inc) under<bench_root>/.cc_build/generated/and points the database there, so every in-scope translation unit preprocesses. Acapture_failureslist that is not empty means a database written before that existed — re-run the playbook, which detects and rebuilds it.Frama-C is partial by construction, because EVA analyzes one entry point at a time and a real codebase has no single one. Read
docs/design/framac-realworld.mdbefore quoting any Frama-C real-world number: a finding count from it is a floor, and recall is not expressible from it at all.
Reference Shadow Set (optional)
The reference shadow set is a list of pinned C codebases, disjoint from the
real-world benchmarks, that is never labeled or tuned against. Its only use
is python -m bench reference-ab: what a change moves on code nobody
tuned against (docs/design/reference-corpus-ab.md). Nothing else needs
it, so it is a one-time setup for anyone who wants to run that A/B:
ansible-playbook playbooks/setup-shadow-corpus.yml -i "localhost," -c local
ansible-playbook playbooks/setup-shadow-corpus.yml -i "localhost," -c local \
-e shadow_tier=all # every tier, the large one included
It downloads shallow checkouts, each at its pin in
data/reference_corpus.json, under $SQC_BENCH_ROOT/shadow_corpus/.
The tiers are cumulative: quick, standard (the default, about 1 GB),
and large or all (about 5 GB). The playbook checks first that about
twice the tier’s size is free, unless every checkout is already at its pin.
The checkouts stay where they are: reference-ab re-checks and cleans
each one at its pin before scanning it, so nothing needs fetching again
until the pins change. python -m bench reference-fetch --tier T does the
same fetch without Ansible.
The large tier’s scans are slow and memory-heavy. Expect hours for an A/B
over it, and a peak near 12 GB resident at 8 threads; on a machine with
under 16 GiB of RAM, reference-ab halves its default --jobs for it.
Juliet Test Suite Setup
The setup playbook does not fetch Juliet; it only creates the parent
directory and reports whether the suite is already in place. The NIST
Juliet Test Suite v1.3
page links a single zip that downloads directly, with no click-through.
NIST does not promise that URL stays put, so if the curl below fails,
start from that page. The archive extracts to C/, which is renamed to
the directory name bench/config.py expects:
sudo apt install -y curl unzip
mkdir -p $SQC_BENCH_ROOT/benchmarks # default: ~/toolchain/benchmarks
cd $SQC_BENCH_ROOT/benchmarks
curl -LO https://samate.nist.gov/SARD/downloads/test-suites/2017-10-01-juliet-test-suite-for-c-cplusplus-v1-3.zip
unzip -q 2017-10-01-juliet-test-suite-for-c-cplusplus-v1-3.zip
mv C juliet-test-suite-c
# Expected structure:
# juliet-test-suite-c/testcases/CWE*/ (118 CWE directories)
# juliet-test-suite-c/testcasesupport/ (shared helper functions)
The per-CWE manifests that fast mode runs (rules_templates/cwe/*.toml)
are generated by scripts/generate_rule_cwe_map.py and are not
committed, so that directory is nearly empty in a fresh clone.
python -m bench juliet runs the script for you before every run, which
also refreshes data/rule_cwe_map.json.
The Benchmark Environment
A benchmark run, real-world or Juliet, is pinned by five things: the corpus
commit, the aurora-lint commit, the oracle commit, the settings hash and the
environment it ran in (docs/adr/0018). The environment is a container
image, container/benchmark.Dockerfile, holding a Debian 12 (bookworm)
snapshot, every benchmark’s dependency set (the system headers it is scanned
against, each in its own tree), the corpus build tools and the comparison
tools at exact versions. The corpus checkouts and this repository are
mounted into it at run time, and aurora-lint is built inside it.
Build the image with podman (rootless is fine). The build is amd64 on every host, an arm64 Mac included:
podman build --platform linux/amd64 --target tools \
-f container/benchmark.Dockerfile -t aurora-bench-tools:dev .
podman build --platform linux/amd64 \
-f container/benchmark.Dockerfile -t aurora-bench:shared .
That image leaves out the Win32 corpus’s set, the Windows SDK and MSVC CRT headers, which are Microsoft’s: building the layer that adds them accepts Microsoft’s licence terms for your machine, and the result must never be copied to another one:
podman build --platform linux/amd64 -f container/licence.Dockerfile \
--build-arg BENCH_IMAGE=aurora-bench:shared \
--build-arg ACCEPT_MICROSOFT_LICENSE=true -t aurora-bench:dev .
The image records its environment manifest in
/etc/aurora-bench/environment.json; the SHA-256 of that manifest is the
pin. data/benchmark_environment.json declares the pin this commit
expects, so you can check your own build against it: python -m bench
container-run prints the pin of the image it runs.
data/benchmark_environment.json also names the shared, licence-free
image (shared_image) and its tools stage (tools_image) by digest,
with each one’s pin. Where they are pulled from is up to your deployment;
python -m bench.environment declared tools_image prints a field and
checks the file.
When the image changes
Any change to the Dockerfiles, a dependency set or the manifest generator is a new environment. In the same commit as the change:
Rebuild the shared image and its
toolsstage (podman build --target tools), and the licence layer on top.Record each pin:
python3 -m bench.environment hash /etc/aurora-bench/environment.jsonin the licensed image (manifest_sha256) and in the shared one (shared_manifest_sha256), and thetools_stagethe shared manifest records.Push the shared image and the tools image wherever your machines pull from, and record each digest the push reports (
--digestfile) asshared_imageandtools_image.Check the file with
python -m bench.environment declared manifest_sha256, and A/B the suite before and after, as for any change to the environment.
Then build each corpus’s compile database (its declared default build, run
inside a throwaway container from the tools stage) and run the
benchmarks in the image:
python -m bench container-build-db --codebase curl,lua
python -m bench container-run -- realworld-run --tool sqc --codebase curl,lua
python -m bench container-run -- juliet
A run in the declared environment gets the plain run id. The same run in
another image is suffixed -env<hash>; outside any image (bench
realworld-run or bench juliet on the host) -hostenv; against
another header tree -hdr-<id>, or -hdr-host for the host’s own
headers. Only the plain id is a benchmark run.
container-run builds aurora-lint inside the image, into a podman volume
named aurora-bench-target-<pin>-<checkout>: one per environment pin
and per checkout path. Every checkout is mounted at the same /work and
cargo judges a build fresh by source mtimes, so a target shared between
checkouts could hand one checkout another’s binary. A separate volume per
checkout prevents that. The cost is that the first run from a new checkout
path is a cold release build (a few minutes), and each throwaway A/B
worktree gets a fresh volume holding a full release target (most of a
gigabyte). Deleting a worktree does not delete its volume, and rebuilding
the image leaves the old layers behind as dangling images, so storage
grows with every A/B until it is pruned:
python -m bench container-prune # list what would go
python -m bench container-prune --yes # remove it
A volume is removed only when its checkout hash matches none of this
repository’s live checkouts (the checkout itself and its git worktree
list), hashed as container-run names it, plus the older per-pin-only
aurora-bench-target-<pin> volumes, which nothing uses any more. A worktree
whose path is missing counts as deleted unless it is locked (git worktree
lock), which is how to keep the volume of a worktree on a disk that is not
mounted.
Every other clone counts as deleted. Name each clone whose volumes should
stay with --keep DIR (repeatable); its worktrees are kept with it. A
--keep path that is not a git checkout stops the prune rather than keeping
nothing.
The images listed and removed are every dangling image of this user, not
only those an aurora-bench build left behind, since podman records
nothing that tells them apart. Each is removed by its id, without -f, so
an image a container was made from is kept. Tagged images and the
aurora-bench-cargo-registry volume are never touched.
The prune refuses while a benchmark container (a container-run or
container-build-db) or any container mounting a target volume is
running, and podman itself refuses a volume some container still uses. It
exits 0 when everything listed was removed (or, without --yes, listed),
1 when podman kept something, which it names, and 2 when it refused: podman
or git worktree list could not list what it needs, a --keep path is
no checkout, or a container is running.
The build also records the checkout’s commit in the binary’s --version,
and before scanning container-run checks it (python -m bench.container
check-binary). If the binary does not report exactly the commit the run
will be labelled with, the run stops with exit status 2 and a line ending
refusing to scan, and nothing is scanned or recorded. The line names the
commit the binary reports, if any. A build given the commit always
recompiles for it, so this means the volume holds a binary the build did
not replace. Remove that checkout’s volume and run again. If the refusal
persists, the build is not embedding the commit it is given; report it
rather than working around it, because a run past this check would measure
an unknown binary under this commit’s run id.
A run is labelled with the checkout’s commit, so it must measure that
commit. realworld-run, juliet and container-run refuse to start
when the checkout has uncommitted changes, staged, unstaged or untracked,
to anything that decides what a run measures. That covers the binary
(src/, build.rs, Cargo.toml, Cargo.lock,
rust-toolchain.toml, .cargo/config.toml), the manifests the
scans run under (rules_templates/, conf/), the runner itself
(bench/ apart from bench/tests/, and
scripts/generate_rule_cwe_map.py, which a Juliet run executes), and
the inputs it reads (data/benchmark_repos.json,
data/benchmark_deps/, data/benchmark_environment.json,
data/rule_cwe_map.json). The list is DIRTY_PATHS in
bench/config.py. The refusal lists the changed files. Commit them
before benchmarking. Changes elsewhere (docs/, tests/, an untracked
note at the top level) and gitignored files do not count.
For an exploratory run on uncommitted work, pass --allow-dirty to the
subcommand (python -m bench juliet --allow-dirty, python -m bench
container-run -- realworld-run --allow-dirty). The run is then recorded
as <sha>+dirty<hash>, where the hash covers the changes themselves, so
it never shares a run id with the clean commit or with a different set of
changes. Such a run is not a benchmark of any commit. In the container the
binary reports the same label, and check-binary expects it. The run
listings (realworld-runs, realworld, realworld-score) show the
mark after the shortened SHA.
A Juliet run rebuilds its rule-CWE map (data/rule_cwe_map.json and the
per-CWE manifests) before it scans. On a clean tree, if that rewrites a
committed file, the run refuses to continue: commit the regenerated files
first. With --allow-dirty the map is rebuilt before the run is
labelled, so the label covers the files the run scans. container-run
rebuilds it on the host before hashing the tree, for the same reason.
Untracked sources inside a corpus checkout are corpus-check’s
concern, not this check’s.
Both runners refuse to scan a benchmark whose dependency set is missing or
differs from its pin. On the host, fetch a set with python -m bench.deps
fetch <name> (data/benchmark_deps/<name>.json; it needs Python 3 and
the network, not dpkg), or all of them with
playbooks/setup-benchmark-deps.yml.
python -m bench corpus-check checks the sets where this machine’s runs
read them, which it is told with --mode (or AURORA_BENCH_MODE in
.env; the default is host). In host mode a set missing from
$SQC_BENCH_ROOT/deps fails the check. On a machine that runs benchmarks
only through container-run, use --mode container. The host sets are
then reported as not needed, and the check instead requires the benchmark
image (--image, default container-run’s) to be present with the
declared manifest_sha256.
The sections below describe scanning against the host’s own headers: local and shadow scans, which are never benchmark runs.
Third-Party Library Headers
aurora-lint uses -I include paths to resolve #include directives from third-party
libraries. Without these, functions declared in external headers produce
DCL31-C/DCL07-C false positives. The installed headers are therefore part
of what a run measures – once the full list below is installed, the same
binary on two machines differs by only a small number of keys (see
Reproducing the Published Numbers).
That “small number of keys” figure describes drift after the full list is
installed. Installing only part of it is a different, much larger failure
mode, and an easy one to fall into since the per-project breakdown below is
split into several blocks: a scan missing even one project’s packages
(easy to do working through them top-to-bottom rather than installing the
one-liner) has measurably produced several hundred extra
findings on a several-thousand-finding run (a high-single-digit percentage
of the total), and a project’s precision can sit measurably off a published
number until every package for it is installed – with nothing in the tool’s
own output signaling that more packages apply. corpus-check does not
catch this either: it checks corpus pin drift and untracked/gitignored
.c/.h files, not installed packages.
Install the full one-liner below rather than reasoning about which per-project subset your codebase needs:
sudo apt-get install -y libssl-dev libcjson-dev zlib1g-dev libcunit1-dev \
libsqlite3-dev libmbedtls-dev libgnutls28-dev libc-ares-dev tcl-dev \
libnl-3-dev libnl-genl-3-dev libdbus-1-dev libgcrypt20-dev libpcap-dev \
libwolfssl-dev libtomcrypt-dev
# x86_64 hosts only: the package exists only on amd64
sudo apt-get install -y libc6-dev-i386
The benchmark runner scans curl, hostap, mosquitto, sqlite and valkey
against these host headers, with aurora-lint as with cppcheck and clang-tidy.
The benchmark node carries every one of them except libnl-genl-3-dev;
scanning against the node’s headers plus that package reproduced its hostap
findings key for key, so the extra package moves nothing. Which of them are
present, not only their versions, changes findings: see
What another machine reproduces in Reproducing the Published Numbers.
That is expected, not a defect to engineer away. Official figures come from
the benchmark node alone, on its own headers, and the maintainer records that
node’s header packages; a run on any other machine is for A/B comparison
during development and is not citable on its own
(docs/adr/0004-postgres-is-the-single-source-of-truth.md). For a run that
must not depend on the host, the pinned trees in Debian System Headers
below are an opt-in.
Per-project breakdown, for reference (installing the one-liner above already covers all of these):
# Core dependencies (covers most projects)
sudo apt-get install -y \
libssl-dev libcjson-dev zlib1g-dev
# mosquitto
sudo apt-get install -y libcunit1-dev libsqlite3-dev
# curl TLS backends and the c-ares resolver (curl's DCL31-C results also
# depend on whether /usr/include/sys exists; see
# reproducing-published-numbers)
sudo apt-get install -y libmbedtls-dev libgnutls28-dev libc-ares-dev
# sqlite test infrastructure
sudo apt-get install -y tcl-dev
# hostap
sudo apt-get install -y \
libnl-3-dev libnl-genl-3-dev libdbus-1-dev \
libgcrypt20-dev libpcap-dev libwolfssl-dev libtomcrypt-dev
# the benchmark node's multilib layout: puts sys/, bits/ and asm/ directly
# under /usr/include, which every host-header scan then reaches. amd64
# only; an arm64 host has no equivalent
sudo apt-get install -y libc6-dev-i386
Windows SDK and CRT Headers (ventoy)
ventoy is Win32 code, so a Linux or macOS node has none of its system
headers. Without <windows.h> every Win32 call is an undeclared identifier:
the scan reports several hundred DCL31-C findings that describe the node, not
the code, and API00-C and INT30-C see different code too. So ventoy is scanned
against a pinned tree of the Windows SDK and MSVC CRT headers, declared in
data/benchmark_repos.json (header_tree) next to its commit pin: the
exact xwin versions that produce it,
and a manifest hash over the header files (definition in
bench/header_tree.py). The tree was pinned after a scan against it gave
findings identical to a scan against a native Visual Studio installation of
the same SDK and CRT.
The headers are Microsoft’s. Fetching them accepts Microsoft’s license terms for the Windows SDK and the MSVC CRT, so the playbook only does it when you say so:
ansible-playbook playbooks/setup-benchmark-repos.yml -i "localhost," -c local \
-e accept_microsoft_license=true --tags header-trees
(setup-benchmark-repos-macos.yml on macOS; the step is the same.) It
installs the pinned xwin with cargo if needed, downloads about 120 MiB,
keeps the headers (about 370 MB) under
$SQC_BENCH_ROOT/header-trees/<id>/, and fails if their hash differs from
the pin. Each machine fetches its own copy: never commit the tree or copy it
between machines.
corpus-check reports the tree as MISSING or MISMATCH like a
drifted commit, and realworld-run refuses to scan ventoy without a
matching tree rather than produce header-less findings. Each scan records the
tree it used (header_tree) in its .meta.json sidecar. cppcheck and
clang-tidy are not given the tree: their ventoy baselines remain header-less.
To re-pin (a new SDK or CRT), change the fetch versions and the id,
fetch the tree, print its hash with
python3 -m bench.header_tree hash ventoy <tree-dir>, confirm the findings
match a native-header scan, and only then declare the new hash.
Debian System Headers (curl, hostap, mosquitto, sqlite, valkey)
These five corpora pass -I /usr/include (and a few of its
subdirectories), so their findings depend on the host: on which -dev
packages it has installed, not only on their versions. A second machine’s
run of one release differed from the benchmark node’s by about a hundred
keys, every one traced to a header package (see What another machine
reproduces in Reproducing the Published Numbers). By default the runner
scans them against the host’s own headers all the same, and records that in
each scan’s .meta.json sidecar (header_tree: {"id": "host", ...}):
official figures are the benchmark node’s, on that node’s headers, and a
local run is for A/B comparison on one machine, where both arms share the
host (docs/adr/0004-postgres-is-the-single-source-of-truth.md).
Pinned Debian 12 /usr/include trees are an opt-in reproduction aid, for
when a run must not depend on the host: reproducing the benchmark node’s
environment on another machine, or comparing machines.
python -m bench realworld-run --header-tree <id> scans the five corpora
against one, under a run id of its own (-hdr-<id>), so its findings never
land under the default run’s. --header-tree host is the default spelled
out. The trees are declared in data/benchmark_repos.json under
header_trees; the five corpora name the prefix they read from the host
(host_headers), which a tree must replace to stand in for it:
debian12-declared-*: each corpus’s declared dependencies (OpenSSL, zlib, SQLite, Tcl and ICU for sqlite; CUnit and cJSON for mosquitto; mbed TLS, GnuTLS and c-ares for curl; libgcrypt, libpcap, libnl, D-Bus, wolfSSL and libtomcrypt for hostap) plus the C library and kernel headers.debian12-benchmark-node-*: the header packages the benchmark node had when thev0.6.0figures were measured. Scanning against it reproduced those runs key for key on another machine.
Each package is pinned by version and by the sha256 of its .deb at
snapshot.debian.org. The runner rewrites every -I /usr/include...
into the tree and puts the tree’s own include directories (the declared
tree’s multiarch usr/include/x86_64-linux-gnu) first, in a compiler’s
search order. Fetching needs Python 3 and the network, not dpkg, so a
macOS, Fedora or FreeBSD node unpacks the same tree a Debian one does.
Nothing fetches a tree by default; ask for it by id:
python3 -m bench.header_tree fetch debian12-declared-3af9978e
or through the playbook, with
--tags header-trees -e debian_header_trees=<id>[,<id>] (no license flag).
A tree takes about 30 MB under $SQC_BENCH_ROOT/header-trees/<id>/, plus a
cache of its .deb files beside it. corpus-check does not require
one. realworld-run --header-tree <id> refuses to scan against a tree
that is missing or whose hash differs from the pin.
Read a comparison between two trees, or between a tree and the host, as the
whole environment change, not one part of it: the two trees differ in their
package sets and in how <sys/...> and <bits/...> are found (the
declared tree searches its multiarch directory first; the benchmark-node
tree has none on the search path and reaches those headers through the
multilib links under /usr/include, as the node itself did).
A run with --compile-commands also passes the compile database’s own
-I paths, which are not rewritten into the tree.
To re-pin, list the packages as PACKAGE=VERSION and run
python3 -m bench.header_tree pin PKG=VER ..., which prints the debs
entries (file, snapshot URL, sha256). Unpack them, print the tree’s hash with
python3 -m bench.header_tree hash <old-id> <tree-dir>, and declare a new
id and hash; a changed tree changes findings, so re-pin at a release
boundary and measure the change.
Per-Project Include Paths
The real-world runner passes each codebase’s -I paths automatically, from
the includes field of its entry in CODEBASES in
bench/realworld_runner.py; that registry is the list. The corpora that
read /usr/include are the five named under Third-Party Library Headers
above.
Real-World Project Setup
Pinned Source Commits
All benchmark results are run against these exact commits – the full set of
projects in bench/realworld_runner.py’s CODEBASES registry
(which stores paths and per-tool flags, not the pins).
Note
The single source of truth for the pins is
data/benchmark_repos.json. Both
playbooks/setup-benchmark-repos.yml (provisioning) and python -m
bench corpus-check (verification) read it. The table below is a
human-readable mirror – if the two disagree, the JSON wins. Do not add
a third copy.
Each checkout must sit at a detached HEAD on its pinned commit. A
checkout left on a tracking branch is not pinned even while it happens to
match: the next git pull moves it silently, and every finding then lands
at (file, line) pairs the ground_truth oracle – keyed on
project+commit+file+line+rule – was never adjudicated against, so it drops
out of the precision/recall denominator without any error. This is not
hypothetical; see Verifying the Pins below.
Project |
Repository |
Commit SHA |
Checkout state |
|---|---|---|---|
libcrc |
|
detached |
|
sqlite |
|
detached |
|
mosquitto |
|
detached |
|
curl |
|
detached |
|
hostap |
|
detached |
|
lua |
|
detached |
|
raylib |
|
detached |
|
pureftpd |
|
detached |
|
sel4 |
|
detached |
|
mbedtls |
|
detached |
|
valkey |
|
detached |
|
ventoy |
|
detached |
Important
pureftpd’s checkout directory must be named pureftpd (no
hyphen), not the upstream pure-ftpd – two path-parsing spots in
bench/db.py (result-filename parsing, ground-truth path
normalization) require the checkout dir basename to match the registry
key exactly. The playbook already clones it to the right name.
Verifying the Pins
Provisioning pins the checkouts once; nothing kept them pinned. Run this before any real-world benchmark or precision claim:
python -m bench corpus-check # exits 1 if anything is off
python -m bench corpus-check --json # machine-readable
It reports one row per project, worst first:
Status |
Meaning |
|---|---|
MISSING |
Checkout directory absent |
NOT_GIT |
Directory exists but is not a git checkout |
PIN_ABSENT |
Pinned commit not present locally (needs a fetch) |
DRIFTED |
|
UNPINNED |
|
OK |
Detached at the pinned commit |
It also flags three contamination cases independent of status:
tracked files modified, so the scanned source is not the pin;
untracked ``*.c``/``*.h`` files, which aurora-lint will scan and attribute to the pinned commit (untracked files aurora-lint ignores are counted separately and are harmless);
gitignored ``*.c``/``*.h`` files, which are invisible to
git statusbut which aurora-lint scans anyway – it dispatches on file extension and never consults git. The case to watch is a build run inside a checkout: sqlite’s build generates a gitignoredsqlite3.camalgamation, which would silently add ~250k lines to every sqlite scan.
Tip
Keep the checkouts pristine. Build artifacts belong in a separate tree,
not inside $SQC_BENCH_ROOT.
Warning
This check exists because the failure is silent and did happen. On one
machine curl, hostap and sqlite had all drifted, while libcrc and lua
sat on tracking branches matching their pins only by coincidence. A gate
run against the drifted trees reported slightly more hostap and sqlite
findings than the pinned snapshots give – close enough to look
plausible, and not comparable to ground_truth at all. Fix drift with
the git checkout --detach command the check prints for each row.
Clone and Pin
Preferred: run playbooks/setup-benchmark-repos.yml (see Benchmark Host
Layout above) – it clones, pins, and verifies every pinned checkout in one pass,
reading the pins from data/benchmark_repos.json.
Manual fallback: clone each repository in data/benchmark_repos.json
under its name (the registry key, so pure-ftpd lands in pureftpd),
detach at its version, and then verify:
mkdir -p $SQC_BENCH_ROOT # default: ~/toolchain
cd $SQC_BENCH_ROOT
python3 -c 'import json, sys
for r in json.load(open(sys.argv[1]))["repos"]:
print(r["repo"], r["name"], r["version"])' \
$AURORA_LINT_SRC_ROOT/aurora-lint/data/benchmark_repos.json |
while read -r repo name sha; do
git clone "$repo" "$name" && git -C "$name" checkout --detach "$sha"
done
git -C mbedtls submodule update --init -- framework
cd $AURORA_LINT_SRC_ROOT/aurora-lint && python -m bench corpus-check
A corpus with git submodules lists them under submodules (today only
mbedtls’s framework), and each is checked out at the commit the pinned
superproject records. The aurora-lint scan of library/ never reads
framework/, but the library build does, so the compile database cannot be
built without it. corpus-check fails a declared submodule that is missing,
at another commit or modified, and a submodule in a pinned tree that is not
declared. Do not build inside the valkey
checkout: make leaves a gitignored src/release.h that corpus-check
will (correctly) refuse.
Running Each Tool Manually
Note
The examples below use an illustrative ~/data/comparisons/<project>
checkout path for ad hoc manual runs. If you provisioned codebases via
playbooks/setup-benchmark-repos.yml, substitute
$SQC_BENCH_ROOT/<project> (default ~/toolchain/<project>) instead
– that’s the path bench/realworld_runner.py itself reads.
aurora-lint
# Build
cd $AURORA_LINT_SRC_ROOT/aurora-lint
cargo build --release
# Basic run
./target/release/aurora-lint /path/to/source/ --export results.json
# With cross-file context
./target/release/aurora-lint /path/to/source/ -d /path/to/source/ --export results.json
# When running from outside the aurora-lint repo, pass --manifest explicitly
./target/release/aurora-lint /path/to/source/ \
--manifest $AURORA_LINT_SRC_ROOT/aurora-lint/rules_templates/rules-all.toml \
--export results.json
cppcheck
cppcheck --enable=all --std=c11 --xml --xml-version=2 \
--suppress=missingIncludeSystem \
-I /path/to/include \
/path/to/source/ \
2> results.xml
Warning
cppcheck writes XML to stderr, not stdout
Never pass
-I /usr/include– causes parse errors in system headers--addon=certdoes not exist: open-source cppcheck removed the CERT addon in 2.8 (CERT checking is Cppcheck Premium).--enable=allturns on cppcheck’s native checks, not CERT ones
clang-tidy
# Option A: With compile_commands.json (recommended)
cd /path/to/project
make clean && bear -- make -j$(nproc)
run-clang-tidy -checks='-*,cert-*,clang-analyzer-*' -p .
# Option B: Direct file scan (no build required)
find /path/to/source/ -name '*.c' | \
xargs -P $(nproc) -I{} clang-tidy \
-checks='-*,cert-*,clang-analyzer-*' \
{} -- -std=c11 -I /path/to/include
Warning
bearrequiresmake cleanfirst – it only captures invocations during an actual buildVerify
compile_commands.jsonis non-empty before running clang-tidy
Per-Project Commands
libcrc
# aurora-lint
$AURORA_LINT_SRC_ROOT/aurora-lint/target/release/aurora-lint ~/data/comparisons/libcrc \
-d ~/data/comparisons/libcrc \
--manifest $AURORA_LINT_SRC_ROOT/aurora-lint/rules_templates/rules-all.toml \
--export ~/data/comparisons/results/aurora-lint/libcrc/results.json
# cppcheck
cppcheck --enable=all --std=c11 --xml --xml-version=2 \
--suppress=missingIncludeSystem \
-I ~/data/comparisons/libcrc/include \
~/data/comparisons/libcrc/src/ \
2> ~/data/comparisons/results/cppcheck/libcrc/results.xml
# clang-tidy
cd ~/data/comparisons/libcrc && make clean && bear -- make
run-clang-tidy -checks='-*,cert-*,clang-analyzer-*' -p . \
2>&1 | tee ~/data/comparisons/results/clang-tidy/libcrc/results.txt
sqlite
# aurora-lint
$AURORA_LINT_SRC_ROOT/aurora-lint/target/release/aurora-lint ~/data/comparisons/sqlite \
-d ~/data/comparisons/sqlite \
--manifest $AURORA_LINT_SRC_ROOT/aurora-lint/rules_templates/rules-all.toml \
--export ~/data/comparisons/results/aurora-lint/sqlite/results.json
# cppcheck
cppcheck --enable=all --std=c11 --xml --xml-version=2 \
--suppress=missingIncludeSystem \
-I ~/data/comparisons/sqlite/src \
~/data/comparisons/sqlite/src/ \
2> ~/data/comparisons/results/cppcheck/sqlite/results.xml
# clang-tidy
cd ~/data/comparisons/sqlite && ./configure && make clean && bear -- make -j$(nproc)
run-clang-tidy -checks='-*,cert-*,clang-analyzer-*' -p . \
2>&1 | tee ~/data/comparisons/results/clang-tidy/sqlite/results.txt
mosquitto
# aurora-lint
$AURORA_LINT_SRC_ROOT/aurora-lint/target/release/aurora-lint ~/data/comparisons/mosquitto \
-d ~/data/comparisons/mosquitto \
--manifest $AURORA_LINT_SRC_ROOT/aurora-lint/rules_templates/rules-all.toml \
--export ~/data/comparisons/results/aurora-lint/mosquitto/results.json
# cppcheck
cppcheck --enable=all --std=c11 --xml --xml-version=2 \
--suppress=missingIncludeSystem \
-I ~/data/comparisons/mosquitto/include \
-I ~/data/comparisons/mosquitto/common \
-i ~/data/comparisons/mosquitto/deps \
~/data/comparisons/mosquitto/lib/ ~/data/comparisons/mosquitto/src/ \
2> ~/data/comparisons/results/cppcheck/mosquitto/results.xml
# clang-tidy
cmake -S ~/data/comparisons/mosquitto -B ~/data/comparisons/mosquitto/build \
-DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DWITH_TLS=OFF -DWITH_WEBSOCKETS=OFF -DWITH_TESTS=OFF
cmake --build ~/data/comparisons/mosquitto/build --clean-first -j$(nproc)
run-clang-tidy -checks='-*,cert-*,clang-analyzer-*' \
-p ~/data/comparisons/mosquitto/build \
2>&1 | tee ~/data/comparisons/results/clang-tidy/mosquitto/results.txt
curl
# aurora-lint
$AURORA_LINT_SRC_ROOT/aurora-lint/target/release/aurora-lint ~/data/comparisons/curl \
-d ~/data/comparisons/curl \
--manifest $AURORA_LINT_SRC_ROOT/aurora-lint/rules_templates/rules-all.toml \
--export ~/data/comparisons/results/aurora-lint/curl/results.json
# cppcheck
cppcheck --enable=all --std=c11 --xml --xml-version=2 \
--suppress=missingIncludeSystem \
-I ~/data/comparisons/curl/include -I ~/data/comparisons/curl/lib \
~/data/comparisons/curl/lib/ ~/data/comparisons/curl/src/ \
2> ~/data/comparisons/results/cppcheck/curl/results.xml
# clang-tidy
cmake -S ~/data/comparisons/curl -B ~/data/comparisons/curl/build \
-DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DBUILD_SHARED_LIBS=OFF \
-DCURL_USE_OPENSSL=OFF -DCURL_DISABLE_LDAP=ON \
-DUSE_LIBIDN2=OFF -DUSE_NGHTTP2=OFF -DCURL_ZSTD=OFF
cmake --build ~/data/comparisons/curl/build --clean-first -j$(nproc)
run-clang-tidy -checks='-*,cert-*,clang-analyzer-*' \
-p ~/data/comparisons/curl/build \
2>&1 | tee ~/data/comparisons/results/clang-tidy/curl/results.txt
hostap
# aurora-lint
$AURORA_LINT_SRC_ROOT/aurora-lint/target/release/aurora-lint ~/data/comparisons/hostap \
-d ~/data/comparisons/hostap/src \
-d ~/data/comparisons/hostap/wpa_supplicant \
--manifest $AURORA_LINT_SRC_ROOT/aurora-lint/rules_templates/rules-all.toml \
--export ~/data/comparisons/results/aurora-lint/hostap/results.json
# cppcheck
cppcheck --enable=all --std=c11 --xml --xml-version=2 \
--suppress=missingIncludeSystem \
-I ~/data/comparisons/hostap/src \
-I ~/data/comparisons/hostap/src/utils \
-I ~/data/comparisons/hostap/src/common \
~/data/comparisons/hostap/src/ ~/data/comparisons/hostap/wpa_supplicant/ \
2> ~/data/comparisons/results/cppcheck/hostap/results.xml
# clang-tidy (requires .config for wpa_supplicant)
cd ~/data/comparisons/hostap/wpa_supplicant
cat > .config <<'EOF'
CONFIG_DRIVER_NL80211=y
CONFIG_LIBNL32=y
CONFIG_IEEE8021X_EAPOL=y
CONFIG_EAP_MD5=y
EOF
make clean 2>/dev/null; bear -- make -j$(nproc) wpa_supplicant 2>/dev/null || true
cd ~/data/comparisons/hostap
python3 gen_compile_commands.py -o compile_commands.json wpa_supplicant/
run-clang-tidy -checks='-*,cert-*,clang-analyzer-*' -p . \
2>&1 | tee ~/data/comparisons/results/clang-tidy/hostap/results.txt
Verifying Results
PROJECT=libcrc
RESULTS=~/data/comparisons/results
echo "=== aurora-lint ==="
python3 -c "
import json, collections
data = json.load(open('$RESULTS/aurora-lint/$PROJECT/results.json'))
rules = collections.Counter(v['rule_id'] for v in data)
print(f'Total: {len(data)} findings, {len(rules)} rules')
for r, c in rules.most_common(10): print(f' {r}: {c}')
"
echo "=== cppcheck ==="
python3 -c "
import xml.etree.ElementTree as ET, collections
errors = ET.parse('$RESULTS/cppcheck/$PROJECT/results.xml').getroot().findall('.//error')
ids = collections.Counter(e.get('id') for e in errors)
print(f'Total: {len(errors)} findings')
for i, c in ids.most_common(10): print(f' {i}: {c}')
"
echo "=== clang-tidy ==="
grep -c "warning:\|error:" $RESULTS/clang-tidy/$PROJECT/results.txt
Known Pitfalls
Pitfall |
Symptom |
Fix |
|---|---|---|
aurora-lint manifest not found |
|
Pass |
cppcheck |
|
Remove it; the addon is gone from open-source cppcheck (2.8 on), so there are no CERT checks to enable |
cppcheck |
|
Never pass system include paths to cppcheck |
Empty compile_commands.json |
clang-tidy shows no diagnostics |
Run |
Output Format Reference
cppcheck XML (v2):
<error id="bufferAccessOutOfBounds" severity="error"
msg="Array 'arr[10]' accessed at index 10..." cwe="788">
<location file="example.c" line="5" column="5"/>
</error>
clang-tidy text:
example.c:5:5: warning: ... [cert-arr30-c]
aurora-lint JSON:
[{"rule_id": "ARR30-C", "severity": "High", "file": "example.c", "line": 5, "message": "..."}]
aurora-lint SARIF: SARIF 2.1.0 compatible output with ruleId matching CERT C rule IDs.
Infer JSON (infer-out/report.json):
[{"bug_type": "NULLPTR_DEREFERENCE", "procedure": "func_bad",
"file": "test.c", "line": 26, "severity": "ERROR"}]
Frama-C EVA (stderr, no structured output):
[eva:alarm] test.c:26: Warning:
out of bounds read. assert \valid_read(&ptr->field);
assertion 'Eva,mem_access' got final status invalid.
Distributed Benchmarking
Multi-host runs belong to the maintainer’s benchmarking_db infrastructure,
not to this repo.
Exporting competitor results for ingest
bench/competitors.py writes one JSON blob per run into
data/competitor_results/, and those blobs are committed — they are the
archival form and should not be edited or deleted. They are not, however,
ingestible: the shape is nested and the per-CWE map is keyed by CWE id, so a
loader would have to know that module’s internal layout to read it.
python -m bench competitor-export flattens every run JSON in that
directory into three CSVs beside them:
competitor_runs.csv one row per run (tool, version, totals)
competitor_cwe_results.csv one row per run × CWE (the measurements)
competitor_cwe_errors.csv one row per error (usually empty)
run_key joins them and is the JSON’s own basename (e.g.
framac_20260403_222053), so it is unique, stable across re-exports, and
traceable back to the file it came from.
Run the export after every competitor benchmark and commit the result.
The benchmark host ingests these CSVs into sqc_bench; a table missing its
newest run looks exactly like a run that never happened.
python -m bench competitor-export --check exits nonzero if the CSVs are
stale, and writes nothing.
This repo stays Postgres-blind — no DSN, no connection code (see
CLAUDE.md). It emits a flat, diffable table; benchmarking_db owns
the ingest, so nothing here changes when the target schema does. The CSVs
are a transport, not a source of truth: Postgres remains the only place an
official number comes from, and re-exporting is idempotent and lossless with
respect to the JSON.
Note
The run JSONs record no hostname. run_hosts.json attributes each
run, and the CSV carries it as hostname + hostname_source.
Wall clock is hardware-dependent, so compare durations only between runs
on the same host.