Benchmark Setup

This section covers installing comparison tools, setting up the Juliet test suite, and configuring real-world codebases for benchmarking.

Benchmark Host Layout

Every machine that runs benchmarks needs one directory – $SQC_BENCH_ROOT (default ~/toolchain) – holding the Juliet suite and every real-world codebase checkout. bench/config.py and bench/realworld_runner.py both read this from the SQC_BENCH_ROOT environment variable, or from a .env file at the repo root (copy .env.example – it’s gitignored, so each machine sets its own).

To provision a fresh node, run the Ansible playbook (clones and pins every real-world codebases; does not fetch Juliet – see below):

ansible-playbook playbooks/setup-benchmark-repos.yml -i "localhost," -c local
# Or a different root:
ansible-playbook playbooks/setup-benchmark-repos.yml -i "localhost," -c local \
  -e bench_root=/data/toolchain

Keep bench_root and SQC_BENCH_ROOT/.env pointed at the same directory – the playbook provisions the checkouts the code actually reads. A mismatch is easy to miss: if SQC_BENCH_ROOT is unset and the corpora do not live in ~/toolchain, BENCH_ROOT silently resolves to a directory that does not exist. python -m bench corpus-check reports the resolved root and says so outright.

A second, separate root names where the source clones are: $AURORA_LINT_SRC_ROOT is the parent directory of this checkout ($AURORA_LINT_SRC_ROOT/aurora-lint), and, on the maintainer’s machine, of the sibling benchmarking_db and sqc_paper clones too. It defaults to the checkout’s own parent, so nothing here requires it; the copy-pasteable commands below use it so they hold on any machine regardless of where the clone lives (~/data is the conventional value, and the .env.example default):

export AURORA_LINT_SRC_ROOT=~/data   # parent of your aurora-lint clone

Installing Comparison Tools

cppcheck

sudo apt update
sudo apt install -y cppcheck

# Verify
cppcheck --version
# Expected: Cppcheck 2.x.x

For the latest release, build from source:

sudo apt install -y cmake libpcre3-dev
git clone https://github.com/danmar/cppcheck.git
cd cppcheck
cmake -DCMAKE_BUILD_TYPE=Release -DUSE_MATCHCOMPILER=ON .
make -j$(nproc)
sudo make install

clang-tidy

Pin LLVM 21 — do not take the distro default. Ubuntu 24.04 ships clang-tidy 18.1.3 and tops out at 20, while the published comparison in docs/tool-comparison.rst is LLVM 21.1. Taking the distro package makes a freshly-provisioned node measure an older competitor than the table it will be compared against, and clang-tidy is the tool beating aurora-lint on the Juliet overlap in that comparison — so that regression would flatter aurora-lint by understating a rival. playbooks/install-static-analyzers.yml does this automatically; by hand:

sudo install -d -m 0755 /etc/apt/keyrings
wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key \
  | sudo tee /etc/apt/keyrings/apt.llvm.org.asc > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/apt.llvm.org.asc] \
  http://apt.llvm.org/noble/ llvm-toolchain-noble-21 main" \
  | sudo tee /etc/apt/sources.list.d/llvm-21.list
sudo apt update && sudo apt install -y clang-tidy-21
sudo update-alternatives --install /usr/bin/clang-tidy clang-tidy \
  /usr/bin/clang-tidy-21 100

# Verify
clang-tidy --version
# Expected: Ubuntu LLVM version 21.1.x

The distro package is fine for anything that is not a published comparison:

sudo apt install -y clang clang-tidy

bear (for compile_commands.json)

Required for clang-tidy on projects with build systems:

sudo apt install -y bear

Facebook Infer

Infer v1.2.0 is installed from prebuilt Linux binaries. The playbook at playbooks/install-static-analyzers.yml automates this:

ansible-playbook playbooks/install-static-analyzers.yml \
  -i "localhost," -c local --ask-become-pass

Manual install:

# libtinfo5 required (Infer links against libtinfo.so.5)
sudo apt install -y libtinfo5 || \
  sudo ln -s /usr/lib/x86_64-linux-gnu/libtinfo.so.6 \
              /usr/lib/x86_64-linux-gnu/libtinfo.so.5

VERSION=1.2.0
curl -sSL "https://github.com/facebook/infer/releases/download/v$VERSION/infer-linux-x86_64-v$VERSION.tar.xz" \
  | sudo tar -C /opt -xJ
sudo ln -s "/opt/infer-linux-x86_64-v$VERSION/bin/infer" /usr/local/bin/infer

# Verify
infer --version
# Expected: Infer version v1.2.0

Frama-C

Frama-C is installed via opam (the Debian 12 apt package is version 25/Manganese, too old for benchmarking). The playbook handles this automatically.

Manual install:

sudo apt install -y opam gcc g++ make m4 pkg-config autoconf \
  libgmp-dev zlib1g-dev libgtk-3-dev libgtksourceview-3.0-dev graphviz

opam init --bare --no-setup --disable-sandboxing
opam switch create default 4.14.2
eval $(opam env)
opam install -y frama-c

# Verify (requires eval $(opam env) in each shell session)
frama-c -version
# Expected: 33.0 (Arsenic) as of 2026-09-04; 32.0 (Germanium) is what the
# competitor Juliet runs in data/competitor_results/ were taken with

Note

Add eval $(opam env) to your shell profile (~/.bashrc) to make frama-c available without manual activation. The benchmark runner does not need it: bench/realworld_runner.py wraps every Frama-C invocation in eval $(opam env) itself, because it is not a login shell.

Warning

Frama-C renamed the compile-database option between the two versions above — -json-compilation-database through 32.0, -compilation-db from 33.0. The runner probes frama-c -kernel-h and picks whichever the installed binary accepts, so both work; a hand-written command line copied from an older doc will abort with “option is unknown”.

Build-based tools need a compile database

Infer and Frama-C cannot be pointed at a source tree the way aurora-lint and cppcheck can — both need a real preprocess. On the real-world suite they are driven from each checkout’s compile_commands.json, so playbooks/setup-compile-commands.yml must have run for a codebase before either tool can scan it. python -m bench realworld-run --tool infer fails fast with the exact command to fix it if the database is missing.

Expect a partial result from both, and check the coverage block each run writes:

  • Infer captures whatever preprocesses. setup-compile-commands.yml restores each checkout to pristine after building it, which deletes the build’s generated headers; the playbook now stashes the ones the compile database still references (pure-ftpd’s config.h, sqlite’s sqlite_cfg.h, valkey’s jemalloc/jemalloc.h, libcrc’s tab/gentab*.inc) under <bench_root>/.cc_build/generated/ and points the database there, so every in-scope translation unit preprocesses. A capture_failures list that is not empty means a database written before that existed — re-run the playbook, which detects and rebuilds it.

  • Frama-C is partial by construction, because EVA analyzes one entry point at a time and a real codebase has no single one. Read docs/design/framac-realworld.md before quoting any Frama-C real-world number: a finding count from it is a floor, and recall is not expressible from it at all.

Reference Shadow Set (optional)

The reference shadow set is a list of pinned C codebases, disjoint from the real-world benchmarks, that is never labeled or tuned against. Its only use is python -m bench reference-ab: what a change moves on code nobody tuned against (docs/design/reference-corpus-ab.md). Nothing else needs it, so it is a one-time setup for anyone who wants to run that A/B:

ansible-playbook playbooks/setup-shadow-corpus.yml -i "localhost," -c local
ansible-playbook playbooks/setup-shadow-corpus.yml -i "localhost," -c local \
  -e shadow_tier=all     # every tier, the large one included

It downloads shallow checkouts, each at its pin in data/reference_corpus.json, under $SQC_BENCH_ROOT/shadow_corpus/. The tiers are cumulative: quick, standard (the default, about 1 GB), and large or all (about 5 GB). The playbook checks first that about twice the tier’s size is free, unless every checkout is already at its pin. The checkouts stay where they are: reference-ab re-checks and cleans each one at its pin before scanning it, so nothing needs fetching again until the pins change. python -m bench reference-fetch --tier T does the same fetch without Ansible.

The large tier’s scans are slow and memory-heavy. Expect hours for an A/B over it, and a peak near 12 GB resident at 8 threads; on a machine with under 16 GiB of RAM, reference-ab halves its default --jobs for it.

Juliet Test Suite Setup

The setup playbook does not fetch Juliet; it only creates the parent directory and reports whether the suite is already in place. The NIST Juliet Test Suite v1.3 page links a single zip that downloads directly, with no click-through. NIST does not promise that URL stays put, so if the curl below fails, start from that page. The archive extracts to C/, which is renamed to the directory name bench/config.py expects:

sudo apt install -y curl unzip
mkdir -p $SQC_BENCH_ROOT/benchmarks   # default: ~/toolchain/benchmarks
cd $SQC_BENCH_ROOT/benchmarks
curl -LO https://samate.nist.gov/SARD/downloads/test-suites/2017-10-01-juliet-test-suite-for-c-cplusplus-v1-3.zip
unzip -q 2017-10-01-juliet-test-suite-for-c-cplusplus-v1-3.zip
mv C juliet-test-suite-c
# Expected structure:
#   juliet-test-suite-c/testcases/CWE*/     (118 CWE directories)
#   juliet-test-suite-c/testcasesupport/    (shared helper functions)

The per-CWE manifests that fast mode runs (rules_templates/cwe/*.toml) are generated by scripts/generate_rule_cwe_map.py and are not committed, so that directory is nearly empty in a fresh clone. python -m bench juliet runs the script for you before every run, which also refreshes data/rule_cwe_map.json.

The Benchmark Environment

A benchmark run, real-world or Juliet, is pinned by five things: the corpus commit, the aurora-lint commit, the oracle commit, the settings hash and the environment it ran in (docs/adr/0018). The environment is a container image, container/benchmark.Dockerfile, holding a Debian 12 (bookworm) snapshot, every benchmark’s dependency set (the system headers it is scanned against, each in its own tree), the corpus build tools and the comparison tools at exact versions. The corpus checkouts and this repository are mounted into it at run time, and aurora-lint is built inside it.

Build the image with podman (rootless is fine). The build is amd64 on every host, an arm64 Mac included:

podman build --platform linux/amd64 --target tools \
  -f container/benchmark.Dockerfile -t aurora-bench-tools:dev .
podman build --platform linux/amd64 \
  -f container/benchmark.Dockerfile -t aurora-bench:shared .

That image leaves out the Win32 corpus’s set, the Windows SDK and MSVC CRT headers, which are Microsoft’s: building the layer that adds them accepts Microsoft’s licence terms for your machine, and the result must never be copied to another one:

podman build --platform linux/amd64 -f container/licence.Dockerfile \
  --build-arg BENCH_IMAGE=aurora-bench:shared \
  --build-arg ACCEPT_MICROSOFT_LICENSE=true -t aurora-bench:dev .

The image records its environment manifest in /etc/aurora-bench/environment.json; the SHA-256 of that manifest is the pin. data/benchmark_environment.json declares the pin this commit expects, so you can check your own build against it: python -m bench container-run prints the pin of the image it runs.

data/benchmark_environment.json also names the shared, licence-free image (shared_image) and its tools stage (tools_image) by digest, with each one’s pin. Where they are pulled from is up to your deployment; python -m bench.environment declared tools_image prints a field and checks the file.

When the image changes

Any change to the Dockerfiles, a dependency set or the manifest generator is a new environment. In the same commit as the change:

  1. Rebuild the shared image and its tools stage (podman build --target tools), and the licence layer on top.

  2. Record each pin: python3 -m bench.environment hash /etc/aurora-bench/environment.json in the licensed image (manifest_sha256) and in the shared one (shared_manifest_sha256), and the tools_stage the shared manifest records.

  3. Push the shared image and the tools image wherever your machines pull from, and record each digest the push reports (--digestfile) as shared_image and tools_image.

  4. Check the file with python -m bench.environment declared manifest_sha256, and A/B the suite before and after, as for any change to the environment.

Then build each corpus’s compile database (its declared default build, run inside a throwaway container from the tools stage) and run the benchmarks in the image:

python -m bench container-build-db --codebase curl,lua
python -m bench container-run -- realworld-run --tool sqc --codebase curl,lua
python -m bench container-run -- juliet

A run in the declared environment gets the plain run id. The same run in another image is suffixed -env<hash>; outside any image (bench realworld-run or bench juliet on the host) -hostenv; against another header tree -hdr-<id>, or -hdr-host for the host’s own headers. Only the plain id is a benchmark run.

container-run builds aurora-lint inside the image, into a podman volume named aurora-bench-target-<pin>-<checkout>: one per environment pin and per checkout path. Every checkout is mounted at the same /work and cargo judges a build fresh by source mtimes, so a target shared between checkouts could hand one checkout another’s binary. A separate volume per checkout prevents that. The cost is that the first run from a new checkout path is a cold release build (a few minutes), and each throwaway A/B worktree gets a fresh volume holding a full release target (most of a gigabyte). Deleting a worktree does not delete its volume, and rebuilding the image leaves the old layers behind as dangling images, so storage grows with every A/B until it is pruned:

python -m bench container-prune          # list what would go
python -m bench container-prune --yes    # remove it

A volume is removed only when its checkout hash matches none of this repository’s live checkouts (the checkout itself and its git worktree list), hashed as container-run names it, plus the older per-pin-only aurora-bench-target-<pin> volumes, which nothing uses any more. A worktree whose path is missing counts as deleted unless it is locked (git worktree lock), which is how to keep the volume of a worktree on a disk that is not mounted.

Every other clone counts as deleted. Name each clone whose volumes should stay with --keep DIR (repeatable); its worktrees are kept with it. A --keep path that is not a git checkout stops the prune rather than keeping nothing.

The images listed and removed are every dangling image of this user, not only those an aurora-bench build left behind, since podman records nothing that tells them apart. Each is removed by its id, without -f, so an image a container was made from is kept. Tagged images and the aurora-bench-cargo-registry volume are never touched.

The prune refuses while a benchmark container (a container-run or container-build-db) or any container mounting a target volume is running, and podman itself refuses a volume some container still uses. It exits 0 when everything listed was removed (or, without --yes, listed), 1 when podman kept something, which it names, and 2 when it refused: podman or git worktree list could not list what it needs, a --keep path is no checkout, or a container is running.

The build also records the checkout’s commit in the binary’s --version, and before scanning container-run checks it (python -m bench.container check-binary). If the binary does not report exactly the commit the run will be labelled with, the run stops with exit status 2 and a line ending refusing to scan, and nothing is scanned or recorded. The line names the commit the binary reports, if any. A build given the commit always recompiles for it, so this means the volume holds a binary the build did not replace. Remove that checkout’s volume and run again. If the refusal persists, the build is not embedding the commit it is given; report it rather than working around it, because a run past this check would measure an unknown binary under this commit’s run id.

A run is labelled with the checkout’s commit, so it must measure that commit. realworld-run, juliet and container-run refuse to start when the checkout has uncommitted changes, staged, unstaged or untracked, to anything that decides what a run measures. That covers the binary (src/, build.rs, Cargo.toml, Cargo.lock, rust-toolchain.toml, .cargo/config.toml), the manifests the scans run under (rules_templates/, conf/), the runner itself (bench/ apart from bench/tests/, and scripts/generate_rule_cwe_map.py, which a Juliet run executes), and the inputs it reads (data/benchmark_repos.json, data/benchmark_deps/, data/benchmark_environment.json, data/rule_cwe_map.json). The list is DIRTY_PATHS in bench/config.py. The refusal lists the changed files. Commit them before benchmarking. Changes elsewhere (docs/, tests/, an untracked note at the top level) and gitignored files do not count.

For an exploratory run on uncommitted work, pass --allow-dirty to the subcommand (python -m bench juliet --allow-dirty, python -m bench container-run -- realworld-run --allow-dirty). The run is then recorded as <sha>+dirty<hash>, where the hash covers the changes themselves, so it never shares a run id with the clean commit or with a different set of changes. Such a run is not a benchmark of any commit. In the container the binary reports the same label, and check-binary expects it. The run listings (realworld-runs, realworld, realworld-score) show the mark after the shortened SHA.

A Juliet run rebuilds its rule-CWE map (data/rule_cwe_map.json and the per-CWE manifests) before it scans. On a clean tree, if that rewrites a committed file, the run refuses to continue: commit the regenerated files first. With --allow-dirty the map is rebuilt before the run is labelled, so the label covers the files the run scans. container-run rebuilds it on the host before hashing the tree, for the same reason.

Untracked sources inside a corpus checkout are corpus-check’s concern, not this check’s.

Both runners refuse to scan a benchmark whose dependency set is missing or differs from its pin. On the host, fetch a set with python -m bench.deps fetch <name> (data/benchmark_deps/<name>.json; it needs Python 3 and the network, not dpkg), or all of them with playbooks/setup-benchmark-deps.yml.

python -m bench corpus-check checks the sets where this machine’s runs read them, which it is told with --mode (or AURORA_BENCH_MODE in .env; the default is host). In host mode a set missing from $SQC_BENCH_ROOT/deps fails the check. On a machine that runs benchmarks only through container-run, use --mode container. The host sets are then reported as not needed, and the check instead requires the benchmark image (--image, default container-run’s) to be present with the declared manifest_sha256.

The sections below describe scanning against the host’s own headers: local and shadow scans, which are never benchmark runs.

Third-Party Library Headers

aurora-lint uses -I include paths to resolve #include directives from third-party libraries. Without these, functions declared in external headers produce DCL31-C/DCL07-C false positives. The installed headers are therefore part of what a run measures – once the full list below is installed, the same binary on two machines differs by only a small number of keys (see Reproducing the Published Numbers).

That “small number of keys” figure describes drift after the full list is installed. Installing only part of it is a different, much larger failure mode, and an easy one to fall into since the per-project breakdown below is split into several blocks: a scan missing even one project’s packages (easy to do working through them top-to-bottom rather than installing the one-liner) has measurably produced several hundred extra findings on a several-thousand-finding run (a high-single-digit percentage of the total), and a project’s precision can sit measurably off a published number until every package for it is installed – with nothing in the tool’s own output signaling that more packages apply. corpus-check does not catch this either: it checks corpus pin drift and untracked/gitignored .c/.h files, not installed packages.

Install the full one-liner below rather than reasoning about which per-project subset your codebase needs:

sudo apt-get install -y libssl-dev libcjson-dev zlib1g-dev libcunit1-dev \
  libsqlite3-dev libmbedtls-dev libgnutls28-dev libc-ares-dev tcl-dev \
  libnl-3-dev libnl-genl-3-dev libdbus-1-dev libgcrypt20-dev libpcap-dev \
  libwolfssl-dev libtomcrypt-dev
# x86_64 hosts only: the package exists only on amd64
sudo apt-get install -y libc6-dev-i386

The benchmark runner scans curl, hostap, mosquitto, sqlite and valkey against these host headers, with aurora-lint as with cppcheck and clang-tidy. The benchmark node carries every one of them except libnl-genl-3-dev; scanning against the node’s headers plus that package reproduced its hostap findings key for key, so the extra package moves nothing. Which of them are present, not only their versions, changes findings: see What another machine reproduces in Reproducing the Published Numbers. That is expected, not a defect to engineer away. Official figures come from the benchmark node alone, on its own headers, and the maintainer records that node’s header packages; a run on any other machine is for A/B comparison during development and is not citable on its own (docs/adr/0004-postgres-is-the-single-source-of-truth.md). For a run that must not depend on the host, the pinned trees in Debian System Headers below are an opt-in.

Per-project breakdown, for reference (installing the one-liner above already covers all of these):

# Core dependencies (covers most projects)
sudo apt-get install -y \
  libssl-dev libcjson-dev zlib1g-dev

# mosquitto
sudo apt-get install -y libcunit1-dev libsqlite3-dev

# curl TLS backends and the c-ares resolver (curl's DCL31-C results also
# depend on whether /usr/include/sys exists; see
# reproducing-published-numbers)
sudo apt-get install -y libmbedtls-dev libgnutls28-dev libc-ares-dev

# sqlite test infrastructure
sudo apt-get install -y tcl-dev

# hostap
sudo apt-get install -y \
  libnl-3-dev libnl-genl-3-dev libdbus-1-dev \
  libgcrypt20-dev libpcap-dev libwolfssl-dev libtomcrypt-dev

# the benchmark node's multilib layout: puts sys/, bits/ and asm/ directly
# under /usr/include, which every host-header scan then reaches. amd64
# only; an arm64 host has no equivalent
sudo apt-get install -y libc6-dev-i386

Windows SDK and CRT Headers (ventoy)

ventoy is Win32 code, so a Linux or macOS node has none of its system headers. Without <windows.h> every Win32 call is an undeclared identifier: the scan reports several hundred DCL31-C findings that describe the node, not the code, and API00-C and INT30-C see different code too. So ventoy is scanned against a pinned tree of the Windows SDK and MSVC CRT headers, declared in data/benchmark_repos.json (header_tree) next to its commit pin: the exact xwin versions that produce it, and a manifest hash over the header files (definition in bench/header_tree.py). The tree was pinned after a scan against it gave findings identical to a scan against a native Visual Studio installation of the same SDK and CRT.

The headers are Microsoft’s. Fetching them accepts Microsoft’s license terms for the Windows SDK and the MSVC CRT, so the playbook only does it when you say so:

ansible-playbook playbooks/setup-benchmark-repos.yml -i "localhost," -c local \
  -e accept_microsoft_license=true --tags header-trees

(setup-benchmark-repos-macos.yml on macOS; the step is the same.) It installs the pinned xwin with cargo if needed, downloads about 120 MiB, keeps the headers (about 370 MB) under $SQC_BENCH_ROOT/header-trees/<id>/, and fails if their hash differs from the pin. Each machine fetches its own copy: never commit the tree or copy it between machines.

corpus-check reports the tree as MISSING or MISMATCH like a drifted commit, and realworld-run refuses to scan ventoy without a matching tree rather than produce header-less findings. Each scan records the tree it used (header_tree) in its .meta.json sidecar. cppcheck and clang-tidy are not given the tree: their ventoy baselines remain header-less.

To re-pin (a new SDK or CRT), change the fetch versions and the id, fetch the tree, print its hash with python3 -m bench.header_tree hash ventoy <tree-dir>, confirm the findings match a native-header scan, and only then declare the new hash.

Debian System Headers (curl, hostap, mosquitto, sqlite, valkey)

These five corpora pass -I /usr/include (and a few of its subdirectories), so their findings depend on the host: on which -dev packages it has installed, not only on their versions. A second machine’s run of one release differed from the benchmark node’s by about a hundred keys, every one traced to a header package (see What another machine reproduces in Reproducing the Published Numbers). By default the runner scans them against the host’s own headers all the same, and records that in each scan’s .meta.json sidecar (header_tree: {"id": "host", ...}): official figures are the benchmark node’s, on that node’s headers, and a local run is for A/B comparison on one machine, where both arms share the host (docs/adr/0004-postgres-is-the-single-source-of-truth.md).

Pinned Debian 12 /usr/include trees are an opt-in reproduction aid, for when a run must not depend on the host: reproducing the benchmark node’s environment on another machine, or comparing machines. python -m bench realworld-run --header-tree <id> scans the five corpora against one, under a run id of its own (-hdr-<id>), so its findings never land under the default run’s. --header-tree host is the default spelled out. The trees are declared in data/benchmark_repos.json under header_trees; the five corpora name the prefix they read from the host (host_headers), which a tree must replace to stand in for it:

  • debian12-declared-*: each corpus’s declared dependencies (OpenSSL, zlib, SQLite, Tcl and ICU for sqlite; CUnit and cJSON for mosquitto; mbed TLS, GnuTLS and c-ares for curl; libgcrypt, libpcap, libnl, D-Bus, wolfSSL and libtomcrypt for hostap) plus the C library and kernel headers.

  • debian12-benchmark-node-*: the header packages the benchmark node had when the v0.6.0 figures were measured. Scanning against it reproduced those runs key for key on another machine.

Each package is pinned by version and by the sha256 of its .deb at snapshot.debian.org. The runner rewrites every -I /usr/include... into the tree and puts the tree’s own include directories (the declared tree’s multiarch usr/include/x86_64-linux-gnu) first, in a compiler’s search order. Fetching needs Python 3 and the network, not dpkg, so a macOS, Fedora or FreeBSD node unpacks the same tree a Debian one does. Nothing fetches a tree by default; ask for it by id:

python3 -m bench.header_tree fetch debian12-declared-3af9978e

or through the playbook, with --tags header-trees -e debian_header_trees=<id>[,<id>] (no license flag). A tree takes about 30 MB under $SQC_BENCH_ROOT/header-trees/<id>/, plus a cache of its .deb files beside it. corpus-check does not require one. realworld-run --header-tree <id> refuses to scan against a tree that is missing or whose hash differs from the pin.

Read a comparison between two trees, or between a tree and the host, as the whole environment change, not one part of it: the two trees differ in their package sets and in how <sys/...> and <bits/...> are found (the declared tree searches its multiarch directory first; the benchmark-node tree has none on the search path and reaches those headers through the multilib links under /usr/include, as the node itself did).

A run with --compile-commands also passes the compile database’s own -I paths, which are not rewritten into the tree.

To re-pin, list the packages as PACKAGE=VERSION and run python3 -m bench.header_tree pin PKG=VER ..., which prints the debs entries (file, snapshot URL, sha256). Unpack them, print the tree’s hash with python3 -m bench.header_tree hash <old-id> <tree-dir>, and declare a new id and hash; a changed tree changes findings, so re-pin at a release boundary and measure the change.

Per-Project Include Paths

The real-world runner passes each codebase’s -I paths automatically, from the includes field of its entry in CODEBASES in bench/realworld_runner.py; that registry is the list. The corpora that read /usr/include are the five named under Third-Party Library Headers above.

Real-World Project Setup

Pinned Source Commits

All benchmark results are run against these exact commits – the full set of projects in bench/realworld_runner.py’s CODEBASES registry (which stores paths and per-tool flags, not the pins).

Note

The single source of truth for the pins is data/benchmark_repos.json. Both playbooks/setup-benchmark-repos.yml (provisioning) and python -m bench corpus-check (verification) read it. The table below is a human-readable mirror – if the two disagree, the JSON wins. Do not add a third copy.

Each checkout must sit at a detached HEAD on its pinned commit. A checkout left on a tracking branch is not pinned even while it happens to match: the next git pull moves it silently, and every finding then lands at (file, line) pairs the ground_truth oracle – keyed on project+commit+file+line+rule – was never adjudicated against, so it drops out of the precision/recall denominator without any error. This is not hypothetical; see Verifying the Pins below.

Project

Repository

Commit SHA

Checkout state

libcrc

https://github.com/lammertb/libcrc

7719e2112a9a960b1bba130d02bebdf58e8701f1

detached

sqlite

https://github.com/sqlite/sqlite.git

b1a73ba34d05b32007315e4065c6468cc638e3af

detached

mosquitto

https://github.com/eclipse-mosquitto/mosquitto

d3ee5c5ca62c0fa4983308c6fff558ee978e878c

detached

curl

https://github.com/curl/curl.git

3e198f75861cc2e12daf299689e145949dddd19b

detached

hostap

https://git.w1.fi/hostap.git

dcee60436390dd34731560657c4257c3b4c839a6

detached

lua

https://github.com/lua/lua.git

40b76de2d77e66b70a9d4bf989c3f5340919973f

detached

raylib

https://github.com/raysan5/raylib.git

962bbfc6bfbd7a5acd08e21314fcfa161003a589

detached

pureftpd

https://github.com/jedisct1/pure-ftpd.git

cc28bff52ca28e1d122a2142bf37f2dc578f4d3e

detached

sel4

https://github.com/seL4/seL4.git

1326364bc9135d9445d936ebc01e38a402c1f4c6

detached

mbedtls

https://github.com/Mbed-TLS/mbedtls.git

068ff080b369adfac81509f9b57b2afabaf82dc5

detached

valkey

https://github.com/valkey-io/valkey.git

7f1dffedff6de73058b2c2a389422b6ecd56c8fb

detached

ventoy

https://github.com/ventoy/Ventoy.git

7cbdc5cf69935bcf1f085ae67f40e70ea7e74bae

detached

Important

pureftpd’s checkout directory must be named pureftpd (no hyphen), not the upstream pure-ftpd – two path-parsing spots in bench/db.py (result-filename parsing, ground-truth path normalization) require the checkout dir basename to match the registry key exactly. The playbook already clones it to the right name.

Verifying the Pins

Provisioning pins the checkouts once; nothing kept them pinned. Run this before any real-world benchmark or precision claim:

python -m bench corpus-check          # exits 1 if anything is off
python -m bench corpus-check --json   # machine-readable

It reports one row per project, worst first:

Status

Meaning

MISSING

Checkout directory absent

NOT_GIT

Directory exists but is not a git checkout

PIN_ABSENT

Pinned commit not present locally (needs a fetch)

DRIFTED

HEAD is not the pinned commit

UNPINNED

HEAD matches the pin but sits on a branch, so the next git pull silently drifts it

OK

Detached at the pinned commit

It also flags three contamination cases independent of status:

  • tracked files modified, so the scanned source is not the pin;

  • untracked ``*.c``/``*.h`` files, which aurora-lint will scan and attribute to the pinned commit (untracked files aurora-lint ignores are counted separately and are harmless);

  • gitignored ``*.c``/``*.h`` files, which are invisible to git status but which aurora-lint scans anyway – it dispatches on file extension and never consults git. The case to watch is a build run inside a checkout: sqlite’s build generates a gitignored sqlite3.c amalgamation, which would silently add ~250k lines to every sqlite scan.

Tip

Keep the checkouts pristine. Build artifacts belong in a separate tree, not inside $SQC_BENCH_ROOT.

Warning

This check exists because the failure is silent and did happen. On one machine curl, hostap and sqlite had all drifted, while libcrc and lua sat on tracking branches matching their pins only by coincidence. A gate run against the drifted trees reported slightly more hostap and sqlite findings than the pinned snapshots give – close enough to look plausible, and not comparable to ground_truth at all. Fix drift with the git checkout --detach command the check prints for each row.

Clone and Pin

Preferred: run playbooks/setup-benchmark-repos.yml (see Benchmark Host Layout above) – it clones, pins, and verifies every pinned checkout in one pass, reading the pins from data/benchmark_repos.json.

Manual fallback: clone each repository in data/benchmark_repos.json under its name (the registry key, so pure-ftpd lands in pureftpd), detach at its version, and then verify:

mkdir -p $SQC_BENCH_ROOT   # default: ~/toolchain
cd $SQC_BENCH_ROOT
python3 -c 'import json, sys
for r in json.load(open(sys.argv[1]))["repos"]:
    print(r["repo"], r["name"], r["version"])' \
  $AURORA_LINT_SRC_ROOT/aurora-lint/data/benchmark_repos.json |
while read -r repo name sha; do
  git clone "$repo" "$name" && git -C "$name" checkout --detach "$sha"
done
git -C mbedtls submodule update --init -- framework
cd $AURORA_LINT_SRC_ROOT/aurora-lint && python -m bench corpus-check

A corpus with git submodules lists them under submodules (today only mbedtls’s framework), and each is checked out at the commit the pinned superproject records. The aurora-lint scan of library/ never reads framework/, but the library build does, so the compile database cannot be built without it. corpus-check fails a declared submodule that is missing, at another commit or modified, and a submodule in a pinned tree that is not declared. Do not build inside the valkey checkout: make leaves a gitignored src/release.h that corpus-check will (correctly) refuse.

Running Each Tool Manually

Note

The examples below use an illustrative ~/data/comparisons/<project> checkout path for ad hoc manual runs. If you provisioned codebases via playbooks/setup-benchmark-repos.yml, substitute $SQC_BENCH_ROOT/<project> (default ~/toolchain/<project>) instead – that’s the path bench/realworld_runner.py itself reads.

aurora-lint

# Build
cd $AURORA_LINT_SRC_ROOT/aurora-lint
cargo build --release

# Basic run
./target/release/aurora-lint /path/to/source/ --export results.json

# With cross-file context
./target/release/aurora-lint /path/to/source/ -d /path/to/source/ --export results.json

# When running from outside the aurora-lint repo, pass --manifest explicitly
./target/release/aurora-lint /path/to/source/ \
  --manifest $AURORA_LINT_SRC_ROOT/aurora-lint/rules_templates/rules-all.toml \
  --export results.json

cppcheck

cppcheck --enable=all --std=c11 --xml --xml-version=2 \
  --suppress=missingIncludeSystem \
  -I /path/to/include \
  /path/to/source/ \
  2> results.xml

Warning

  • cppcheck writes XML to stderr, not stdout

  • Never pass -I /usr/include – causes parse errors in system headers

  • --addon=cert does not exist: open-source cppcheck removed the CERT addon in 2.8 (CERT checking is Cppcheck Premium). --enable=all turns on cppcheck’s native checks, not CERT ones

clang-tidy

# Option A: With compile_commands.json (recommended)
cd /path/to/project
make clean && bear -- make -j$(nproc)
run-clang-tidy -checks='-*,cert-*,clang-analyzer-*' -p .

# Option B: Direct file scan (no build required)
find /path/to/source/ -name '*.c' | \
  xargs -P $(nproc) -I{} clang-tidy \
    -checks='-*,cert-*,clang-analyzer-*' \
    {} -- -std=c11 -I /path/to/include

Warning

  • bear requires make clean first – it only captures invocations during an actual build

  • Verify compile_commands.json is non-empty before running clang-tidy

Per-Project Commands

libcrc

# aurora-lint
$AURORA_LINT_SRC_ROOT/aurora-lint/target/release/aurora-lint ~/data/comparisons/libcrc \
  -d ~/data/comparisons/libcrc \
  --manifest $AURORA_LINT_SRC_ROOT/aurora-lint/rules_templates/rules-all.toml \
  --export ~/data/comparisons/results/aurora-lint/libcrc/results.json

# cppcheck
cppcheck --enable=all --std=c11 --xml --xml-version=2 \
  --suppress=missingIncludeSystem \
  -I ~/data/comparisons/libcrc/include \
  ~/data/comparisons/libcrc/src/ \
  2> ~/data/comparisons/results/cppcheck/libcrc/results.xml

# clang-tidy
cd ~/data/comparisons/libcrc && make clean && bear -- make
run-clang-tidy -checks='-*,cert-*,clang-analyzer-*' -p . \
  2>&1 | tee ~/data/comparisons/results/clang-tidy/libcrc/results.txt

sqlite

# aurora-lint
$AURORA_LINT_SRC_ROOT/aurora-lint/target/release/aurora-lint ~/data/comparisons/sqlite \
  -d ~/data/comparisons/sqlite \
  --manifest $AURORA_LINT_SRC_ROOT/aurora-lint/rules_templates/rules-all.toml \
  --export ~/data/comparisons/results/aurora-lint/sqlite/results.json

# cppcheck
cppcheck --enable=all --std=c11 --xml --xml-version=2 \
  --suppress=missingIncludeSystem \
  -I ~/data/comparisons/sqlite/src \
  ~/data/comparisons/sqlite/src/ \
  2> ~/data/comparisons/results/cppcheck/sqlite/results.xml

# clang-tidy
cd ~/data/comparisons/sqlite && ./configure && make clean && bear -- make -j$(nproc)
run-clang-tidy -checks='-*,cert-*,clang-analyzer-*' -p . \
  2>&1 | tee ~/data/comparisons/results/clang-tidy/sqlite/results.txt

mosquitto

# aurora-lint
$AURORA_LINT_SRC_ROOT/aurora-lint/target/release/aurora-lint ~/data/comparisons/mosquitto \
  -d ~/data/comparisons/mosquitto \
  --manifest $AURORA_LINT_SRC_ROOT/aurora-lint/rules_templates/rules-all.toml \
  --export ~/data/comparisons/results/aurora-lint/mosquitto/results.json

# cppcheck
cppcheck --enable=all --std=c11 --xml --xml-version=2 \
  --suppress=missingIncludeSystem \
  -I ~/data/comparisons/mosquitto/include \
  -I ~/data/comparisons/mosquitto/common \
  -i ~/data/comparisons/mosquitto/deps \
  ~/data/comparisons/mosquitto/lib/ ~/data/comparisons/mosquitto/src/ \
  2> ~/data/comparisons/results/cppcheck/mosquitto/results.xml

# clang-tidy
cmake -S ~/data/comparisons/mosquitto -B ~/data/comparisons/mosquitto/build \
  -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DWITH_TLS=OFF -DWITH_WEBSOCKETS=OFF -DWITH_TESTS=OFF
cmake --build ~/data/comparisons/mosquitto/build --clean-first -j$(nproc)
run-clang-tidy -checks='-*,cert-*,clang-analyzer-*' \
  -p ~/data/comparisons/mosquitto/build \
  2>&1 | tee ~/data/comparisons/results/clang-tidy/mosquitto/results.txt

curl

# aurora-lint
$AURORA_LINT_SRC_ROOT/aurora-lint/target/release/aurora-lint ~/data/comparisons/curl \
  -d ~/data/comparisons/curl \
  --manifest $AURORA_LINT_SRC_ROOT/aurora-lint/rules_templates/rules-all.toml \
  --export ~/data/comparisons/results/aurora-lint/curl/results.json

# cppcheck
cppcheck --enable=all --std=c11 --xml --xml-version=2 \
  --suppress=missingIncludeSystem \
  -I ~/data/comparisons/curl/include -I ~/data/comparisons/curl/lib \
  ~/data/comparisons/curl/lib/ ~/data/comparisons/curl/src/ \
  2> ~/data/comparisons/results/cppcheck/curl/results.xml

# clang-tidy
cmake -S ~/data/comparisons/curl -B ~/data/comparisons/curl/build \
  -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DBUILD_SHARED_LIBS=OFF \
  -DCURL_USE_OPENSSL=OFF -DCURL_DISABLE_LDAP=ON \
  -DUSE_LIBIDN2=OFF -DUSE_NGHTTP2=OFF -DCURL_ZSTD=OFF
cmake --build ~/data/comparisons/curl/build --clean-first -j$(nproc)
run-clang-tidy -checks='-*,cert-*,clang-analyzer-*' \
  -p ~/data/comparisons/curl/build \
  2>&1 | tee ~/data/comparisons/results/clang-tidy/curl/results.txt

hostap

# aurora-lint
$AURORA_LINT_SRC_ROOT/aurora-lint/target/release/aurora-lint ~/data/comparisons/hostap \
  -d ~/data/comparisons/hostap/src \
  -d ~/data/comparisons/hostap/wpa_supplicant \
  --manifest $AURORA_LINT_SRC_ROOT/aurora-lint/rules_templates/rules-all.toml \
  --export ~/data/comparisons/results/aurora-lint/hostap/results.json

# cppcheck
cppcheck --enable=all --std=c11 --xml --xml-version=2 \
  --suppress=missingIncludeSystem \
  -I ~/data/comparisons/hostap/src \
  -I ~/data/comparisons/hostap/src/utils \
  -I ~/data/comparisons/hostap/src/common \
  ~/data/comparisons/hostap/src/ ~/data/comparisons/hostap/wpa_supplicant/ \
  2> ~/data/comparisons/results/cppcheck/hostap/results.xml

# clang-tidy (requires .config for wpa_supplicant)
cd ~/data/comparisons/hostap/wpa_supplicant
cat > .config <<'EOF'
CONFIG_DRIVER_NL80211=y
CONFIG_LIBNL32=y
CONFIG_IEEE8021X_EAPOL=y
CONFIG_EAP_MD5=y
EOF
make clean 2>/dev/null; bear -- make -j$(nproc) wpa_supplicant 2>/dev/null || true
cd ~/data/comparisons/hostap
python3 gen_compile_commands.py -o compile_commands.json wpa_supplicant/
run-clang-tidy -checks='-*,cert-*,clang-analyzer-*' -p . \
  2>&1 | tee ~/data/comparisons/results/clang-tidy/hostap/results.txt

Verifying Results

PROJECT=libcrc
RESULTS=~/data/comparisons/results

echo "=== aurora-lint ==="
python3 -c "
import json, collections
data = json.load(open('$RESULTS/aurora-lint/$PROJECT/results.json'))
rules = collections.Counter(v['rule_id'] for v in data)
print(f'Total: {len(data)} findings, {len(rules)} rules')
for r, c in rules.most_common(10): print(f'  {r}: {c}')
"

echo "=== cppcheck ==="
python3 -c "
import xml.etree.ElementTree as ET, collections
errors = ET.parse('$RESULTS/cppcheck/$PROJECT/results.xml').getroot().findall('.//error')
ids = collections.Counter(e.get('id') for e in errors)
print(f'Total: {len(errors)} findings')
for i, c in ids.most_common(10): print(f'  {i}: {c}')
"

echo "=== clang-tidy ==="
grep -c "warning:\|error:" $RESULTS/clang-tidy/$PROJECT/results.txt

Known Pitfalls

Pitfall

Symptom

Fix

aurora-lint manifest not found

Failed to read manifest file

Pass --manifest with absolute path

cppcheck --addon=cert missing

Did not find addon cert.py

Remove it; the addon is gone from open-source cppcheck (2.8 on), so there are no CERT checks to enable

cppcheck -I /usr/include

syntaxError in stdlib.h

Never pass system include paths to cppcheck

Empty compile_commands.json

clang-tidy shows no diagnostics

Run make clean before bear -- make

Output Format Reference

cppcheck XML (v2):

<error id="bufferAccessOutOfBounds" severity="error"
       msg="Array 'arr[10]' accessed at index 10..." cwe="788">
  <location file="example.c" line="5" column="5"/>
</error>

clang-tidy text:

example.c:5:5: warning: ... [cert-arr30-c]

aurora-lint JSON:

[{"rule_id": "ARR30-C", "severity": "High", "file": "example.c", "line": 5, "message": "..."}]

aurora-lint SARIF: SARIF 2.1.0 compatible output with ruleId matching CERT C rule IDs.

Infer JSON (infer-out/report.json):

[{"bug_type": "NULLPTR_DEREFERENCE", "procedure": "func_bad",
  "file": "test.c", "line": 26, "severity": "ERROR"}]

Frama-C EVA (stderr, no structured output):

[eva:alarm] test.c:26: Warning:
  out of bounds read. assert \valid_read(&ptr->field);
assertion 'Eva,mem_access' got final status invalid.

Distributed Benchmarking

Multi-host runs belong to the maintainer’s benchmarking_db infrastructure, not to this repo.

Exporting competitor results for ingest

bench/competitors.py writes one JSON blob per run into data/competitor_results/, and those blobs are committed — they are the archival form and should not be edited or deleted. They are not, however, ingestible: the shape is nested and the per-CWE map is keyed by CWE id, so a loader would have to know that module’s internal layout to read it.

python -m bench competitor-export flattens every run JSON in that directory into three CSVs beside them:

competitor_runs.csv         one row per run       (tool, version, totals)
competitor_cwe_results.csv  one row per run × CWE (the measurements)
competitor_cwe_errors.csv   one row per error     (usually empty)

run_key joins them and is the JSON’s own basename (e.g. framac_20260403_222053), so it is unique, stable across re-exports, and traceable back to the file it came from.

Run the export after every competitor benchmark and commit the result. The benchmark host ingests these CSVs into sqc_bench; a table missing its newest run looks exactly like a run that never happened. python -m bench competitor-export --check exits nonzero if the CSVs are stale, and writes nothing.

This repo stays Postgres-blind — no DSN, no connection code (see CLAUDE.md). It emits a flat, diffable table; benchmarking_db owns the ingest, so nothing here changes when the target schema does. The CSVs are a transport, not a source of truth: Postgres remains the only place an official number comes from, and re-exporting is idempotent and lossless with respect to the JSON.

Note

The run JSONs record no hostname. run_hosts.json attributes each run, and the CSV carries it as hostname + hostname_source. Wall clock is hardware-dependent, so compare durations only between runs on the same host.