Benchmark Setup =============== This section covers installing comparison tools, setting up the Juliet test suite, and configuring real-world codebases for benchmarking. Benchmark Host Layout ---------------------- Every machine that runs benchmarks needs one directory -- ``$SQC_BENCH_ROOT`` (default ``~/toolchain``) -- holding the Juliet suite and every real-world codebase checkout. ``bench/config.py`` and ``bench/realworld_runner.py`` both read this from the ``SQC_BENCH_ROOT`` environment variable, or from a ``.env`` file at the repo root (copy ``.env.example`` -- it's gitignored, so each machine sets its own). To provision a fresh node, run the Ansible playbook (clones and pins every real-world codebases; does not fetch Juliet -- see below): .. code-block:: bash ansible-playbook playbooks/setup-benchmark-repos.yml -i "localhost," -c local # Or a different root: ansible-playbook playbooks/setup-benchmark-repos.yml -i "localhost," -c local \ -e bench_root=/data/toolchain Keep ``bench_root`` and ``SQC_BENCH_ROOT``/``.env`` pointed at the same directory -- the playbook provisions the checkouts the code actually reads. A mismatch is easy to miss: if ``SQC_BENCH_ROOT`` is unset and the corpora do not live in ``~/toolchain``, ``BENCH_ROOT`` silently resolves to a directory that does not exist. ``python -m bench corpus-check`` reports the resolved root and says so outright. A second, separate root names where the *source* clones are: ``$AURORA_LINT_SRC_ROOT`` is the parent directory of this checkout (``$AURORA_LINT_SRC_ROOT/aurora-lint``), and, on the maintainer's machine, of the sibling ``benchmarking_db`` and ``sqc_paper`` clones too. It defaults to the checkout's own parent, so nothing here requires it; the copy-pasteable commands below use it so they hold on any machine regardless of where the clone lives (``~/data`` is the conventional value, and the ``.env.example`` default): .. code-block:: bash export AURORA_LINT_SRC_ROOT=~/data # parent of your aurora-lint clone Installing Comparison Tools --------------------------- cppcheck ~~~~~~~~ .. code-block:: bash sudo apt update sudo apt install -y cppcheck # Verify cppcheck --version # Expected: Cppcheck 2.x.x For the latest release, build from source: .. code-block:: bash sudo apt install -y cmake libpcre3-dev git clone https://github.com/danmar/cppcheck.git cd cppcheck cmake -DCMAKE_BUILD_TYPE=Release -DUSE_MATCHCOMPILER=ON . make -j$(nproc) sudo make install clang-tidy ~~~~~~~~~~ **Pin LLVM 21 — do not take the distro default.** Ubuntu 24.04 ships clang-tidy 18.1.3 and tops out at 20, while the published comparison in ``docs/tool-comparison.rst`` is LLVM 21.1. Taking the distro package makes a freshly-provisioned node measure an *older* competitor than the table it will be compared against, and clang-tidy is the tool beating aurora-lint on the Juliet overlap in that comparison — so that regression would flatter aurora-lint by understating a rival. ``playbooks/install-static-analyzers.yml`` does this automatically; by hand: .. code-block:: bash sudo install -d -m 0755 /etc/apt/keyrings wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key \ | sudo tee /etc/apt/keyrings/apt.llvm.org.asc > /dev/null echo "deb [signed-by=/etc/apt/keyrings/apt.llvm.org.asc] \ http://apt.llvm.org/noble/ llvm-toolchain-noble-21 main" \ | sudo tee /etc/apt/sources.list.d/llvm-21.list sudo apt update && sudo apt install -y clang-tidy-21 sudo update-alternatives --install /usr/bin/clang-tidy clang-tidy \ /usr/bin/clang-tidy-21 100 # Verify clang-tidy --version # Expected: Ubuntu LLVM version 21.1.x The distro package is fine for anything that is not a published comparison: .. code-block:: bash sudo apt install -y clang clang-tidy bear (for compile_commands.json) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Required for clang-tidy on projects with build systems: .. code-block:: bash sudo apt install -y bear Facebook Infer ~~~~~~~~~~~~~~ Infer v1.2.0 is installed from prebuilt Linux binaries. The playbook at ``playbooks/install-static-analyzers.yml`` automates this: .. code-block:: bash ansible-playbook playbooks/install-static-analyzers.yml \ -i "localhost," -c local --ask-become-pass Manual install: .. code-block:: bash # libtinfo5 required (Infer links against libtinfo.so.5) sudo apt install -y libtinfo5 || \ sudo ln -s /usr/lib/x86_64-linux-gnu/libtinfo.so.6 \ /usr/lib/x86_64-linux-gnu/libtinfo.so.5 VERSION=1.2.0 curl -sSL "https://github.com/facebook/infer/releases/download/v$VERSION/infer-linux-x86_64-v$VERSION.tar.xz" \ | sudo tar -C /opt -xJ sudo ln -s "/opt/infer-linux-x86_64-v$VERSION/bin/infer" /usr/local/bin/infer # Verify infer --version # Expected: Infer version v1.2.0 Frama-C ~~~~~~~ Frama-C is installed via opam (the Debian 12 apt package is version 25/Manganese, too old for benchmarking). The playbook handles this automatically. Manual install: .. code-block:: bash sudo apt install -y opam gcc g++ make m4 pkg-config autoconf \ libgmp-dev zlib1g-dev libgtk-3-dev libgtksourceview-3.0-dev graphviz opam init --bare --no-setup --disable-sandboxing opam switch create default 4.14.2 eval $(opam env) opam install -y frama-c # Verify (requires eval $(opam env) in each shell session) frama-c -version # Expected: 33.0 (Arsenic) as of 2026-09-04; 32.0 (Germanium) is what the # competitor Juliet runs in data/competitor_results/ were taken with .. note:: Add ``eval $(opam env)`` to your shell profile (``~/.bashrc``) to make ``frama-c`` available without manual activation. The benchmark runner does not need it: ``bench/realworld_runner.py`` wraps every Frama-C invocation in ``eval $(opam env)`` itself, because it is not a login shell. .. warning:: Frama-C renamed the compile-database option between the two versions above — ``-json-compilation-database`` through 32.0, ``-compilation-db`` from 33.0. The runner probes ``frama-c -kernel-h`` and picks whichever the installed binary accepts, so both work; a hand-written command line copied from an older doc will abort with "option is unknown". Build-based tools need a compile database ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Infer and Frama-C cannot be pointed at a source tree the way aurora-lint and cppcheck can — both need a real preprocess. On the real-world suite they are driven from each checkout's ``compile_commands.json``, so ``playbooks/setup-compile-commands.yml`` must have run for a codebase before either tool can scan it. ``python -m bench realworld-run --tool infer`` fails fast with the exact command to fix it if the database is missing. Expect a *partial* result from both, and check the ``coverage`` block each run writes: - **Infer** captures whatever preprocesses. ``setup-compile-commands.yml`` restores each checkout to pristine after building it, which deletes the build's generated headers; the playbook now stashes the ones the compile database still references (pure-ftpd's ``config.h``, sqlite's ``sqlite_cfg.h``, valkey's ``jemalloc/jemalloc.h``, libcrc's ``tab/gentab*.inc``) under ``/.cc_build/generated/`` and points the database there, so every in-scope translation unit preprocesses. A ``capture_failures`` list that is not empty means a database written before that existed — re-run the playbook, which detects and rebuilds it. - **Frama-C** is partial by construction, because EVA analyzes one entry point at a time and a real codebase has no single one. Read ``docs/design/framac-realworld.md`` before quoting any Frama-C real-world number: a finding count from it is a floor, and recall is not expressible from it at all. Reference Shadow Set (optional) ------------------------------- The reference shadow set is a list of pinned C codebases, disjoint from the real-world benchmarks, that is never labeled or tuned against. Its only use is ``python -m bench reference-ab``: what a change moves on code nobody tuned against (``docs/design/reference-corpus-ab.md``). Nothing else needs it, so it is a one-time setup for anyone who wants to run that A/B: .. code-block:: bash ansible-playbook playbooks/setup-shadow-corpus.yml -i "localhost," -c local ansible-playbook playbooks/setup-shadow-corpus.yml -i "localhost," -c local \ -e shadow_tier=all # every tier, the large one included It downloads shallow checkouts, each at its pin in ``data/reference_corpus.json``, under ``$SQC_BENCH_ROOT/shadow_corpus/``. The tiers are cumulative: ``quick``, ``standard`` (the default, about 1 GB), and ``large`` or ``all`` (about 5 GB). The playbook checks first that about twice the tier's size is free, unless every checkout is already at its pin. The checkouts stay where they are: ``reference-ab`` re-checks and cleans each one at its pin before scanning it, so nothing needs fetching again until the pins change. ``python -m bench reference-fetch --tier T`` does the same fetch without Ansible. The large tier's scans are slow and memory-heavy. Expect hours for an A/B over it, and a peak near 12 GB resident at 8 threads; on a machine with under 16 GiB of RAM, ``reference-ab`` halves its default ``--jobs`` for it. Juliet Test Suite Setup ----------------------- The setup playbook does not fetch Juliet; it only creates the parent directory and reports whether the suite is already in place. The `NIST Juliet Test Suite v1.3 `_ page links a single zip that downloads directly, with no click-through. NIST does not promise that URL stays put, so if the ``curl`` below fails, start from that page. The archive extracts to ``C/``, which is renamed to the directory name ``bench/config.py`` expects: .. code-block:: bash sudo apt install -y curl unzip mkdir -p $SQC_BENCH_ROOT/benchmarks # default: ~/toolchain/benchmarks cd $SQC_BENCH_ROOT/benchmarks curl -LO https://samate.nist.gov/SARD/downloads/test-suites/2017-10-01-juliet-test-suite-for-c-cplusplus-v1-3.zip unzip -q 2017-10-01-juliet-test-suite-for-c-cplusplus-v1-3.zip mv C juliet-test-suite-c # Expected structure: # juliet-test-suite-c/testcases/CWE*/ (118 CWE directories) # juliet-test-suite-c/testcasesupport/ (shared helper functions) The per-CWE manifests that fast mode runs (``rules_templates/cwe/*.toml``) are generated by ``scripts/generate_rule_cwe_map.py`` and are not committed, so that directory is nearly empty in a fresh clone. ``python -m bench juliet`` runs the script for you before every run, which also refreshes ``data/rule_cwe_map.json``. The Benchmark Environment ------------------------- A benchmark run, real-world or Juliet, is pinned by five things: the corpus commit, the aurora-lint commit, the oracle commit, the settings hash and the environment it ran in (``docs/adr/0018``). The environment is a container image, ``container/benchmark.Dockerfile``, holding a Debian 12 (bookworm) snapshot, every benchmark's dependency set (the system headers it is scanned against, each in its own tree), the corpus build tools and the comparison tools at exact versions. The corpus checkouts and this repository are mounted into it at run time, and aurora-lint is built inside it. Build the image with `podman `_ (rootless is fine). The build is amd64 on every host, an arm64 Mac included: .. code-block:: bash podman build --platform linux/amd64 --target tools \ -f container/benchmark.Dockerfile -t aurora-bench-tools:dev . podman build --platform linux/amd64 \ -f container/benchmark.Dockerfile -t aurora-bench:shared . That image leaves out the Win32 corpus's set, the Windows SDK and MSVC CRT headers, which are Microsoft's: building the layer that adds them accepts Microsoft's licence terms for your machine, and the result must never be copied to another one: .. code-block:: bash podman build --platform linux/amd64 -f container/licence.Dockerfile \ --build-arg BENCH_IMAGE=aurora-bench:shared \ --build-arg ACCEPT_MICROSOFT_LICENSE=true -t aurora-bench:dev . The image records its environment manifest in ``/etc/aurora-bench/environment.json``; the SHA-256 of that manifest is the pin. ``data/benchmark_environment.json`` declares the pin this commit expects, so you can check your own build against it: ``python -m bench container-run`` prints the pin of the image it runs. ``data/benchmark_environment.json`` also names the shared, licence-free image (``shared_image``) and its ``tools`` stage (``tools_image``) by digest, with each one's pin. Where they are pulled from is up to your deployment; ``python -m bench.environment declared tools_image`` prints a field and checks the file. When the image changes ~~~~~~~~~~~~~~~~~~~~~~ Any change to the Dockerfiles, a dependency set or the manifest generator is a new environment. In the same commit as the change: 1. Rebuild the shared image and its ``tools`` stage (``podman build --target tools``), and the licence layer on top. 2. Record each pin: ``python3 -m bench.environment hash /etc/aurora-bench/environment.json`` in the licensed image (``manifest_sha256``) and in the shared one (``shared_manifest_sha256``), and the ``tools_stage`` the shared manifest records. 3. Push the shared image and the tools image wherever your machines pull from, and record each digest the push reports (``--digestfile``) as ``shared_image`` and ``tools_image``. 4. Check the file with ``python -m bench.environment declared manifest_sha256``, and A/B the suite before and after, as for any change to the environment. Then build each corpus's compile database (its declared default build, run inside a throwaway container from the ``tools`` stage) and run the benchmarks in the image: .. code-block:: bash python -m bench container-build-db --codebase curl,lua python -m bench container-run -- realworld-run --tool sqc --codebase curl,lua python -m bench container-run -- juliet A run in the declared environment gets the plain run id. The same run in another image is suffixed ``-env``; outside any image (``bench realworld-run`` or ``bench juliet`` on the host) ``-hostenv``; against another header tree ``-hdr-``, or ``-hdr-host`` for the host's own headers. Only the plain id is a benchmark run. ``container-run`` builds aurora-lint inside the image, into a podman volume named ``aurora-bench-target--``: one per environment pin *and* per checkout path. Every checkout is mounted at the same ``/work`` and cargo judges a build fresh by source mtimes, so a target shared between checkouts could hand one checkout another's binary. A separate volume per checkout prevents that. The cost is that the first run from a new checkout path is a cold release build (a few minutes), and each throwaway A/B worktree gets a fresh volume holding a full release target (most of a gigabyte). Deleting a worktree does not delete its volume, and rebuilding the image leaves the old layers behind as dangling images, so storage grows with every A/B until it is pruned: .. code-block:: bash python -m bench container-prune # list what would go python -m bench container-prune --yes # remove it A volume is removed only when its checkout hash matches none of this repository's live checkouts (the checkout itself and its ``git worktree list``), hashed as ``container-run`` names it, plus the older per-pin-only ``aurora-bench-target-`` volumes, which nothing uses any more. A worktree whose path is missing counts as deleted unless it is locked (``git worktree lock``), which is how to keep the volume of a worktree on a disk that is not mounted. **Every other clone counts as deleted.** Name each clone whose volumes should stay with ``--keep DIR`` (repeatable); its worktrees are kept with it. A ``--keep`` path that is not a git checkout stops the prune rather than keeping nothing. The images listed and removed are every dangling image of this user, not only those an ``aurora-bench`` build left behind, since podman records nothing that tells them apart. Each is removed by its id, without ``-f``, so an image a container was made from is kept. Tagged images and the ``aurora-bench-cargo-registry`` volume are never touched. The prune refuses while a benchmark container (a ``container-run`` or ``container-build-db``) or any container mounting a target volume is running, and podman itself refuses a volume some container still uses. It exits 0 when everything listed was removed (or, without ``--yes``, listed), 1 when podman kept something, which it names, and 2 when it refused: podman or ``git worktree list`` could not list what it needs, a ``--keep`` path is no checkout, or a container is running. The build also records the checkout's commit in the binary's ``--version``, and before scanning ``container-run`` checks it (``python -m bench.container check-binary``). If the binary does not report exactly the commit the run will be labelled with, the run stops with exit status 2 and a line ending ``refusing to scan``, and nothing is scanned or recorded. The line names the commit the binary reports, if any. A build given the commit always recompiles for it, so this means the volume holds a binary the build did not replace. Remove that checkout's volume and run again. If the refusal persists, the build is not embedding the commit it is given; report it rather than working around it, because a run past this check would measure an unknown binary under this commit's run id. A run is labelled with the checkout's commit, so it must measure that commit. ``realworld-run``, ``juliet`` and ``container-run`` refuse to start when the checkout has uncommitted changes, staged, unstaged or untracked, to anything that decides what a run measures. That covers the binary (``src/``, ``build.rs``, ``Cargo.toml``, ``Cargo.lock``, ``rust-toolchain.toml``, ``.cargo/config.toml``), the manifests the scans run under (``rules_templates/``, ``conf/``), the runner itself (``bench/`` apart from ``bench/tests/``, and ``scripts/generate_rule_cwe_map.py``, which a Juliet run executes), and the inputs it reads (``data/benchmark_repos.json``, ``data/benchmark_deps/``, ``data/benchmark_environment.json``, ``data/rule_cwe_map.json``). The list is ``DIRTY_PATHS`` in ``bench/config.py``. The refusal lists the changed files. Commit them before benchmarking. Changes elsewhere (``docs/``, ``tests/``, an untracked note at the top level) and gitignored files do not count. For an exploratory run on uncommitted work, pass ``--allow-dirty`` to the subcommand (``python -m bench juliet --allow-dirty``, ``python -m bench container-run -- realworld-run --allow-dirty``). The run is then recorded as ``+dirty``, where the hash covers the changes themselves, so it never shares a run id with the clean commit or with a different set of changes. Such a run is not a benchmark of any commit. In the container the binary reports the same label, and ``check-binary`` expects it. The run listings (``realworld-runs``, ``realworld``, ``realworld-score``) show the mark after the shortened SHA. A Juliet run rebuilds its rule-CWE map (``data/rule_cwe_map.json`` and the per-CWE manifests) before it scans. On a clean tree, if that rewrites a committed file, the run refuses to continue: commit the regenerated files first. With ``--allow-dirty`` the map is rebuilt before the run is labelled, so the label covers the files the run scans. ``container-run`` rebuilds it on the host before hashing the tree, for the same reason. Untracked sources inside a *corpus* checkout are ``corpus-check``'s concern, not this check's. Both runners refuse to scan a benchmark whose dependency set is missing or differs from its pin. On the host, fetch a set with ``python -m bench.deps fetch `` (``data/benchmark_deps/.json``; it needs Python 3 and the network, not ``dpkg``), or all of them with ``playbooks/setup-benchmark-deps.yml``. ``python -m bench corpus-check`` checks the sets where this machine's runs read them, which it is told with ``--mode`` (or ``AURORA_BENCH_MODE`` in ``.env``; the default is ``host``). In ``host`` mode a set missing from ``$SQC_BENCH_ROOT/deps`` fails the check. On a machine that runs benchmarks only through ``container-run``, use ``--mode container``. The host sets are then reported as not needed, and the check instead requires the benchmark image (``--image``, default ``container-run``'s) to be present with the declared ``manifest_sha256``. The sections below describe scanning against the host's own headers: local and shadow scans, which are never benchmark runs. Third-Party Library Headers --------------------------- aurora-lint uses ``-I`` include paths to resolve ``#include`` directives from third-party libraries. Without these, functions declared in external headers produce DCL31-C/DCL07-C false positives. The installed headers are therefore part of what a run measures -- once the full list below is installed, the same binary on two machines differs by only a small number of keys (see :doc:`reproducing-published-numbers`). **That "small number of keys" figure describes drift after the full list is installed.** Installing only part of it is a different, much larger failure mode, and an easy one to fall into since the per-project breakdown below is split into several blocks: a scan missing even one project's packages (easy to do working through them top-to-bottom rather than installing the one-liner) has measurably produced several hundred extra findings on a several-thousand-finding run (a high-single-digit percentage of the total), and a project's precision can sit measurably off a published number until every package for it is installed -- with nothing in the tool's own output signaling that more packages apply. ``corpus-check`` does not catch this either: it checks corpus pin drift and untracked/gitignored ``.c``/``.h`` files, not installed packages. Install the full one-liner below rather than reasoning about which per-project subset your codebase needs: .. code-block:: bash sudo apt-get install -y libssl-dev libcjson-dev zlib1g-dev libcunit1-dev \ libsqlite3-dev libmbedtls-dev libgnutls28-dev libc-ares-dev tcl-dev \ libnl-3-dev libnl-genl-3-dev libdbus-1-dev libgcrypt20-dev libpcap-dev \ libwolfssl-dev libtomcrypt-dev # x86_64 hosts only: the package exists only on amd64 sudo apt-get install -y libc6-dev-i386 The benchmark runner scans curl, hostap, mosquitto, sqlite and valkey against these host headers, with aurora-lint as with cppcheck and clang-tidy. The benchmark node carries every one of them except ``libnl-genl-3-dev``; scanning against the node's headers plus that package reproduced its hostap findings key for key, so the extra package moves nothing. Which of them are present, not only their versions, changes findings: see *What another machine reproduces* in :doc:`reproducing-published-numbers`. That is expected, not a defect to engineer away. Official figures come from the benchmark node alone, on its own headers, and the maintainer records that node's header packages; a run on any other machine is for A/B comparison during development and is not citable on its own (``docs/adr/0004-postgres-is-the-single-source-of-truth.md``). For a run that must not depend on the host, the pinned trees in *Debian System Headers* below are an opt-in. Per-project breakdown, for reference (installing the one-liner above already covers all of these): .. code-block:: bash # Core dependencies (covers most projects) sudo apt-get install -y \ libssl-dev libcjson-dev zlib1g-dev # mosquitto sudo apt-get install -y libcunit1-dev libsqlite3-dev # curl TLS backends and the c-ares resolver (curl's DCL31-C results also # depend on whether /usr/include/sys exists; see # reproducing-published-numbers) sudo apt-get install -y libmbedtls-dev libgnutls28-dev libc-ares-dev # sqlite test infrastructure sudo apt-get install -y tcl-dev # hostap sudo apt-get install -y \ libnl-3-dev libnl-genl-3-dev libdbus-1-dev \ libgcrypt20-dev libpcap-dev libwolfssl-dev libtomcrypt-dev # the benchmark node's multilib layout: puts sys/, bits/ and asm/ directly # under /usr/include, which every host-header scan then reaches. amd64 # only; an arm64 host has no equivalent sudo apt-get install -y libc6-dev-i386 Windows SDK and CRT Headers (ventoy) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ``ventoy`` is Win32 code, so a Linux or macOS node has none of its system headers. Without ```` every Win32 call is an undeclared identifier: the scan reports several hundred DCL31-C findings that describe the node, not the code, and API00-C and INT30-C see different code too. So ventoy is scanned against a pinned tree of the Windows SDK and MSVC CRT headers, declared in ``data/benchmark_repos.json`` (``header_tree``) next to its commit pin: the exact `xwin `_ versions that produce it, and a manifest hash over the header files (definition in ``bench/header_tree.py``). The tree was pinned after a scan against it gave findings identical to a scan against a native Visual Studio installation of the same SDK and CRT. The headers are Microsoft's. Fetching them accepts Microsoft's license terms for the Windows SDK and the MSVC CRT, so the playbook only does it when you say so: .. code-block:: bash ansible-playbook playbooks/setup-benchmark-repos.yml -i "localhost," -c local \ -e accept_microsoft_license=true --tags header-trees (``setup-benchmark-repos-macos.yml`` on macOS; the step is the same.) It installs the pinned xwin with ``cargo`` if needed, downloads about 120 MiB, keeps the headers (about 370 MB) under ``$SQC_BENCH_ROOT/header-trees//``, and fails if their hash differs from the pin. Each machine fetches its own copy: never commit the tree or copy it between machines. ``corpus-check`` reports the tree as ``MISSING`` or ``MISMATCH`` like a drifted commit, and ``realworld-run`` refuses to scan ventoy without a matching tree rather than produce header-less findings. Each scan records the tree it used (``header_tree``) in its ``.meta.json`` sidecar. cppcheck and clang-tidy are not given the tree: their ventoy baselines remain header-less. To re-pin (a new SDK or CRT), change the ``fetch`` versions and the ``id``, fetch the tree, print its hash with ``python3 -m bench.header_tree hash ventoy ``, confirm the findings match a native-header scan, and only then declare the new hash. Debian System Headers (curl, hostap, mosquitto, sqlite, valkey) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ These five corpora pass ``-I /usr/include`` (and a few of its subdirectories), so their findings depend on the host: on which ``-dev`` packages it has installed, not only on their versions. A second machine's run of one release differed from the benchmark node's by about a hundred keys, every one traced to a header package (see *What another machine reproduces* in :doc:`reproducing-published-numbers`). By default the runner scans them against the host's own headers all the same, and records that in each scan's ``.meta.json`` sidecar (``header_tree``: ``{"id": "host", ...}``): official figures are the benchmark node's, on that node's headers, and a local run is for A/B comparison on one machine, where both arms share the host (``docs/adr/0004-postgres-is-the-single-source-of-truth.md``). Pinned Debian 12 ``/usr/include`` trees are an opt-in reproduction aid, for when a run must not depend on the host: reproducing the benchmark node's environment on another machine, or comparing machines. ``python -m bench realworld-run --header-tree `` scans the five corpora against one, under a run id of its own (``-hdr-``), so its findings never land under the default run's. ``--header-tree host`` is the default spelled out. The trees are declared in ``data/benchmark_repos.json`` under ``header_trees``; the five corpora name the prefix they read from the host (``host_headers``), which a tree must replace to stand in for it: * ``debian12-declared-*``: each corpus's declared dependencies (OpenSSL, zlib, SQLite, Tcl and ICU for sqlite; CUnit and cJSON for mosquitto; mbed TLS, GnuTLS and c-ares for curl; libgcrypt, libpcap, libnl, D-Bus, wolfSSL and libtomcrypt for hostap) plus the C library and kernel headers. * ``debian12-benchmark-node-*``: the header packages the benchmark node had when the ``v0.6.0`` figures were measured. Scanning against it reproduced those runs key for key on another machine. Each package is pinned by version and by the sha256 of its ``.deb`` at ``snapshot.debian.org``. The runner rewrites every ``-I /usr/include...`` into the tree and puts the tree's own include directories (the declared tree's multiarch ``usr/include/x86_64-linux-gnu``) first, in a compiler's search order. Fetching needs Python 3 and the network, not ``dpkg``, so a macOS, Fedora or FreeBSD node unpacks the same tree a Debian one does. Nothing fetches a tree by default; ask for it by id: .. code-block:: bash python3 -m bench.header_tree fetch debian12-declared-3af9978e or through the playbook, with ``--tags header-trees -e debian_header_trees=[,]`` (no license flag). A tree takes about 30 MB under ``$SQC_BENCH_ROOT/header-trees//``, plus a cache of its ``.deb`` files beside it. ``corpus-check`` does not require one. ``realworld-run --header-tree `` refuses to scan against a tree that is missing or whose hash differs from the pin. Read a comparison between two trees, or between a tree and the host, as the whole environment change, not one part of it: the two trees differ in their package sets *and* in how ```` and ```` are found (the declared tree searches its multiarch directory first; the benchmark-node tree has none on the search path and reaches those headers through the multilib links under ``/usr/include``, as the node itself did). A run with ``--compile-commands`` also passes the compile database's own ``-I`` paths, which are not rewritten into the tree. To re-pin, list the packages as ``PACKAGE=VERSION`` and run ``python3 -m bench.header_tree pin PKG=VER ...``, which prints the ``debs`` entries (file, snapshot URL, sha256). Unpack them, print the tree's hash with ``python3 -m bench.header_tree hash ``, and declare a new ``id`` and hash; a changed tree changes findings, so re-pin at a release boundary and measure the change. Per-Project Include Paths ~~~~~~~~~~~~~~~~~~~~~~~~~ The real-world runner passes each codebase's ``-I`` paths automatically, from the ``includes`` field of its entry in ``CODEBASES`` in ``bench/realworld_runner.py``; that registry is the list. The corpora that read ``/usr/include`` are the five named under *Third-Party Library Headers* above. Real-World Project Setup ------------------------ Pinned Source Commits ~~~~~~~~~~~~~~~~~~~~~ All benchmark results are run against these exact commits -- the full set of projects in ``bench/realworld_runner.py``'s ``CODEBASES`` registry (which stores paths and per-tool flags, not the pins). .. note:: The **single source of truth for the pins is** ``data/benchmark_repos.json``. Both ``playbooks/setup-benchmark-repos.yml`` (provisioning) and ``python -m bench corpus-check`` (verification) read it. The table below is a human-readable mirror -- if the two disagree, the JSON wins. Do not add a third copy. Each checkout must sit at a **detached HEAD** on its pinned commit. A checkout left on a tracking branch is not pinned even while it happens to match: the next ``git pull`` moves it silently, and every finding then lands at ``(file, line)`` pairs the ``ground_truth`` oracle -- keyed on project+commit+file+line+rule -- was never adjudicated against, so it drops out of the precision/recall denominator without any error. This is not hypothetical; see `Verifying the Pins`_ below. .. list-table:: :header-rows: 1 :widths: 12 40 38 10 * - Project - Repository - Commit SHA - Checkout state * - libcrc - https://github.com/lammertb/libcrc - ``7719e2112a9a960b1bba130d02bebdf58e8701f1`` - detached * - sqlite - https://github.com/sqlite/sqlite.git - ``b1a73ba34d05b32007315e4065c6468cc638e3af`` - detached * - mosquitto - https://github.com/eclipse-mosquitto/mosquitto - ``d3ee5c5ca62c0fa4983308c6fff558ee978e878c`` - detached * - curl - https://github.com/curl/curl.git - ``3e198f75861cc2e12daf299689e145949dddd19b`` - detached * - hostap - https://git.w1.fi/hostap.git - ``dcee60436390dd34731560657c4257c3b4c839a6`` - detached * - lua - https://github.com/lua/lua.git - ``40b76de2d77e66b70a9d4bf989c3f5340919973f`` - detached * - raylib - https://github.com/raysan5/raylib.git - ``962bbfc6bfbd7a5acd08e21314fcfa161003a589`` - detached * - pureftpd - https://github.com/jedisct1/pure-ftpd.git - ``cc28bff52ca28e1d122a2142bf37f2dc578f4d3e`` - detached * - sel4 - https://github.com/seL4/seL4.git - ``1326364bc9135d9445d936ebc01e38a402c1f4c6`` - detached * - mbedtls - https://github.com/Mbed-TLS/mbedtls.git - ``068ff080b369adfac81509f9b57b2afabaf82dc5`` - detached * - valkey - https://github.com/valkey-io/valkey.git - ``7f1dffedff6de73058b2c2a389422b6ecd56c8fb`` - detached * - ventoy - https://github.com/ventoy/Ventoy.git - ``7cbdc5cf69935bcf1f085ae67f40e70ea7e74bae`` - detached .. important:: ``pureftpd``'s checkout directory must be named ``pureftpd`` (no hyphen), not the upstream ``pure-ftpd`` -- two path-parsing spots in ``bench/db.py`` (result-filename parsing, ground-truth path normalization) require the checkout dir basename to match the registry key exactly. The playbook already clones it to the right name. Verifying the Pins ~~~~~~~~~~~~~~~~~~ Provisioning pins the checkouts once; nothing kept them pinned. Run this before any real-world benchmark or precision claim: .. code-block:: bash python -m bench corpus-check # exits 1 if anything is off python -m bench corpus-check --json # machine-readable It reports one row per project, worst first: =========== ========================================================= Status Meaning =========== ========================================================= MISSING Checkout directory absent NOT_GIT Directory exists but is not a git checkout PIN_ABSENT Pinned commit not present locally (needs a fetch) DRIFTED ``HEAD`` is not the pinned commit UNPINNED ``HEAD`` matches the pin but sits on a branch, so the next ``git pull`` silently drifts it OK Detached at the pinned commit =========== ========================================================= It also flags three contamination cases independent of status: * tracked files modified, so the scanned source is not the pin; * **untracked ``*.c``/``*.h`` files, which aurora-lint will scan** and attribute to the pinned commit (untracked files aurora-lint ignores are counted separately and are harmless); * **gitignored ``*.c``/``*.h`` files**, which are invisible to ``git status`` but which aurora-lint scans anyway -- it dispatches on file extension and never consults git. The case to watch is a build run inside a checkout: sqlite's build generates a gitignored ``sqlite3.c`` amalgamation, which would silently add ~250k lines to every sqlite scan. .. tip:: Keep the checkouts pristine. Build artifacts belong in a separate tree, not inside ``$SQC_BENCH_ROOT``. .. warning:: This check exists because the failure is silent and did happen. On one machine curl, hostap and sqlite had all drifted, while libcrc and lua sat on tracking branches matching their pins only by coincidence. A gate run against the drifted trees reported slightly more hostap and sqlite findings than the pinned snapshots give -- close enough to look plausible, and not comparable to ``ground_truth`` at all. Fix drift with the ``git checkout --detach`` command the check prints for each row. Clone and Pin ~~~~~~~~~~~~~ Preferred: run ``playbooks/setup-benchmark-repos.yml`` (see `Benchmark Host Layout`_ above) -- it clones, pins, and verifies every pinned checkout in one pass, reading the pins from ``data/benchmark_repos.json``. Manual fallback: clone each repository in ``data/benchmark_repos.json`` under its ``name`` (the registry key, so pure-ftpd lands in ``pureftpd``), detach at its ``version``, and then verify: .. code-block:: bash mkdir -p $SQC_BENCH_ROOT # default: ~/toolchain cd $SQC_BENCH_ROOT python3 -c 'import json, sys for r in json.load(open(sys.argv[1]))["repos"]: print(r["repo"], r["name"], r["version"])' \ $AURORA_LINT_SRC_ROOT/aurora-lint/data/benchmark_repos.json | while read -r repo name sha; do git clone "$repo" "$name" && git -C "$name" checkout --detach "$sha" done git -C mbedtls submodule update --init -- framework cd $AURORA_LINT_SRC_ROOT/aurora-lint && python -m bench corpus-check A corpus with git submodules lists them under ``submodules`` (today only mbedtls's ``framework``), and each is checked out at the commit the pinned superproject records. The aurora-lint scan of ``library/`` never reads ``framework/``, but the library build does, so the compile database cannot be built without it. ``corpus-check`` fails a declared submodule that is missing, at another commit or modified, and a submodule in a pinned tree that is not declared. Do not build inside the valkey checkout: ``make`` leaves a gitignored ``src/release.h`` that corpus-check will (correctly) refuse. Running Each Tool Manually -------------------------- .. note:: The examples below use an illustrative ``~/data/comparisons/`` checkout path for ad hoc manual runs. If you provisioned codebases via ``playbooks/setup-benchmark-repos.yml``, substitute ``$SQC_BENCH_ROOT/`` (default ``~/toolchain/``) instead -- that's the path ``bench/realworld_runner.py`` itself reads. aurora-lint ~~~~~~~~~~~ .. code-block:: bash # Build cd $AURORA_LINT_SRC_ROOT/aurora-lint cargo build --release # Basic run ./target/release/aurora-lint /path/to/source/ --export results.json # With cross-file context ./target/release/aurora-lint /path/to/source/ -d /path/to/source/ --export results.json # When running from outside the aurora-lint repo, pass --manifest explicitly ./target/release/aurora-lint /path/to/source/ \ --manifest $AURORA_LINT_SRC_ROOT/aurora-lint/rules_templates/rules-all.toml \ --export results.json cppcheck ~~~~~~~~ .. code-block:: bash cppcheck --enable=all --std=c11 --xml --xml-version=2 \ --suppress=missingIncludeSystem \ -I /path/to/include \ /path/to/source/ \ 2> results.xml .. warning:: - cppcheck writes XML to **stderr**, not stdout - Never pass ``-I /usr/include`` -- causes parse errors in system headers - ``--addon=cert`` does not exist: open-source cppcheck removed the CERT addon in 2.8 (CERT checking is Cppcheck Premium). ``--enable=all`` turns on cppcheck's native checks, not CERT ones clang-tidy ~~~~~~~~~~ .. code-block:: bash # Option A: With compile_commands.json (recommended) cd /path/to/project make clean && bear -- make -j$(nproc) run-clang-tidy -checks='-*,cert-*,clang-analyzer-*' -p . # Option B: Direct file scan (no build required) find /path/to/source/ -name '*.c' | \ xargs -P $(nproc) -I{} clang-tidy \ -checks='-*,cert-*,clang-analyzer-*' \ {} -- -std=c11 -I /path/to/include .. warning:: - ``bear`` requires ``make clean`` first -- it only captures invocations during an actual build - Verify ``compile_commands.json`` is non-empty before running clang-tidy Per-Project Commands -------------------- libcrc ~~~~~~ .. code-block:: bash # aurora-lint $AURORA_LINT_SRC_ROOT/aurora-lint/target/release/aurora-lint ~/data/comparisons/libcrc \ -d ~/data/comparisons/libcrc \ --manifest $AURORA_LINT_SRC_ROOT/aurora-lint/rules_templates/rules-all.toml \ --export ~/data/comparisons/results/aurora-lint/libcrc/results.json # cppcheck cppcheck --enable=all --std=c11 --xml --xml-version=2 \ --suppress=missingIncludeSystem \ -I ~/data/comparisons/libcrc/include \ ~/data/comparisons/libcrc/src/ \ 2> ~/data/comparisons/results/cppcheck/libcrc/results.xml # clang-tidy cd ~/data/comparisons/libcrc && make clean && bear -- make run-clang-tidy -checks='-*,cert-*,clang-analyzer-*' -p . \ 2>&1 | tee ~/data/comparisons/results/clang-tidy/libcrc/results.txt sqlite ~~~~~~ .. code-block:: bash # aurora-lint $AURORA_LINT_SRC_ROOT/aurora-lint/target/release/aurora-lint ~/data/comparisons/sqlite \ -d ~/data/comparisons/sqlite \ --manifest $AURORA_LINT_SRC_ROOT/aurora-lint/rules_templates/rules-all.toml \ --export ~/data/comparisons/results/aurora-lint/sqlite/results.json # cppcheck cppcheck --enable=all --std=c11 --xml --xml-version=2 \ --suppress=missingIncludeSystem \ -I ~/data/comparisons/sqlite/src \ ~/data/comparisons/sqlite/src/ \ 2> ~/data/comparisons/results/cppcheck/sqlite/results.xml # clang-tidy cd ~/data/comparisons/sqlite && ./configure && make clean && bear -- make -j$(nproc) run-clang-tidy -checks='-*,cert-*,clang-analyzer-*' -p . \ 2>&1 | tee ~/data/comparisons/results/clang-tidy/sqlite/results.txt mosquitto ~~~~~~~~~ .. code-block:: bash # aurora-lint $AURORA_LINT_SRC_ROOT/aurora-lint/target/release/aurora-lint ~/data/comparisons/mosquitto \ -d ~/data/comparisons/mosquitto \ --manifest $AURORA_LINT_SRC_ROOT/aurora-lint/rules_templates/rules-all.toml \ --export ~/data/comparisons/results/aurora-lint/mosquitto/results.json # cppcheck cppcheck --enable=all --std=c11 --xml --xml-version=2 \ --suppress=missingIncludeSystem \ -I ~/data/comparisons/mosquitto/include \ -I ~/data/comparisons/mosquitto/common \ -i ~/data/comparisons/mosquitto/deps \ ~/data/comparisons/mosquitto/lib/ ~/data/comparisons/mosquitto/src/ \ 2> ~/data/comparisons/results/cppcheck/mosquitto/results.xml # clang-tidy cmake -S ~/data/comparisons/mosquitto -B ~/data/comparisons/mosquitto/build \ -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DWITH_TLS=OFF -DWITH_WEBSOCKETS=OFF -DWITH_TESTS=OFF cmake --build ~/data/comparisons/mosquitto/build --clean-first -j$(nproc) run-clang-tidy -checks='-*,cert-*,clang-analyzer-*' \ -p ~/data/comparisons/mosquitto/build \ 2>&1 | tee ~/data/comparisons/results/clang-tidy/mosquitto/results.txt curl ~~~~ .. code-block:: bash # aurora-lint $AURORA_LINT_SRC_ROOT/aurora-lint/target/release/aurora-lint ~/data/comparisons/curl \ -d ~/data/comparisons/curl \ --manifest $AURORA_LINT_SRC_ROOT/aurora-lint/rules_templates/rules-all.toml \ --export ~/data/comparisons/results/aurora-lint/curl/results.json # cppcheck cppcheck --enable=all --std=c11 --xml --xml-version=2 \ --suppress=missingIncludeSystem \ -I ~/data/comparisons/curl/include -I ~/data/comparisons/curl/lib \ ~/data/comparisons/curl/lib/ ~/data/comparisons/curl/src/ \ 2> ~/data/comparisons/results/cppcheck/curl/results.xml # clang-tidy cmake -S ~/data/comparisons/curl -B ~/data/comparisons/curl/build \ -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DBUILD_SHARED_LIBS=OFF \ -DCURL_USE_OPENSSL=OFF -DCURL_DISABLE_LDAP=ON \ -DUSE_LIBIDN2=OFF -DUSE_NGHTTP2=OFF -DCURL_ZSTD=OFF cmake --build ~/data/comparisons/curl/build --clean-first -j$(nproc) run-clang-tidy -checks='-*,cert-*,clang-analyzer-*' \ -p ~/data/comparisons/curl/build \ 2>&1 | tee ~/data/comparisons/results/clang-tidy/curl/results.txt hostap ~~~~~~ .. code-block:: bash # aurora-lint $AURORA_LINT_SRC_ROOT/aurora-lint/target/release/aurora-lint ~/data/comparisons/hostap \ -d ~/data/comparisons/hostap/src \ -d ~/data/comparisons/hostap/wpa_supplicant \ --manifest $AURORA_LINT_SRC_ROOT/aurora-lint/rules_templates/rules-all.toml \ --export ~/data/comparisons/results/aurora-lint/hostap/results.json # cppcheck cppcheck --enable=all --std=c11 --xml --xml-version=2 \ --suppress=missingIncludeSystem \ -I ~/data/comparisons/hostap/src \ -I ~/data/comparisons/hostap/src/utils \ -I ~/data/comparisons/hostap/src/common \ ~/data/comparisons/hostap/src/ ~/data/comparisons/hostap/wpa_supplicant/ \ 2> ~/data/comparisons/results/cppcheck/hostap/results.xml # clang-tidy (requires .config for wpa_supplicant) cd ~/data/comparisons/hostap/wpa_supplicant cat > .config <<'EOF' CONFIG_DRIVER_NL80211=y CONFIG_LIBNL32=y CONFIG_IEEE8021X_EAPOL=y CONFIG_EAP_MD5=y EOF make clean 2>/dev/null; bear -- make -j$(nproc) wpa_supplicant 2>/dev/null || true cd ~/data/comparisons/hostap python3 gen_compile_commands.py -o compile_commands.json wpa_supplicant/ run-clang-tidy -checks='-*,cert-*,clang-analyzer-*' -p . \ 2>&1 | tee ~/data/comparisons/results/clang-tidy/hostap/results.txt Verifying Results ----------------- .. code-block:: bash PROJECT=libcrc RESULTS=~/data/comparisons/results echo "=== aurora-lint ===" python3 -c " import json, collections data = json.load(open('$RESULTS/aurora-lint/$PROJECT/results.json')) rules = collections.Counter(v['rule_id'] for v in data) print(f'Total: {len(data)} findings, {len(rules)} rules') for r, c in rules.most_common(10): print(f' {r}: {c}') " echo "=== cppcheck ===" python3 -c " import xml.etree.ElementTree as ET, collections errors = ET.parse('$RESULTS/cppcheck/$PROJECT/results.xml').getroot().findall('.//error') ids = collections.Counter(e.get('id') for e in errors) print(f'Total: {len(errors)} findings') for i, c in ids.most_common(10): print(f' {i}: {c}') " echo "=== clang-tidy ===" grep -c "warning:\|error:" $RESULTS/clang-tidy/$PROJECT/results.txt Known Pitfalls ~~~~~~~~~~~~~~ .. list-table:: :header-rows: 1 :widths: 30 35 35 * - Pitfall - Symptom - Fix * - aurora-lint manifest not found - ``Failed to read manifest file`` - Pass ``--manifest`` with absolute path * - cppcheck ``--addon=cert`` missing - ``Did not find addon cert.py`` - Remove it; the addon is gone from open-source cppcheck (2.8 on), so there are no CERT checks to enable * - cppcheck ``-I /usr/include`` - ``syntaxError`` in stdlib.h - Never pass system include paths to cppcheck * - Empty compile_commands.json - clang-tidy shows no diagnostics - Run ``make clean`` before ``bear -- make`` Output Format Reference ----------------------- **cppcheck XML (v2)**: .. code-block:: xml **clang-tidy text**:: example.c:5:5: warning: ... [cert-arr30-c] **aurora-lint JSON**: .. code-block:: json [{"rule_id": "ARR30-C", "severity": "High", "file": "example.c", "line": 5, "message": "..."}] **aurora-lint SARIF**: SARIF 2.1.0 compatible output with ``ruleId`` matching CERT C rule IDs. **Infer JSON** (``infer-out/report.json``): .. code-block:: json [{"bug_type": "NULLPTR_DEREFERENCE", "procedure": "func_bad", "file": "test.c", "line": 26, "severity": "ERROR"}] **Frama-C EVA** (stderr, no structured output): :: [eva:alarm] test.c:26: Warning: out of bounds read. assert \valid_read(&ptr->field); assertion 'Eva,mem_access' got final status invalid. Distributed Benchmarking ------------------------ Multi-host runs belong to the maintainer's ``benchmarking_db`` infrastructure, not to this repo. Exporting competitor results for ingest --------------------------------------- ``bench/competitors.py`` writes one JSON blob per run into ``data/competitor_results/``, and those blobs are committed — they are the archival form and should not be edited or deleted. They are not, however, ingestible: the shape is nested and the per-CWE map is keyed by CWE id, so a loader would have to know that module's internal layout to read it. ``python -m bench competitor-export`` flattens every run JSON in that directory into three CSVs beside them: .. code-block:: text competitor_runs.csv one row per run (tool, version, totals) competitor_cwe_results.csv one row per run × CWE (the measurements) competitor_cwe_errors.csv one row per error (usually empty) ``run_key`` joins them and is the JSON's own basename (e.g. ``framac_20260403_222053``), so it is unique, stable across re-exports, and traceable back to the file it came from. **Run the export after every competitor benchmark and commit the result.** The benchmark host ingests these CSVs into ``sqc_bench``; a table missing its newest run looks exactly like a run that never happened. ``python -m bench competitor-export --check`` exits nonzero if the CSVs are stale, and writes nothing. This repo stays Postgres-blind — no DSN, no connection code (see ``CLAUDE.md``). It *emits* a flat, diffable table; ``benchmarking_db`` owns the ingest, so nothing here changes when the target schema does. The CSVs are a transport, not a source of truth: Postgres remains the only place an official number comes from, and re-exporting is idempotent and lossless with respect to the JSON. .. note:: The run JSONs record no hostname. ``run_hosts.json`` attributes each run, and the CSV carries it as ``hostname`` + ``hostname_source``. Wall clock is hardware-dependent, so compare durations only between runs on the same host.