Benchmark Setup

This section covers installing comparison tools, setting up the Juliet test suite, and configuring real-world codebases for benchmarking.

Benchmark Host Layout

Every benchmark node needs one directory – $SQC_BENCH_ROOT (default ~/toolchain) – holding the Juliet suite and every real-world codebase checkout. bench/config.py and bench/realworld_runner.py both read this from the SQC_BENCH_ROOT environment variable, or from a .env file at the repo root (copy .env.example – it’s gitignored, so each machine sets its own).

To provision a fresh node, run the Ansible playbook (clones and pins every real-world codebases; does not fetch Juliet – see below):

ansible-playbook playbooks/setup-benchmark-repos.yml -i "localhost," -c local
# Or a different root:
ansible-playbook playbooks/setup-benchmark-repos.yml -i "localhost," -c local \
  -e bench_root=/data/toolchain

Keep bench_root and SQC_BENCH_ROOT/.env pointed at the same directory – the playbook provisions the checkouts the code actually reads. A mismatch is easy to miss: if SQC_BENCH_ROOT is unset and the corpora do not live in ~/toolchain, BENCH_ROOT silently resolves to a directory that does not exist. python -m bench corpus-check reports the resolved root and says so outright.

A second, separate root names where the source clones are: $AURORA_LINT_SRC_ROOT is the parent directory of this checkout ($AURORA_LINT_SRC_ROOT/aurora-lint), and on a maintainer node of the sibling benchmarking_db and sqc_paper clones too. It defaults to the checkout’s own parent, so nothing here requires it; the copy-pasteable commands below use it so they hold on every node regardless of where the clone lives (~/data is the conventional value, and the .env.example default):

export AURORA_LINT_SRC_ROOT=~/data   # parent of your aurora-lint clone

Installing Comparison Tools

cppcheck

sudo apt update
sudo apt install -y cppcheck

# Verify
cppcheck --version
# Expected: Cppcheck 2.x.x

For the latest release, build from source:

sudo apt install -y cmake libpcre3-dev
git clone https://github.com/danmar/cppcheck.git
cd cppcheck
cmake -DCMAKE_BUILD_TYPE=Release -DUSE_MATCHCOMPILER=ON .
make -j$(nproc)
sudo make install

clang-tidy

Pin LLVM 21 — do not take the distro default. Ubuntu 24.04 ships clang-tidy 18.1.3 and tops out at 20, while the published comparison in docs/tool-comparison.rst is LLVM 21.1. Taking the distro package makes a freshly-provisioned node measure an older competitor than the table it will be compared against, and clang-tidy is the tool currently beating aurora-lint on the Juliet overlap (99.2% vs 81.7%) — so that regression would flatter aurora-lint by understating a rival. playbooks/install-static-analyzers.yml does this automatically; by hand:

sudo install -d -m 0755 /etc/apt/keyrings
wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key \
  | sudo tee /etc/apt/keyrings/apt.llvm.org.asc > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/apt.llvm.org.asc] \
  http://apt.llvm.org/noble/ llvm-toolchain-noble-21 main" \
  | sudo tee /etc/apt/sources.list.d/llvm-21.list
sudo apt update && sudo apt install -y clang-tidy-21
sudo update-alternatives --install /usr/bin/clang-tidy clang-tidy \
  /usr/bin/clang-tidy-21 100

# Verify
clang-tidy --version
# Expected: Ubuntu LLVM version 21.1.x

The distro package is fine for anything that is not a published comparison:

sudo apt install -y clang clang-tidy

bear (for compile_commands.json)

Required for clang-tidy on projects with build systems:

sudo apt install -y bear

Facebook Infer

Infer v1.2.0 is installed from prebuilt Linux binaries. The playbook at playbooks/install-static-analyzers.yml automates this:

ansible-playbook playbooks/install-static-analyzers.yml \
  -i "localhost," -c local --ask-become-pass

Manual install:

# libtinfo5 required (Infer links against libtinfo.so.5)
sudo apt install -y libtinfo5 || \
  sudo ln -s /usr/lib/x86_64-linux-gnu/libtinfo.so.6 \
              /usr/lib/x86_64-linux-gnu/libtinfo.so.5

VERSION=1.2.0
curl -sSL "https://github.com/facebook/infer/releases/download/v$VERSION/infer-linux-x86_64-v$VERSION.tar.xz" \
  | sudo tar -C /opt -xJ
sudo ln -s "/opt/infer-linux-x86_64-v$VERSION/bin/infer" /usr/local/bin/infer

# Verify
infer --version
# Expected: Infer version v1.2.0

Frama-C

Frama-C is installed via opam (the Debian 12 apt package is version 25/Manganese, too old for benchmarking). The playbook handles this automatically.

Manual install:

sudo apt install -y opam gcc g++ make m4 pkg-config autoconf \
  libgmp-dev zlib1g-dev libgtk-3-dev libgtksourceview-3.0-dev graphviz

opam init --bare --no-setup --disable-sandboxing
opam switch create default 4.14.2
eval $(opam env)
opam install -y frama-c

# Verify (requires eval $(opam env) in each shell session)
frama-c -version
# Expected: 33.0 (Arsenic) as of 2026-09-04; 32.0 (Germanium) is what the
# competitor Juliet runs in data/competitor_results/ were taken with

Note

Add eval $(opam env) to your shell profile (~/.bashrc) to make frama-c available without manual activation. The benchmark runner does not need it: bench/realworld_runner.py wraps every Frama-C invocation in eval $(opam env) itself, because it is not a login shell.

Warning

Frama-C renamed the compile-database option between the two versions above — -json-compilation-database through 32.0, -compilation-db from 33.0. The runner probes frama-c -kernel-h and picks whichever the installed binary accepts, so both work; a hand-written command line copied from an older doc will abort with “option is unknown”.

Build-based tools need a compile database

Infer and Frama-C cannot be pointed at a source tree the way aurora-lint and cppcheck can — both need a real preprocess. On the real-world suite they are driven from each checkout’s compile_commands.json, so playbooks/setup-compile-commands.yml must have run for a codebase before either tool can scan it. python -m bench realworld-run --tool infer fails fast with the exact command to fix it if the database is missing.

Expect a partial result from both, and check the coverage block each run writes:

• Infer captures whatever preprocesses. setup-compile-commands.yml restores each checkout to pristine after building it, which deletes the build’s generated headers; the playbook now stashes the ones the compile database still references (pure-ftpd’s config.h, sqlite’s sqlite_cfg.h, valkey’s jemalloc/jemalloc.h, libcrc’s tab/gentab*.inc) under <bench_root>/.cc_build/generated/ and points the database there, so every in-scope translation unit preprocesses. A capture_failures list that is not empty means a database written before that existed — re-run the playbook, which detects and rebuilds it.

• Frama-C is partial by construction, because EVA analyzes one entry point at a time and a real codebase has no single one. Read docs/design/framac-realworld.md before quoting any Frama-C real-world number: a finding count from it is a floor, and recall is not expressible from it at all.

Juliet Test Suite Setup

NIST SARD is a click-through portal with no stable direct-download URL, so this step is manual (the setup playbook only creates the parent directory and reports whether the suite is already in place). Download the NIST Juliet Test Suite v1.3:

mkdir -p $SQC_BENCH_ROOT/benchmarks   # default: ~/toolchain/benchmarks
cd $SQC_BENCH_ROOT/benchmarks
# Download and extract juliet-test-suite-c
# Expected structure:
#   juliet-test-suite-c/testcases/CWE*/     (118 CWE directories)
#   juliet-test-suite-c/testcasesupport/    (shared helper functions)

Third-Party Library Headers

aurora-lint uses -I include paths to resolve #include directives from third-party libraries. Without these, functions declared in external headers produce DCL31-C/DCL07-C false positives. The installed headers are therefore part of what a run measures – the same binary on two machines with different -dev packages or glibc releases differs by a small number of keys (see Reproducing the Published Numbers), so install this list before comparing a run against a published one.

# Core dependencies (covers most projects)
sudo apt-get install -y \
  libssl-dev libcjson-dev zlib1g-dev

# mosquitto
sudo apt-get install -y libcunit1-dev libsqlite3-dev

# curl TLS and SASL backends (without libgsasl-dev, lib/vauth/gsasl.c
# reports every gsasl_* call as DCL31-C "called without prior declaration")
sudo apt-get install -y libmbedtls-dev libgnutls28-dev libgsasl-dev

# sqlite test infrastructure
sudo apt-get install -y tcl-dev

# hostap
sudo apt-get install -y \
  libnl-3-dev libnl-genl-3-dev libdbus-1-dev \
  libgcrypt20-dev libpcap-dev libwolfssl-dev

One-liner for all hosts:

sudo apt-get install -y libssl-dev libcjson-dev zlib1g-dev libcunit1-dev \
  libsqlite3-dev libmbedtls-dev libgnutls28-dev libgsasl-dev tcl-dev \
  libnl-3-dev libnl-genl-3-dev libdbus-1-dev libgcrypt20-dev libpcap-dev \
  libwolfssl-dev

Per-Project Include Paths

The real-world runner passes these automatically via the includes field in each codebase’s config:

Project	-I Paths	Resolves
libcrc	(none)	Pure C, no external deps
sqlite	/usr/include	OpenSSL, zlib, Tcl
mosquitto	/usr/include, /usr/include/cjson	OpenSSL, cJSON, CUnit, sqlite3
curl	/usr/include, {path}/lib	OpenSSL, mbedTLS, GnuTLS
hostap	/usr/include, /usr/include/libnl3, /usr/include/dbus-1.0	OpenSSL, wolfSSL, libnl, D-Bus, libgcrypt, libpcap

Real-World Project Setup

Pinned Source Commits

All benchmark results are run against these exact commits – the full set of projects in bench/realworld_runner.py’s CODEBASES registry (which stores paths and per-tool flags, not the pins).

Note

The single source of truth for the pins is data/benchmark_repos.json. Both playbooks/setup-benchmark-repos.yml (provisioning) and python -m bench corpus-check (verification) read it. The table below is a human-readable mirror – if the two disagree, the JSON wins. Do not add a third copy.

Each checkout must sit at a detached HEAD on its pinned commit. A checkout left on a tracking branch is not pinned even while it happens to match: the next git pull moves it silently, and every finding then lands at (file, line) pairs the ground_truth oracle – keyed on project+commit+file+line+rule – was never adjudicated against, so it drops out of the precision/recall denominator without any error. This is not hypothetical; see Verifying the Pins below.

Project	Repository	Commit SHA	Checkout state
libcrc	https://github.com/lammertb/libcrc	7719e2112a9a960b1bba130d02bebdf58e8701f1	detached
sqlite	https://github.com/sqlite/sqlite.git	b1a73ba34d05b32007315e4065c6468cc638e3af	detached
mosquitto	https://github.com/eclipse-mosquitto/mosquitto	d3ee5c5ca62c0fa4983308c6fff558ee978e878c	detached
curl	https://github.com/curl/curl.git	3e198f75861cc2e12daf299689e145949dddd19b	detached
hostap	https://git.w1.fi/hostap.git	dcee60436390dd34731560657c4257c3b4c839a6	detached
lua	https://github.com/lua/lua.git	40b76de2d77e66b70a9d4bf989c3f5340919973f	detached
raylib	https://github.com/raysan5/raylib.git	962bbfc6bfbd7a5acd08e21314fcfa161003a589	detached
pureftpd	https://github.com/jedisct1/pure-ftpd.git	cc28bff52ca28e1d122a2142bf37f2dc578f4d3e	detached
sel4	https://github.com/seL4/seL4.git	1326364bc9135d9445d936ebc01e38a402c1f4c6	detached
mbedtls	https://github.com/Mbed-TLS/mbedtls.git	068ff080b369adfac81509f9b57b2afabaf82dc5	detached
valkey	https://github.com/valkey-io/valkey.git	7f1dffedff6de73058b2c2a389422b6ecd56c8fb	detached
ventoy	https://github.com/ventoy/Ventoy.git	7cbdc5cf69935bcf1f085ae67f40e70ea7e74bae	detached

Important

pureftpd’s checkout directory must be named pureftpd (no hyphen), not the upstream pure-ftpd – two path-parsing spots in bench/db.py (result-filename parsing, ground-truth path normalization) require the checkout dir basename to match the registry key exactly. The playbook already clones it to the right name.

Verifying the Pins

Provisioning pins the checkouts once; nothing kept them pinned. Run this before any real-world benchmark or precision claim:

python -m bench corpus-check          # exits 1 if anything is off
python -m bench corpus-check --json   # machine-readable

It reports one row per project, worst first:

Status	Meaning
MISSING	Checkout directory absent
NOT_GIT	Directory exists but is not a git checkout
PIN_ABSENT	Pinned commit not present locally (needs a fetch)
DRIFTED	HEAD is not the pinned commit
UNPINNED	HEAD matches the pin but sits on a branch, so the next git pull silently drifts it
OK	Detached at the pinned commit

It also flags three contamination cases independent of status:

• tracked files modified, so the scanned source is not the pin;

• untracked ``*.c``/``*.h`` files, which aurora-lint will scan and attribute to the pinned commit (untracked files aurora-lint ignores are counted separately and are harmless);

• gitignored ``*.c``/``*.h`` files, which are invisible to git status but which aurora-lint scans anyway – it dispatches on file extension and never consults git. The case to watch is a build run inside a checkout: sqlite’s build generates a gitignored sqlite3.c amalgamation, which would silently add ~250k lines to every sqlite scan.

Tip

Keep the checkouts pristine. Build artifacts belong in a separate tree, not inside $SQC_BENCH_ROOT.

Warning

This check exists because the failure is silent and did happen. On the work node curl, hostap and sqlite had all drifted, while libcrc and lua sat on tracking branches matching their pins only by coincidence. A gate run against the drifted trees reported hostap 452 / sqlite 516 findings where the pinned snapshots give 447 / 506 – close enough to look plausible, and not comparable to ground_truth at all. Fix drift with the git checkout --detach command the check prints for each row.

Clone and Pin

Preferred: run playbooks/setup-benchmark-repos.yml (see Benchmark Host Layout above) – it clones, pins, and verifies every pinned checkout in one pass, reading the pins from data/benchmark_repos.json.

Manual fallback:

mkdir -p $SQC_BENCH_ROOT   # default: ~/toolchain
cd $SQC_BENCH_ROOT

git clone https://github.com/lammertb/libcrc.git
cd libcrc && git checkout 7719e2112a9a960b1bba130d02bebdf58e8701f1 && cd ..

git clone https://github.com/sqlite/sqlite.git
cd sqlite && git checkout b1a73ba34d05b32007315e4065c6468cc638e3af && cd ..

git clone https://github.com/eclipse-mosquitto/mosquitto.git
cd mosquitto && git checkout d3ee5c5ca62c0fa4983308c6fff558ee978e878c && cd ..

git clone https://github.com/curl/curl.git
cd curl && git checkout 3e198f75861cc2e12daf299689e145949dddd19b && cd ..

git clone https://git.w1.fi/hostap.git
cd hostap && git checkout dcee60436390dd34731560657c4257c3b4c839a6 && cd ..

git clone https://github.com/lua/lua.git
cd lua && git checkout 40b76de2d77e66b70a9d4bf989c3f5340919973f && cd ..

git clone https://github.com/raysan5/raylib.git
cd raylib && git checkout 962bbfc6bfbd7a5acd08e21314fcfa161003a589 && cd ..

# NOTE: checkout dir must be "pureftpd" (no hyphen) -- see the warning above
git clone https://github.com/jedisct1/pure-ftpd.git pureftpd
cd pureftpd && git checkout cc28bff52ca28e1d122a2142bf37f2dc578f4d3e && cd ..

# NOTE: checkout dir must be lowercase "sel4" to match the registry key
git clone https://github.com/seL4/seL4.git sel4
cd sel4 && git checkout 1326364bc9135d9445d936ebc01e38a402c1f4c6 && cd ..

# The framework submodule is only needed for the compile_commands.json
# capture (playbooks/setup-compile-commands.yml); the aurora-lint scan
# of library/ does not need it.
git clone https://github.com/Mbed-TLS/mbedtls.git
cd mbedtls && git checkout 068ff080b369adfac81509f9b57b2afabaf82dc5 && cd ..

# Do not build inside this checkout: `make` leaves a gitignored
# src/release.h that corpus-check will (correctly) refuse.
git clone https://github.com/valkey-io/valkey.git
cd valkey && git checkout 7f1dffedff6de73058b2c2a389422b6ecd56c8fb && cd ..

# NOTE: checkout dir must be lowercase "ventoy" to match the registry key.
# Only Ventoy2Disk/Ventoy2Disk/ is scanned; the rest of the repo is
# Linux/GRUB/firmware code outside a CERT-C scan's interest.
git clone https://github.com/ventoy/Ventoy.git ventoy
cd ventoy && git checkout 7cbdc5cf69935bcf1f085ae67f40e70ea7e74bae && cd ..

Running Each Tool Manually

Note

The examples below use an illustrative ~/data/comparisons/<project> checkout path for ad hoc manual runs. If you provisioned codebases via playbooks/setup-benchmark-repos.yml, substitute $SQC_BENCH_ROOT/<project> (default ~/toolchain/<project>) instead – that’s the path bench/realworld_runner.py itself reads.

aurora-lint ~~~

# Build
cd $AURORA_LINT_SRC_ROOT/aurora-lint
cargo build --release

# Basic run
./target/release/aurora-lint /path/to/source/ --export results.json

# With cross-file context
./target/release/aurora-lint /path/to/source/ -d /path/to/source/ --export results.json

# When running from outside the aurora-lint repo, pass --manifest explicitly
./target/release/aurora-lint /path/to/source/ \
  --manifest $AURORA_LINT_SRC_ROOT/aurora-lint/rules_templates/rules-all.toml \
  --export results.json

cppcheck

cppcheck --enable=all --std=c11 --xml --xml-version=2 \
  --suppress=missingIncludeSystem \
  -I /path/to/include \
  /path/to/source/ \
  2> results.xml

Warning

• cppcheck writes XML to stderr, not stdout

• Never pass -I /usr/include – causes parse errors in system headers

• --addon=cert is not available on Ubuntu 24.04; --enable=all includes CERT checks

clang-tidy

# Option A: With compile_commands.json (recommended)
cd /path/to/project
make clean && bear -- make -j$(nproc)
run-clang-tidy -checks='-*,cert-*,clang-analyzer-*' -p .

# Option B: Direct file scan (no build required)
find /path/to/source/ -name '*.c' | \
  xargs -P $(nproc) -I{} clang-tidy \
    -checks='-*,cert-*,clang-analyzer-*' \
    {} -- -std=c11 -I /path/to/include

Warning

• bear requires make clean first – it only captures invocations during an actual build

• Verify compile_commands.json is non-empty before running clang-tidy

Per-Project Commands

libcrc

# aurora-lint
$AURORA_LINT_SRC_ROOT/aurora-lint/target/release/aurora-lint ~/data/comparisons/libcrc \
  -d ~/data/comparisons/libcrc \
  --manifest $AURORA_LINT_SRC_ROOT/aurora-lint/rules_templates/rules-all.toml \
  --export ~/data/comparisons/results/aurora-lint/libcrc/results.json

# cppcheck
cppcheck --enable=all --std=c11 --xml --xml-version=2 \
  --suppress=missingIncludeSystem \
  -I ~/data/comparisons/libcrc/include \
  ~/data/comparisons/libcrc/src/ \
  2> ~/data/comparisons/results/cppcheck/libcrc/results.xml

# clang-tidy
cd ~/data/comparisons/libcrc && make clean && bear -- make
run-clang-tidy -checks='-*,cert-*,clang-analyzer-*' -p . \
  2>&1 | tee ~/data/comparisons/results/clang-tidy/libcrc/results.txt

sqlite

# aurora-lint
$AURORA_LINT_SRC_ROOT/aurora-lint/target/release/aurora-lint ~/data/comparisons/sqlite \
  -d ~/data/comparisons/sqlite \
  --manifest $AURORA_LINT_SRC_ROOT/aurora-lint/rules_templates/rules-all.toml \
  --export ~/data/comparisons/results/aurora-lint/sqlite/results.json

# cppcheck
cppcheck --enable=all --std=c11 --xml --xml-version=2 \
  --suppress=missingIncludeSystem \
  -I ~/data/comparisons/sqlite/src \
  ~/data/comparisons/sqlite/src/ \
  2> ~/data/comparisons/results/cppcheck/sqlite/results.xml

# clang-tidy
cd ~/data/comparisons/sqlite && ./configure && make clean && bear -- make -j$(nproc)
run-clang-tidy -checks='-*,cert-*,clang-analyzer-*' -p . \
  2>&1 | tee ~/data/comparisons/results/clang-tidy/sqlite/results.txt

mosquitto

# aurora-lint
$AURORA_LINT_SRC_ROOT/aurora-lint/target/release/aurora-lint ~/data/comparisons/mosquitto \
  -d ~/data/comparisons/mosquitto \
  --manifest $AURORA_LINT_SRC_ROOT/aurora-lint/rules_templates/rules-all.toml \
  --export ~/data/comparisons/results/aurora-lint/mosquitto/results.json

# cppcheck
cppcheck --enable=all --std=c11 --xml --xml-version=2 \
  --suppress=missingIncludeSystem \
  -I ~/data/comparisons/mosquitto/include \
  -I ~/data/comparisons/mosquitto/common \
  -i ~/data/comparisons/mosquitto/deps \
  ~/data/comparisons/mosquitto/lib/ ~/data/comparisons/mosquitto/src/ \
  2> ~/data/comparisons/results/cppcheck/mosquitto/results.xml

# clang-tidy
cmake -S ~/data/comparisons/mosquitto -B ~/data/comparisons/mosquitto/build \
  -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DWITH_TLS=OFF -DWITH_WEBSOCKETS=OFF -DWITH_TESTS=OFF
cmake --build ~/data/comparisons/mosquitto/build --clean-first -j$(nproc)
run-clang-tidy -checks='-*,cert-*,clang-analyzer-*' \
  -p ~/data/comparisons/mosquitto/build \
  2>&1 | tee ~/data/comparisons/results/clang-tidy/mosquitto/results.txt

curl

# aurora-lint
$AURORA_LINT_SRC_ROOT/aurora-lint/target/release/aurora-lint ~/data/comparisons/curl \
  -d ~/data/comparisons/curl \
  --manifest $AURORA_LINT_SRC_ROOT/aurora-lint/rules_templates/rules-all.toml \
  --export ~/data/comparisons/results/aurora-lint/curl/results.json

# cppcheck
cppcheck --enable=all --std=c11 --xml --xml-version=2 \
  --suppress=missingIncludeSystem \
  -I ~/data/comparisons/curl/include -I ~/data/comparisons/curl/lib \
  ~/data/comparisons/curl/lib/ ~/data/comparisons/curl/src/ \
  2> ~/data/comparisons/results/cppcheck/curl/results.xml

# clang-tidy
cmake -S ~/data/comparisons/curl -B ~/data/comparisons/curl/build \
  -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DBUILD_SHARED_LIBS=OFF \
  -DCURL_USE_OPENSSL=OFF -DCURL_DISABLE_LDAP=ON \
  -DUSE_LIBIDN2=OFF -DUSE_NGHTTP2=OFF -DCURL_ZSTD=OFF
cmake --build ~/data/comparisons/curl/build --clean-first -j$(nproc)
run-clang-tidy -checks='-*,cert-*,clang-analyzer-*' \
  -p ~/data/comparisons/curl/build \
  2>&1 | tee ~/data/comparisons/results/clang-tidy/curl/results.txt

hostap

# aurora-lint
$AURORA_LINT_SRC_ROOT/aurora-lint/target/release/aurora-lint ~/data/comparisons/hostap \
  -d ~/data/comparisons/hostap/src \
  -d ~/data/comparisons/hostap/wpa_supplicant \
  --manifest $AURORA_LINT_SRC_ROOT/aurora-lint/rules_templates/rules-all.toml \
  --export ~/data/comparisons/results/aurora-lint/hostap/results.json

# cppcheck
cppcheck --enable=all --std=c11 --xml --xml-version=2 \
  --suppress=missingIncludeSystem \
  -I ~/data/comparisons/hostap/src \
  -I ~/data/comparisons/hostap/src/utils \
  -I ~/data/comparisons/hostap/src/common \
  ~/data/comparisons/hostap/src/ ~/data/comparisons/hostap/wpa_supplicant/ \
  2> ~/data/comparisons/results/cppcheck/hostap/results.xml

# clang-tidy (requires .config for wpa_supplicant)
cd ~/data/comparisons/hostap/wpa_supplicant
cat > .config <<'EOF'
CONFIG_DRIVER_NL80211=y
CONFIG_LIBNL32=y
CONFIG_IEEE8021X_EAPOL=y
CONFIG_EAP_MD5=y
EOF
make clean 2>/dev/null; bear -- make -j$(nproc) wpa_supplicant 2>/dev/null || true
cd ~/data/comparisons/hostap
python3 gen_compile_commands.py -o compile_commands.json wpa_supplicant/
run-clang-tidy -checks='-*,cert-*,clang-analyzer-*' -p . \
  2>&1 | tee ~/data/comparisons/results/clang-tidy/hostap/results.txt

Verifying Results

PROJECT=libcrc
RESULTS=~/data/comparisons/results

echo "=== aurora-lint ==="
python3 -c "
import json, collections
data = json.load(open('$RESULTS/aurora-lint/$PROJECT/results.json'))
rules = collections.Counter(v['rule_id'] for v in data)
print(f'Total: {len(data)} findings, {len(rules)} rules')
for r, c in rules.most_common(10): print(f'  {r}: {c}')
"

echo "=== cppcheck ==="
python3 -c "
import xml.etree.ElementTree as ET, collections
errors = ET.parse('$RESULTS/cppcheck/$PROJECT/results.xml').getroot().findall('.//error')
ids = collections.Counter(e.get('id') for e in errors)
print(f'Total: {len(errors)} findings')
for i, c in ids.most_common(10): print(f'  {i}: {c}')
"

echo "=== clang-tidy ==="
grep -c "warning:\|error:" $RESULTS/clang-tidy/$PROJECT/results.txt

Known Pitfalls

Pitfall	Symptom	Fix
aurora-lint manifest not found	Failed to read manifest file	Pass --manifest with absolute path
cppcheck --addon=cert missing	Did not find addon cert.py	Remove it; --enable=all includes CERT
cppcheck -I /usr/include	syntaxError in stdlib.h	Never pass system include paths to cppcheck
Empty compile_commands.json	clang-tidy shows no diagnostics	Run make clean before bear -- make

Output Format Reference

cppcheck XML (v2):

<error id="bufferAccessOutOfBounds" severity="error"
       msg="Array 'arr[10]' accessed at index 10..." cwe="788">
  <location file="example.c" line="5" column="5"/>
</error>

clang-tidy text:

example.c:5:5: warning: ... [cert-arr30-c]

aurora-lint JSON:

[{"rule_id": "ARR30-C", "severity": "High", "file": "example.c", "line": 5, "message": "..."}]

aurora-lint SARIF: SARIF 2.1.0 compatible output with ruleId matching CERT C rule IDs.

Infer JSON (infer-out/report.json):

[{"bug_type": "NULLPTR_DEREFERENCE", "procedure": "func_bad",
  "file": "test.c", "line": 26, "severity": "ERROR"}]

Frama-C EVA (stderr, no structured output):

[eva:alarm] test.c:26: Warning:
  out of bounds read. assert \valid_read(&ptr->field);
assertion 'Eva,mem_access' got final status invalid.

Distributed Benchmarking with GNU Parallel

For fast re-benchmarking across multiple machines after rule changes. Cppcheck and clang-tidy results are stable across aurora-lint changes – run them once and cache. Only aurora-lint needs re-running.

# Prerequisites
sudo apt install -y parallel
parallel --citation <<< "will cite" 2>/dev/null || true

# SSH key setup
ssh-keygen -t ed25519 -f ~/.ssh/id_benchmark -N ""
for node in node1 node2 node3 node4; do
  ssh-copy-id -i ~/.ssh/id_benchmark.pub $node
done

Node file (~/.benchmark_nodes):

user@node1/8
user@node2/8
user@node3/8
user@node4/8

Fast re-benchmark workflow:

# 1. Rebuild aurora-lint
cd $AURORA_LINT_SRC_ROOT/aurora-lint && cargo build --release

# 2. Push binary to nodes (if no shared FS)
parallel --sshloginfile $NODES_FILE --nonall \
  "rsync -az $SQC_BIN {}/sqc_bin"

# 3. Generate CWE list
find $JULIET_DIR -maxdepth 1 -type d -name 'CWE*' | sort > /tmp/cwe_dirs.txt

# 4. Run in parallel across nodes
parallel --sshloginfile $NODES_FILE -a /tmp/cwe_dirs.txt \
  "$SQC_BIN {} -d $JULIET_DIR -d $JULIET_SUPPORT \
    --export $RESULTS_DIR/aurora-lint/juliet/{/.}.csv"

Exporting competitor results for ingest

bench/competitors.py writes one JSON blob per run into data/competitor_results/, and those blobs are committed — they are the archival form and should not be edited or deleted. They are not, however, ingestible: the shape is nested and the per-CWE map is keyed by CWE id, so a loader would have to know that module’s internal layout to read it.

python -m bench competitor-export flattens every run JSON in that directory into three CSVs beside them:

competitor_runs.csv         one row per run       (tool, version, totals)
competitor_cwe_results.csv  one row per run × CWE (the measurements)
competitor_cwe_errors.csv   one row per error     (usually empty)

run_key joins them and is the JSON’s own basename (e.g. framac_20260403_222053), so it is unique, stable across re-exports, and traceable back to the file it came from.

Run the export after every competitor benchmark and commit the result. The benchmark host ingests these CSVs into sqc_bench; a table missing its newest run looks exactly like a run that never happened. python -m bench competitor-export --check exits nonzero if the CSVs are stale, and writes nothing.

This repo stays Postgres-blind — no DSN, no connection code (see CLAUDE.md). It emits a flat, diffable table; benchmarking_db owns the ingest, so nothing here changes when the target schema does. The CSVs are a transport, not a source of truth: Postgres remains the only place an official number comes from, and re-exporting is idempotent and lossless with respect to the JSON.

Note

The hostname column is blank for every run recorded before 2026-09-04, which is all four April runs. Wall clock is hardware-dependent, so a blank means “do not compare this run’s duration against another host’s”, not “same host”.