Testing Methodology

SqC employs a three-tier testing strategy: unit tests for individual rule logic, the NIST Juliet Test Suite for precision/recall measurement, and real-world open-source codebases for scalability and noise validation.

Benchmark Strategy

SqC is benchmarked on two axes:

  1. Juliet Test Suite (NIST) — 54,484 files with ground truth (OMITBAD/OMITGOOD sections). Measures TP rate, FP rate, and per-CWE coverage.

  2. Real-World Open-Source Projects — 5 codebases (libcrc, sqlite, mosquitto, curl, hostap) analyzed by sqc, cppcheck, and clang-tidy. No ground truth — measures violation counts, rule distribution, and cross-tool agreement.

Why both:

  • Juliet provides precision metrics (TP/FP) but is synthetic single-file code

  • Real-world tests scalability, noise levels, and cross-file analysis on production code

  • Rule improvements are validated on Juliet for TP/FP impact, then verified on real-world for noise reduction

Benchmark cadence:

  • After every significant rule change: Juliet benchmark (MCP server, ~10 min)

  • After version milestones: Full real-world benchmark (MCP server, all 5 codebases × 3 tools)

  • cppcheck/clang-tidy results are stable across sqc changes — run once and cache

Unit Tests

Each CERT C rule has dedicated test cases written as C source files organized under src/rules/cert_c/<CATEGORY>/<RULE-ID>/tests/:

src/rules/cert_c/SIG/SIG01-C/tests/
  fail/                          # C files that SHOULD trigger violations
    testcases_signal_restart_assumption.c
    testcases_concurrent_signals.c
    ...
  pass/                          # C files that should NOT trigger violations
    testcases_proper_signal_handling.c
    ...

Current coverage: 3,322 tests across 290 rules (~3,070 C test files, ~1,820 fail + ~1,250 pass). All tests pass; zero duplicates.

Tests are auto-generated into Rust test functions from .c files — no embedded #[cfg(test)] modules in rule implementation files. Run tests with:

# All tests
cargo test

# Tests for a specific rule
cargo test --package sqc --lib -- rules::cert_c::sig01_c::tests

# Tests for a category
cargo test --package sqc --lib -- rules::cert_c::mem

Test cases are derived from patterns documented in the SEI CERT C Coding Standard wiki. Each rule’s wiki page provides:

  • Non-compliant code examples: patterns that violate the rule

  • Compliant solutions: corrected versions of the same patterns

  • Risk assessment: severity, likelihood, and remediation cost

Test cases map these directly:

  • fail/ cases encode non-compliant patterns (expected violations)

  • pass/ cases encode compliant solutions (expected clean)

NIST Juliet Test Suite Benchmarking

The NIST Juliet Test Suite v1.3 is a collection of 54,484 C/C++ files covering 118 CWE categories, each containing known-bad (OMITGOOD) and known-good (OMITBAD) code sections. This provides ground truth for measuring true positive and false positive rates.

How Juliet Benchmarking Works

  1. CWE-matched manifests: For each CWE, a TOML manifest enables only the CERT C rules that map to that CWE (e.g., CWE-476 enables EXP34-C). This eliminates noise from unrelated rules.

  2. Per-CWE analysis: SqC scans each CWE’s test cases with its matched manifest. Violations in bad functions are true positives; violations in good functions are false positives.

  3. Parallel execution: CWEs are processed in parallel via Python’s ProcessPoolExecutor for fast turnaround (~8-10 min on 4-core, ~3-5 min on 24-core).

  4. Results stored in SQLite: All results go to data/benchmarks.db with per-CWE metrics, per-rule breakdowns, and cross-version comparison support.

Running the benchmark:

# Via CLI
python -m bench juliet          # Fast mode (CWE-matched rules only)
python -m bench juliet --full   # Full suite (all rules on all CWEs)

# Query results
python -m bench runs            # List all benchmark runs
python -m bench status RUN_ID   # Check a running benchmark
python -m bench compare v1 v2   # Compare two runs

Current Results (v0.3.119)

Metric

Value

CWEs Scanned

74

True Positives

24,345

False Positives

11,702

TP Rate (Precision)

67.5%

Per-file Detection Rate

40.8%

100% Precision CWEs

34

FP Reduction from Baseline

-98.6%

SqC achieves 100% precision (zero false positives) on 34 CWEs including:

  • CWE-78 (OS command injection)

  • CWE-416 (Use after free)

  • CWE-481 (Assigning instead of comparing)

  • CWE-467 (sizeof on pointer type)

  • CWE-252 (Unchecked return value)

  • CWE-338 (Weak PRNG)

  • CWE-590 (Free memory not on heap)

  • CWE-761 (Free not at start of buffer)

  • CWE-690 (NULL dereference from return)

  • CWE-789 (Uncontrolled memory allocation)

High-precision (>80% TP rate) on an additional 7 CWEs including CWE-127 (82.4%), CWE-401 (77.6%), CWE-272 (79.9%), and CWE-675 (93.0%).

See JULIET_RESULTS.md for full per-CWE breakdowns.

FP Reduction History

Over 30+ rounds of targeted optimization, SqC has reduced false positives by 98.6% from baseline while improving the TP rate from 41.1% to 67.5%:

Round

Key Changes

FP

TP Rate

FP Delta

Baseline

Initial implementation

839,341

41.1%

–

Round 3

Standard function database

537,589

42.8%

-198,974

Round 6

Cross-file analysis (-d)

327,191

43.1%

-148,622

Round 9

Windows API whitelist

243,849

43.8%

-52,566

Round 12

CFG + inter-procedural analysis

215,671

44.5%

-28,178

v0.2.23

Built-in C limit macros + const_eval

163,585

44.6%

-12,088

v0.3.37

Fast mode, taint tracking

9,067

48.4%

–

v0.3.119

74 CWEs (6 new), precision improvements

11,702

67.5%

+2,635

Note: v0.3.37 and later use fast mode (CWE-matched rules only); earlier rounds used full-suite scoring, so absolute FP counts are not directly comparable across the two methodologies. TP rate is the consistent metric. The FP increase from v0.3.37 to v0.3.119 reflects expanded CWE scope (68 → 74 CWEs) and more test files, not regression — TP rate improved 19.1 percentage points over the same span.

Real-World Code Analysis

SqC is benchmarked against 5 real-world open-source C codebases alongside cppcheck and clang-tidy:

Project

C Files

LOC

sqc

cppcheck

clang-tidy

libcrc

16

2,130

734

43

2

mosquitto

384

88,717

29,824

747

44

curl

697

240,412

63,207

519

114

sqlite

310

402,321

129,035

1,181

135

hostap

505

541,441

179,833

2,118

2,279

Total

1,912

1,275,021

402,633

4,608

2,574

Data from sqc v0.3.5, cppcheck 2.10, clang-tidy 21.1.6.

Why sqc reports more violations: SqC implements 285 CERT C rules (both advisory and mandatory) while cppcheck and clang-tidy implement ~20 checks each. The difference reflects rule coverage breadth, not false positive rate.

Trend: SqC violations on real-world code have decreased steadily from 548,027 (v0.2.7) to 402,633 (v0.3.5) — a 26% reduction through targeted FP reduction, cross-file analysis, and improved type inference.

See REALWORLD_RESULTS.md for full version history and per-rule breakdowns.

Cross-Tool Comparison Methodology

Apples-to-Apples Concerns

  1. Rule coverage: cppcheck/clang-tidy implement ~20 checks each vs. sqc’s 283 rules. Raw violation counts are not directly comparable.

  2. Translation unit scope: Use consistent scope (cross-file -d flag or single-file) when comparing.

  3. Preprocessor handling: cppcheck evaluates all #ifdef configs; clang-tidy sees one; sqc analyzes all visible branches. For Juliet, compile with -DOMITBAD/-DOMITGOOD when needed.

  4. Standard library awareness: cppcheck/clang-tidy have built-in stdlib knowledge. sqc uses std_functions.rs database.

  5. Severity mapping: cppcheck error/warning/style, clang-tidy error/warning, sqc Low/Medium/High/Critical. Map conservatively.

Published CERT-C Results

No published CERT-C violation rates per KLOC on production open-source code exist (Goseva2015). Valid comparison strategies:

  1. sqc vs. cppcheck vs. clang-tidy on same codebase (done for 5 projects)

  2. sqc on JasPer with reference to SEI SCALe 2015 report (only named CERT-C audit)

  3. sqc TP rate vs. TrustInSoft’s synthetic CERT-C benchmark as upper bound

For academic context on tool effectiveness, FP rates, and the Juliet benchmark methodology, see Bibliography.

Test Infrastructure Details

Build-Time Test Generation

  1. Test files: .c files in src/rules/cert_c/CATEGORY/RULE-ID/tests/{fail,pass}/

  2. Build-time generation: build.rs walks the test directories and generates Rust test functions in $OUT_DIR/integration_tests.rs

  3. Test harness: src/rules/cert_c/integration.rs includes the generated tests, records results, and produces docs/test-summary.md

  4. Test logic:

    • fail/ tests: parse the C file, run the rule, assert violations > 0

    • pass/ tests: parse the C file, run the rule, assert violations == 0

  5. Disabled rules: if RULE-ID.toml has enabled = false, tests are generated with #[ignore]

Test File Naming Conventions

Prefix

Origin

Count

Description

wiki_*

CERT wiki examples

~1,120

Directly from CERT C Coding Standard

testcases_*

AI-generated

~1,860

Broader pattern coverage

Other

Mixed

~80

Various

Test Distribution by Rule Size

Test Count Range

Rules

Examples

1–2 tests

3

Remaining sparse rules

3–5 tests

167

Most wiki-sourced rules

6–10 tests

70

DCL06-C, ENV31-C, INT36-C, etc.

11–20 tests

12

INT31-C, DCL37-C, EXP43-C, etc.

21–50 tests

30

Most “large suite” rules

51–100 tests

8

ARR30-C, STR31-C, INT32-C, MEM31-C, etc.

What Tests Do NOT Cover

  • Inter-procedural analysis: No tests exercise -d directory scanning, prescan, or cross-file function resolution

  • Project context: No tests exercise set_project_context() or set_function_cfgs()

  • CFG/dataflow: The CFG builder, null state analysis, value-range analysis, and init state analysis have embedded Rust unit tests but no integration-level C test coverage

  • CLI flags: No tests for --diff, --export, --format, --include-path, --save-prescan, --load-prescan, --jobs

  • Suppression: No tests for .sqc-suppress.toml hash-based suppression

Coverage Gate

Line coverage is enforced at 75% via scripts/coverage-gate.sh, shared by the pre-commit hook and GitHub Actions CI pipeline. The script:

  • Runs tests via cargo llvm-cov

  • Produces lcov.info (publishable as CI artifact)

  • Excludes from threshold: ui/ (GUI), main.rs (CLI entry), integration.rs (test harness), progress.rs (terminal I/O), export/ (SARIF/Excel output), files/ (git/directory I/O), manifest/ (TOML config loading)

  • Fails with clear output showing current coverage and largest uncovered files

Embedded Rust Unit Tests

Files in src/analyze/ with #[cfg(test)] modules:

File

Lines

Tests

prescan.rs

2,741

31

const_eval.rs

2,071

43

value_range.rs

1,778

13

init_state.rs

1,729

6

null_state.rs

1,720

9

function_summary.rs

1,175

14

suppression.rs

1,070

34

dataflow.rs

988

19

cfg.rs

761

7

mod.rs

705

10

context.rs

93

0

Rule implementation files with embedded tests (against project convention): INT34-C, INT33-C, CON31-C, FIO01-C, EXP32-C, EXP30-C, EXP33-C, EXP08-C, EXP42-C, DCL08-C, STR10-C.

Known Rule Implementation Gaps

The following rule-level analysis limitations were discovered during test coverage work. These are cases where valid C patterns should pass/fail but the rule implementation cannot detect them correctly.

  • INT00-C: find_type_in_source() only matches TYPE VAR; or TYPE VAR,, not TYPE VAR = expr;. Variables with initializers get type “unknown”, so format specifier checks cannot validate %ld with long x = 42;.

  • INT08-C: Rule does not recognize SHRT_MAX / CHAR_MAX guard checks before narrow-type arithmetic.

  • INT34-C: is_likely_unsigned() parameter declaration check doesn’t traverse tree-sitter’s function parameter hierarchy. Also, checks_shift_bounds() doesn’t handle reversed comparison form N <= var (only var >= N).

  • POS50-C: is_declared_in_function() doesn’t distinguish static from automatic storage. Static locals passed to pthread_create() produce FPs.

  • FLP00-C: Only detects float equality in if-conditions, not in return statements or assignments.

  • EXP40-C: is_const_qualified() returns false for identifiers — cannot determine if a variable was declared const without a symbol table.

  • STR03-C: strncpy() and snprintf() always trigger violations regardless of whether null-termination is manually added afterward.