DETECTION-ACCURACY BENCHMARK

Benchmark Results

10,898 self-generated + 528 third-party test cases across 13 modules, 37.9 minutes wall time. Generated 7/27/2026, 10:01:07 PM.

New here? See what QShield does and why this benchmark exists →

ACCURACY BY MODULE

Detection accuracy

Module	Corpus	Claim type	N	Precision	Recall	F1	Wall time
AWS Discovery	Self-generated	classification	2,136	100.0% (±0.0%)	100.0% (±0.0%)	1.000	98.9s	Details →
Certificates	Self-generated	classification	2,136	100.0% (±0.0%)	100.0% (±0.0%)	1.000	151.7s	Details →
Source Code	Self-generated	classification	770	100.0% (±0.0%)	100.0% (±0.0%)	1.000	6.1s	Details →
Dependencies	Self-generated	enumeration	770	—	100.0% (±0.0%)	—	3.2s	Details →
Network Services	Self-generated	classification	770	100.0% (±0.0%)	96.2% (±1.3%)	0.981	52.7s	Details →
Containers	Self-generated	classification	44	100.0% (±0.0%)	100.0% (±0.0%)	1.000	1442.7s	Details →
External Corpus Validation (CodeQL)	Third-party	classification	23	100.0% (±0.0%)	100.0% (±0.0%)	1.000	4.2s	Details →
External Corpus Validation (PyJWT)	Third-party	classification	4	100.0% (±0.0%)	100.0% (±0.0%)	1.000	3.8s	Details →
Package Signing	Self-generated	classification	1,068	100.0% (±0.0%)	100.0% (±0.0%)	1.000	40.0s	Details →
VPN Appliances	Self-generated	classification	1,068	100.0% (±0.0%)	100.0% (±0.0%)	1.000	1.6s	Details →
Backup Systems	Self-generated	classification	1,068	100.0% (±0.0%)	100.0% (±0.0%)	1.000	0.0s	Details →
Firmware	Self-generated	classification	1,068	100.0% (±0.0%)	100.0% (±0.0%)	1.000	464.3s	Details →
External Corpus Validation (OWASP Benchmark)	Third-party	classification	501	0.0% (±100.0%)	0.0% (±100.0%)	0.000	4.6s	Details →

KNOWN CRYPTOGRAPHIC FINDINGS

Findings surfaced by this run

[
  {
    "module": "network_services",
    "kind": "false_negative",
    "count": 29,
    "affected_categories": [
      "DSA-2048"
    ],
    "examples": [
      {
        "case_id": "network_services:DSA-2048:00714",
        "outcome": "FN",
        "detail": {
          "family": "DSA-2048",
          "observed_status": "not_reported",
          "observed_tls_version": null,
          "observed_cipher_status": null
        }
      },
      {
        "case_id": "network_services:DSA-2048:00715",
        "outcome": "FN",
        "detail": {
          "family": "DSA-2048",
          "observed_status": "not_reported",
          "observed_tls_version": null,
          "observed_cipher_status": null
        }
      },
      {
        "case_id": "network_services:DSA-2048:00716",
        "outcome": "FN",
        "detail": {
          "family": "DSA-2048",
          "observed_status": "not_reported",
          "observed_tls_version": null,
          "observed_cipher_status": null
        }
      },
      {
        "case_id": "network_services:DSA-2048:00717",
        "outcome": "FN",
        "detail": {
          "family": "DSA-2048",
          "observed_status": "not_reported",
          "observed_tls_version": null,
          "observed_cipher_status": null
        }
      },
      {
        "case_id": "network_services:DSA-2048:00718",
        "outcome": "FN",
        "detail": {
          "family": "DSA-2048",
          "observed_status": "not_reported",
          "observed_tls_version": null,
          "observed_cipher_status": null
        }
      }
    ]
  }
]

COMPATIBILITY

Compatibility notes

aws_discovery exercised only against Moto in this run; real-AWS parity is a separate, credentials-gated run (see CODEX_IMPLEMENTATION_SPEC.md section 36.3) and is not included here.
source_code: semgrep available: True
dependencies: syft available: True
containers: trivy available: True

TEST ENVIRONMENT

Environment & reproducibility

Python3.13.14

PlatformLinux-6.8.0-100-generic-aarch64-with-glibc2.41

Git commitunknown (not a git checkout in this container)

boto31.42.0

moto5.1.17

cryptography46.0.0

fastapi0.128.0

sqlalchemy2.0.50

Corpus composition (algorithm family counts, package lists, file templates) is fixed and versioned in benchmark/corpus/*.py. Underlying asymmetric key material (RSA/EC/DSA/Ed25519/Ed448) is freshly generated each run via the OS CSPRNG, not derived from a fixed seed -- practical deterministic keygen isn't offered by the cryptography library across backends/versions. This does not affect the statistical result, which depends on algorithm family, not exact key bytes.

make benchmark  # docker compose run --rm benchmark python -m benchmark.run_all