Testing and CI¶
How the driver is checked: host tests against a model of the chip, a fuzzer that corrupts what the chip sends back, compiled documentation, an install test, ESP32 and Cortex-M builds, static analysis and measured quality gates, on every pull request.
Host tests¶
The tests build the real driver against a protocol model of the chip (tests/mock_at25040b.hpp). The model is written
from the datasheet on its own: it includes none of the driver's headers and carries its own opcodes, status bits, array
sizes and Table 6-4, each with its datasheet page, and tests/test_model.cpp cross-checks the two maps with
static_assert (handbook KB-TST-030). It models the opcodes with
A8 on the AT25040B, invalid opcodes that leave SO in high impedance, the write enable latch (set by WREN only with WP
high, cleared by WRDI whatever WP does and at the end of every write cycle), sequential READ with roll-over, WRITE
that wraps inside its page, a self-timed write cycle in simulated time during which only RDSR works and reads 0xFF,
block protection that refuses writes into the protected top and survives a power cycle. The faults are the ones
a board can produce or the datasheet names: MISO floating high, stuck low, echoing MOSI or random; HOLD strapped to CS;
WP tied low; a WRITE the chip silently ignores; a cell that does not take its value; a write cycle that never ends;
failed, lost and replaced transfers; a failing WP pin. They build with the firmware's flags,
-fno-exceptions -fno-rtti, -pedantic-errors and warnings as errors, as ISO C++20 (CMAKE_CXX_EXTENSIONS OFF), on
GCC and Clang, with the standard library's own bounds checks on (_GLIBCXX_ASSERTIONS, libc++ hardening).
at25040b_host_tests runs 74 test cases; with the Kconfig, version, install, Doxygen, host-example, compile-fail and
fuzz-corpus tests CTest runs 16 tests. The target at25040b_header_checks compiles each public header alone as
-std=c++20 -pedantic-errors and asserts the dialect; the Cortex-M cross-build compiles the same units, and the ESP32
example compiles at25040b_cxx20_check.cpp the same way inside ESP-IDF, which otherwise builds in a newer GNU dialect.
cmake -S . -B build -DAT25040B_BUILD_TESTS=ON
cmake --build build
ctest --test-dir build --output-on-failure
| File | Covers |
|---|---|
tests/test_at25040b.cpp |
The lifecycle (no I/O before the start, a retried start, re-probe), Validate(), the identity check against every bus fault, reads (chunks, A8, verification, waiting out a write cycle, a glitched status), writes (page split, a 512-byte whole-array write, WP, block protection, an ignored WRITE, a stuck cell, a cycle that never ends, a failing WP pin, the smaller parts), the status and block protection, logging |
tests/test_bus_faults.cpp |
Every transfer and every WP pin write of every operation, EnsureInitialized() included, fails in turn and each returns Bus (a transfer) or Pin (a pin write) with WP parked; every transfer of a write is lost in turn and none yields Ok without the data |
tests/test_model.cpp |
The model's map against the driver's, at compile time: opcodes, status bits, sizes, Table 6-4 for every part and level |
tests/fuzz/ |
The fuzz target; CTest replays its corpus (at25040b_fuzz_at25040b_corpus), see below |
tests/test_errors.cpp |
Every chip-specific value (16 to 21) is returned by a real call |
tests/test_version.cpp |
The version getters follow the generated header (also against a 9.8.7 probe) |
tests/test_kconfig.cpp |
KconfigDefaults() with and without CONFIG_ values |
tests/compile_fail/ |
Code that must not compile, each for its EXPECT: diagnostic: a port without DelayUs(), a void GpioSet(), a class that is not a port, a Transfer() with a narrower length or without noexcept, a DelayUs(uint16_t) |
tests/mock_at25040b.hpp |
The chip model; the mock ports over it are in mock_at25040b_bus.hpp |
tests/check_version_copy.cmake |
_config/Doxyfile carries the CMake version |
examples/host/ |
The host example, built with the test flags and run (at25040b_host_example) |
tests/check_install.cmake |
Installs into a scratch prefix and builds tests/install_consumer/ with find_package(at25040b) |
tests/check_doxygen.cmake |
Doxygen with _config/Doxyfile and warnings as errors, when doxygen is installed |
The model does not reproduce the chip's electrical behaviour, its real write cycle time (it uses 3 ms) or what a partly clocked WRITE does: see Status and limitations.
Fuzz tests¶
Everything the driver receives comes from MISO, so MISO is what a noisy cable, an unplugged module or a broken chip
controls. tests/fuzz/fuzz_at25040b.cpp lets libFuzzer choose a configuration,
then up to 64 driver calls, and on every transfer what the bus does: pass the model's reply, fail the transfer, lose
it (the chip never sees it, MISO floats), flip bits of the reply or replace it; on every WP pin write, pass or fail
it. Between calls it changes the board: MISO floating, stuck low, echoing or noisy, HOLD strapped to CS, WP tied
low, an ignored WRITE, a stuck cell, a slow or running write cycle, the block protection, a power cycle. A transfer
the bus rather than the chip answered counts as tampered. After every call the target checks the driver's promises
and aborts, naming the line, when one breaks:
- a failed call returns an
Errorthe enum defines (never one documented as not returned) andT{}; - before
EnsureInitialized()no call makes a transfer or a pin write, andInvalidArgumentmeans nothing was sent; - nothing is ever written outside the requested range; only
Write()changes the array and onlySetBlockProtection()changes BP1:BP0; - a call never returns
Okfor bytes the chip does not hold: anOkfromRead()orWrite()matches the chip unless two transfers of the call were tampered (one, withVerify::None); the input choosesVerify::NoneorVerify::ReadBackfor the reads and writes; - with
check_write_enable,EnsureInitialized()never returnsOkon a dead bus or with WP tied low unless a transfer was tampered; - WP is always parked after a call, unless the last pin write failed;
- an
Okcall leaves the write enable latch clear if it found it clear, unless a transfer was tampered.
The fuzzer found two bugs before the first commit, both fixed with a unit test and a regress-* input each:
- With the write enable latch already set (a WREN before an MCU reset) and WP held low, the identity check's
"WREN, then WEN = 1" passed although WP had blocked the WREN. The driver now clears the latch first
(
regress-wen-preset-wp-low). - One status read that looked idle (a lost transfer reading 0x00) while a write cycle ran let
Read()go ahead; the busy chip ignored both READs, both copies read 0xFF, andRead()returnedOkwith wrong bytes. An idle status now counts only when a second read confirms it (regress-idle-glitch-during-cycle).
# libFuzzer needs Clang (on macOS, Homebrew's LLVM)
cmake -S . -B build-fuzz -DCMAKE_CXX_COMPILER=clang++ -DAT25040B_BUILD_TESTS=ON -DAT25040B_BUILD_FUZZERS=ON
cmake --build build-fuzz --target fuzz_at25040b
./build-fuzz/tests/fuzz/fuzz_at25040b -max_total_time=300 tests/fuzz/corpus/fuzz_at25040b
The same file also builds without libFuzzer as fuzz_replay_fuzz_at25040b, and CTest replays every input in
tests/fuzz/corpus/fuzz_at25040b/ on GCC and Clang: the seeds, and every input
that ever broke a promise (regress-*), so a fixed bug stays fixed. CI fuzzes for 120 s on every pull request
(job Host quality / fuzz). After the fixes, ten minutes of local fuzzing (9.6 million runs, 451 coverage edges)
broke no promise; after the move to spans, the Pin error and ReadRawStatus(), 200 s more (2.28 million runs,
478 edges) broke none.
Quality gates¶
The Host quality job measures the library code, inc/ and src/ only (tests and examples are left out), and
fails when a value is over the limits of this repository's assurance level, high (set in
.kokeb/handbook.yml, defined by the handbook's ADR-0020):
| Gate | Tool and how it is measured | Limit at high |
This driver (local) |
|---|---|---|---|
| Complexity | lizard 1.24.1, modified cyclomatic complexity (-m: a switch counts once) per function |
CCN at most 15 | highest 11 (waitIdle()) |
| Function length | lizard, lines of code per function | at most 60 | within the limit |
| Parameters | lizard, per function | at most 6 | within the limit |
| Line coverage | GCC 14 --coverage -O0, every CTest, gcovr 8.7 --merge-lines (a template's lines count once across its instances), --exclude-unreachable-branches --exclude-throw-branches |
at least 85 % | 99.5 % |
| Branch coverage | same run | at least 75 % | 98.4 % |
| MC/DC | GCC -fcondition-coverage, same run |
measured, not gated at high (100 % at safety) |
99.2 % |
| Sanitizers | Clang, -fsanitize=address,undefined -fno-sanitize-recover=all, every CTest |
no report | clean in CI |
| Fuzzing | libFuzzer with ASan and UBSan, 120 s per target | no broken promise | 2.28 million runs in 200 s locally, clean |
The local figures are GCC 15 and gcovr 8.7 with the CI's options; CI uses GCC 14. The reports are in the run's
host-quality-reports artifact and its step summary. To run the gates locally:
# Coverage (GCC 14 or later), then the complexity limits
cmake -S . -B build-cov -DCMAKE_CXX_COMPILER=g++-14 -DAT25040B_BUILD_TESTS=ON \
-DCMAKE_CXX_FLAGS="--coverage -O0 -fcondition-coverage"
cmake --build build-cov && ctest --test-dir build-cov
gcovr --root . --filter inc/ --filter src/ --merge-lines --exclude-unreachable-branches --txt build-cov
lizard -m -C 15 -L 60 -a 6 inc/*.hpp src/*.ipp # list the files: lizard skips .ipp in a directory
# Sanitizers (Clang)
cmake -S . -B build-san -DCMAKE_CXX_COMPILER=clang++ -DAT25040B_BUILD_TESTS=ON \
-DCMAKE_CXX_FLAGS="-fsanitize=address,undefined -fno-sanitize-recover=all -O1 -g"
cmake --build build-san && ctest --test-dir build-san --output-on-failure
The fuzz gate runs locally as shown in Fuzz tests.
clang-tidy 20 (_config/.clang-tidy: bugprone, CERT, clang-analyzer,
cppcoreguidelines, modernize, performance, portability and readability, each disabled check with its reason) and
cppcheck (strict) run in the C++ lint and Repository checks jobs and fail on any finding.
Documentation snippets¶
Every C++ block on this site is included from tests/docs/, which the host build
compiles (target at25040b_doc_snippets) with the driver instantiated over the documented port. A change to
the API that breaks a page breaks the build first. The README's example is a generated copy of the same file;
kokeb-docs check fails when it drifts.
CI workflows¶
Everything runs from ci.yml, which calls one workflow file per check and ends in
the CI gate job, the single required check (CI gate / gate).
| Job | What it checks |
|---|---|
| Host tests | GCC, then Clang (one job), build and run every test above, then build and run examples/host on its own |
| Host quality | The gates above: complexity, coverage, sanitizers, fuzzing |
| Cross-compile | arm-none-eabi-g++ for Cortex-M0+, M4F and M7F with the strict flags, warnings as errors (nothing linked) |
| C++ lint | clang-format and clang-tidy over the sources |
| Docs | Strict site build (_config/mkdocs.yml), figure and generated-region drift, Doxygen (_config/Doxyfile), offline link check, Lighthouse |
| Repository checks | One job: DCO on pull requests, the org standard, Markdown and YAML lint, the Kokeb Labs handbook rules for .kokeb/handbook.yml (by rule ID), template drift (warns only) and cppcheck over the driver sources |
| CI gate | Passes only if every job above passed: the one required check |
The ESP32 examples job builds every app in examples/esp32/app_config.yml, for each listed ESP-IDF version and
build type, with the repository root as the driver's component, and lints the example sources. The gate needs it
too.
The workflows call the shared, versioned workflows of
kokeb-labs/ci-tools at @v1.
On hardware¶
The driver is not verified on hardware yet. The ESP32 example (Examples) is a first bench test: it checks the chip, reads the first bytes, counts boots in the last page and probes once a second; unplug the module and plug it back while it runs. The checks that count, behind your handler on your board, are listed in Status and limitations. Record the board, the driver commit and the result there.