Testing and CI¶
How the driver is checked: host tests against an independent model of the sensor for every part, fault injection, a fuzzer that corrupts what the sensor sends back, compiled documentation, compile-fail tests, an install test, ESP32 and Cortex-M builds, static analysis and measured quality gates, on every pull request.
No hardware yet
Everything on this page runs against a model of the sensor. The model is written from the same documents as the driver, so a misreading shows up in both. What is not verified lists what only a bench can settle.
Host tests¶
The tests build the real driver against a model of the sensor, with the firmware's flags, -fno-exceptions -fno-rtti
-Wswitch-enum -Wconversion -Wsign-conversion, and warnings as errors, on GCC and Clang.
cmake -S . -B build -DSFM_FLOW_METER_BUILD_TESTS=ON
cmake --build build
ctest --test-dir build --output-on-failure
| File | Covers |
|---|---|
tests/test_all_parts.cpp |
Every one of the 32 variants through the whole life cycle: identify, scaling, start, sample, stop, reset |
tests/test_parts.cpp |
The table against hand-typed datasheet expectations, Validate(), presets, CheckedConfig, the CRC vectors of both initial values |
tests/test_sf06.cpp |
SF06: framing, commands, mixtures, concentration updates, averaging, status checks, sleep and wake, a sensor that NACKs between samples (N = 1 and N = 128 stay Measuring, with and without a clock, 20000 tight reads), device loss only after the silence bound, rejected frames and raw reads keep the silence clock running, no escalation without a clock, the stop time after a reported reset, start-up from a sleeping sensor, NACKed polls are not bus errors, the identifier poll option, no empty write, no soft reset in sleep, stuck lines |
tests/test_gen2_dual.cpp |
SFM3304 (filter argument, status layout) and SFM3505 (24-bit frame, two flows) |
tests/test_sf05.cpp |
Legacy pointer commands, EEPROM scale, first-byte rule, restart before read, temperature by pointer, the two-pointer serial number and the SFM3000 whose 0x31AF is refused, oxygen refused on the SFM3200 and SFM4200 and nitrogen at the air scale where listed, getters cleared after a failed call, the error limit leading to StateUnknown and a caller's HardReset() |
tests/test_sf04.cpp |
SFM4100: trigger bytes, stretching read, one gas per address |
tests/test_sfx.cpp |
SFM6000D: every [P] fact has a test that flips it |
tests/test_states.cpp |
The state machine: every call in every state, Busy, StateUnknown, Poll() |
tests/test_errors.cpp |
The base Error values are 0 to 9; every error value (16 and up) is returned by some call |
tests/test_kconfig.cpp |
KconfigDefaults() with and without CONFIG_ values |
tests/test_version.cpp |
The version getters follow the generated header (also against a 9.8.7 probe) |
tests/fuzz/ |
The fuzz target; CTest replays its corpus, see below |
tests/compile_fail/ |
Eight pieces of code that must not compile, each for its EXPECT: diagnostic: a class that is not a port, a hook without noexcept, a port without DelayUs() in a blocking call, an invalid CheckedConfig, an SFM4100 object without ReadStretch(), a PowerCycle() returning void, a bad family mask, an enumeration whose last-enumerator trait is missing |
tests/docs/ |
Every C++ block of the documentation, compiled (target sfm_flow_meter_doc_snippets) |
examples/host/ |
The host example, built with the test flags and run (sfm_flow_meter_host_example) |
tests/check_install.cmake |
Installs into a scratch prefix and builds tests/install_consumer/ with find_package(sfm_flow_meter) |
tests/check_doxygen.cmake |
Doxygen with _config/Doxyfile and warnings as errors, when doxygen is installed |
The model¶
tests/model_sfm.hpp is a sensor written as a behaviour, not as a copy of the driver: a
Chip that takes a transaction at the bus, decodes it as the datasheet describes, enforces the timing (the first
sample after a start, the reset and wake times, the half millisecond of the SF05 restart), and keeps a log of every
transaction. It can NACK reads between samples (no_new_sample, one result every N x 0.5 ms), counts empty writes and
commands the datasheet forbids in the mode (a soft reset in sleep), and accepts a wake on any write that reaches its
address. It keeps its own clock of the last frame it delivered (last_served_us) and can refuse the second serial
pointer (no_serial_low). tests/datasheet_expect.hpp holds the expectations of
each variant typed by hand from the documents (address, scale, offset, span, timing, product identifier), so the table
in the driver is compared with a second copy and not with itself.
The model is independent in the way that counts for a typo: it does not include the driver's register or type headers for its constants. It is not independent of the documents. Part of its typing was derived from the table and the design notes, so a misreading of a document appears in both. That is the honest limit of testing without a bench, and every ambiguity that rests on it has a knob (Contract).
Faults the model injects, one knob each: a failed write or read in turn (the nth transfer), a flipped byte of a read, a
stuck line (all zeros or all ones), a bad status nibble or status XOR, a wrong scale, unit, product identifier or
serial, a sensor that ignores a stop, a sensor that is gone (dead), a strict-restart mode, and a bus address
different from the configured one.
Fuzz tests¶
Everything the driver reads comes from the sensor over the bus, so the data read is what a noisy cable, a wrong part or
a broken sensor controls. tests/fuzz/fuzz_sfm_flow_meter.cpp lets libFuzzer
choose a part, a configuration and a sequence of calls; between the calls it changes the sensor: fails a transfer,
flips a bit, sticks the line, reports another product identifier or status, resets the sensor under the driver,
changes the scale it reports and advances the clock. After every call it checks 17 invariants and aborts, naming the
line, when one breaks:
- a failed call returns an
Errorthe enumeration defines; - a failed
ReadSample()returns a zeroSample; - a call that returns
NotInitialized,Busy,InvalidArgumentorUnsupportedsent nothing (StateUnknowncan follow traffic: the SF05 error limit); - a failed call leaves nothing pending;
- a returned wait is at most one second;
- every read has a whole number of three-byte words and at most 18 bytes;
- a returned flow lies inside the span plus the margin unless
RangePolicy::Flagflagged it; ReadSample()succeeds only whileMeasuring;Measuringis entered only by a successfulStartMeasurement()that finished;- a successful reset leaves the driver
Uninitialized; - the counters never decrease;
has_flow2,has_statusandhas_temperaturefollow the part's capabilities;StateUnknownis left only by an initialisation (toIdle, orUninitializedif it failed) or a reset;- a read leaves
Measuringonly withDeviceResetorStateUnknown, never for "no new sample"; - no write is ever empty and no command is sent that the datasheet forbids in the current mode;
DeviceResetfollows real silence: the model, with its own clock of the last frame it delivered (rejected frames included), has been quiet for at least the whole bound (first-result time, N x 0.5 ms with 10 %, 1 ms);- a temperature the getter returns lies in the plausible band, -60 to +150 degC: a stuck bus is never data.
Invariant 9 is asserted in both directions: a successful start ends in Measuring, and no other call enters it.
# libFuzzer needs Clang (on macOS, Homebrew's LLVM)
cmake -S . -B build-fuzz -DCMAKE_CXX_COMPILER=clang++ -DSFM_FLOW_METER_BUILD_TESTS=ON -DSFM_FLOW_METER_BUILD_FUZZERS=ON
cmake --build build-fuzz --target fuzz_sfm_flow_meter
./build-fuzz/tests/fuzz/fuzz_sfm_flow_meter -max_total_time=120 tests/fuzz/corpus/fuzz_sfm_flow_meter
The same file also builds without libFuzzer as a replay, and CTest replays every input in
tests/fuzz/corpus/fuzz_sfm_flow_meter/ on GCC and Clang: the seeds
(one or more per family), and every input that ever broke a promise (name it regress-*), so a fixed bug stays fixed.
Documentation snippets¶
Every C++ block on this site is included from tests/docs/, which the host build
compiles with the driver instantiated over the documented ports. A change to the API that breaks a page breaks the
build first. The README's example is a generated copy of the same file; kokeb-docs check fails when it drifts.
CI workflows¶
Everything runs from ci.yml, which calls one workflow file per check and ends in the
CI gate job, the single required check (CI gate / gate).
| Job | What it checks |
|---|---|
| Host tests | GCC, then Clang (one job), build and run every test above, then build and run examples/host on its own |
| Host quality | The gates below: complexity, coverage, sanitizers, fuzzing |
| Cross-compile | arm-none-eabi-g++ for Cortex-M0+, M4F and M7F with the strict flags, warnings as errors (nothing linked) |
| C++ lint | clang-format and clang-tidy over the sources |
| MISRA | MISRA C++:2023 and the C++20 profile on traced host and Cortex-M builds, at level high (see below) |
| Docs | Strict site build (_config/mkdocs.yml), figure and generated-region drift, Doxygen (_config/Doxyfile), offline link check, Lighthouse |
| Repository checks | DCO on pull requests, the org standard, Markdown and YAML lint, the Kokeb Labs handbook rules for .kokeb/handbook.yml, template drift and cppcheck over the driver sources; a second job runs the handbook's source checks |
| CI gate | Passes only if every job above passed: the one required check |
The ESP32 examples job builds every app in examples/esp32/app_config.yml, for each listed ESP-IDF version and
build type, with the repository root as the driver's component, and lints the example sources. The gate needs it too.
The workflows call the shared, versioned workflows of kokeb-labs/ci-tools
at @v1.
On hardware¶
The examples (Examples) are the bench test: each logs a flow twice a second. Blow through the sensor and watch the value follow. Record the board, the part and the result in the pull request's bench-status section. Until that is done, the claim of this driver is "verified against a model".
Quality gates¶
The Host quality job measures the library code, inc/ and src/ only (tests and examples are left out), and
fails when a value is over the limits of this repository's assurance level, high (set in
.kokeb/handbook.yml, defined by the handbook's ADR-0020):
| Gate | Tool and how it is measured | Limit at high |
|---|---|---|
| Complexity | lizard 1.24.1, modified cyclomatic complexity (-m: a switch counts once) per function |
CCN at most 15 |
| Function length | lizard, lines of code per function | at most 60 |
| Parameters | lizard, per function | at most 6 |
| Line coverage | GCC 14 --coverage -O0, every CTest, gcovr 8.7 --merge-lines (a template's lines count once across its instances), --exclude-unreachable-branches --exclude-throw-branches |
at least 85 % |
| Branch coverage | same run | at least 75 % |
| MC/DC | GCC -fcondition-coverage, same run |
measured, not gated at high (100 % at safety) |
| Sanitizers | Clang, -fsanitize=address,undefined -fno-sanitize-recover=all, every CTest |
no report |
| Fuzzing | libFuzzer with ASan and UBSan, 120 s per target in CI (60 s in the local rehearsal) | no broken promise |
Last container run (GCC 14, driver sources, gcovr 8.7 with --merge-lines): lines 1261 of 1293 (97 %), branches
760 of 888 (85 %, above the 75 % floor), conditions (MC/DC, -fcondition-coverage) 625 of 716 (87 %), decisions
419 of 456 (91 %). The sanitizer and fuzz gates (60 s) passed. The gate script prints the line total only; the other
figures come from gcovr with --txt-metric branch, condition and decision on the same run, per file in the run
log kept with the review records. --merge-lines does not change the decision total (the same with and without it,
and with or without the unreachable and throw exclusions). MC/DC is measured, not gated, at the high level: 87 % is
not the 100 % the safety level asks for.
The reports are in the run's host-quality-reports artifact and its step summary. To run the gates locally:
# Coverage (GCC 14 or later), then the complexity limits
cmake -S . -B build-cov -DCMAKE_CXX_COMPILER=g++-14 -DSFM_FLOW_METER_BUILD_TESTS=ON \
-DCMAKE_CXX_FLAGS="--coverage -O0 -fcondition-coverage"
cmake --build build-cov && ctest --test-dir build-cov
gcovr --root . --filter inc/ --filter src/ --merge-lines --exclude-unreachable-branches --txt build-cov
lizard -m -C 15 -L 60 -a 6 inc src
# Sanitizers (Clang)
cmake -S . -B build-san -DCMAKE_CXX_COMPILER=clang++ -DSFM_FLOW_METER_BUILD_TESTS=ON \
-DCMAKE_CXX_FLAGS="-fsanitize=address,undefined -fno-sanitize-recover=all -O1 -g"
cmake --build build-san && ctest --test-dir build-san --output-on-failure
The fuzz gate runs locally as shown in Fuzz tests.
clang-tidy 20 (_config/.clang-tidy: bugprone, CERT, clang-analyzer,
cppcoreguidelines, modernize, performance, portability and readability, each disabled check with its reason) and
cppcheck (strict) run in the C++ lint and Repository checks jobs and fail on any finding.
The MISRA job (ci-misra.yml) runs the shared ru-misra workflow of
ci-tools at level high: it traces a host build and the Cortex-M build (sfm_flow_meter_optimised_build) of every
translation unit, checks them against MISRA C++:2023 and the handbook's C++20 profile (the CodeQL coding-standards
and profile packs, plus the validated clang-tidy and cppcheck checks), and reports each finding against the
handbook's compliance matrix and deviation records. A finding is fixed in the code; a deliberate exception names
its deviation on the line before it (// codeql::misra_deviation_next_line(MD-CPP-009) above the .ipp include
of inc/sfm_flow_meter.hpp). Until the repository is public the job runs with open_source_prerelease: true, the
declaration that the code is released under the licence in LICENSE and is to be published.