Testing and CI¶
How the driver is checked: host tests against an independent model of the instrument and its serial wire, fault injection, four fuzz targets with checked invariants, compiled documentation, an install test, ESP32 and Cortex-M builds, static analysis and measured quality gates, on every pull request.
Host tests¶
The tests build the real driver against a model of a BASIS 2 instrument. The model
(tests/mock_basis2.hpp) is written from the manual, not from the driver: it has its own
ASCII grammar with plain strings and its own Modbus register file with plain integers, and shares no encoder,
tokenizer or table with the code under test. A wrong constant in the driver therefore cannot hide behind the same
wrong constant in the model (handbook KB-TST-030). The wire has a virtual clock in microseconds, ten bits per
character at the instrument's baud rate, and the instrument's wait of 3.5 characters of silence before it acts.
The model refuses to hear a reply sent at another speed than the port listens at. Knobs added for the safety tests: K-IDIGNORE (a unit ID change answers but changes nothing) and K-BAUDREJECT (a baud change is refused and the speed stays); the model also gates every Modbus register on the firmware that added it.
Every assumption of the manual is a knob of the model, so a test can show that the driver survives either answer:
column-aligned or single-spaced frames, bracketed or bare flags, an error reply with or without the unit ID, the
format of the firmware text, the reply of the setters, a status flag given as an integer. The tests build with the
firmware's flags, -fno-exceptions -fno-rtti, and warnings as errors, on GCC and Clang.
cmake -S . -B build -DALICAT_BASIS2_BUILD_TESTS=ON
cmake --build build
ctest --test-dir build --output-on-failure
| File | Covers |
|---|---|
tests/test_grammar.cpp |
The request factories, the independent wire validators and the golden strings of every command |
tests/test_decode.cpp |
The reply parsers: frames, DV, gas list, text replies, numbers, flags, malformed and hostile input |
tests/test_api_ascii.cpp |
The typed API over ASCII: start-up, reads, setpoints, ramp, hold, tare, gas, totalizer, scans, recovery |
tests/test_api_modbus.cpp |
The same over Modbus RTU, with the decimals, the firmware gates and the register table |
tests/test_modbus_codec.cpp |
CRC-16, frame building, the legality check, reply lengths and exception frames |
tests/test_wire.cpp |
The transaction machine: the gap, the deadlines, chunked and late replies, quarantine, stray bytes, clock wrap, hold-offs |
tests/test_guarded.cpp |
The six guarded commands, every confirmation outcome, StateUnknown and Recover() |
tests/test_safety_kills.cpp |
Behaviours a mutation run found unguarded: no write repeat, slow-command wait, gas verification, Recover() refusals, the unknown-source rate limit, a switch-off in doubt (unit ID, baud, address, factory restore), the saved-setpoint limit after a lost reply, character and length boundaries, the forged-request refusal |
tests/check_doc_symbols.py |
Every API name a page cites in backticks exists in inc/ or src/ (CTest alicat_basis2_doc_symbols) |
tests/test_errors.cpp |
The base Error values are 0 to 9; every chip-specific value is returned by some call and has a name |
tests/test_version.cpp |
The version getters follow the generated header (also against a 9.8.7 probe) |
tests/test_kconfig.cpp |
KconfigDefaults() with and without CONFIG_ values |
tests/compile_fail/ |
Code that must not compile, each for its EXPECT: diagnostic: a guarded call without its token, a request built by hand, a hook with the wrong signature, a port without NowMs(), a configuration that is invalid in a constant expression |
tests/fuzz/ |
The four fuzz targets; CTest replays their corpus (alicat_basis2_fuzz_*_corpus) |
tests/check_version_copy.cmake |
_config/Doxyfile carries the CMake version |
examples/host/ |
The host example, built with the test flags and run (alicat_basis2_host_example) |
tests/check_install.cmake |
Installs into a scratch prefix and builds tests/install_consumer/ against it |
tests/check_doxygen.cmake |
Doxygen with _config/Doxyfile and warnings as errors, when doxygen is installed |
Fault injection¶
The model's wire injects, per transaction or from a given call on: a dropped reply, a delayed reply, a reply split into chunks, a missing carriage return, extra and prefixed and suffixed bytes, a duplicated reply, a reply with the wrong unit ID, a replaced line, a line feed after the carriage return, a bad CRC, a Modbus frame for the wrong address (with a valid CRC), an error reply, a garbled byte, stray bytes between calls, a power cycle, a clamped setpoint and a port that fails to write, flush or initialise. The tests fail every transfer in turn and check that every call returns the right error, leaves the bus usable, and never sends an unlisted command.
Fuzz tests¶
Everything the driver reads comes from the instrument, so the data read is what a noisy cable, a wrong baud rate or a
second instrument controls. There are four libFuzzer targets in tests/fuzz/, each with
a seed corpus, replayed by CTest on every host build so a regression shows up without a fuzzing run:
| Target | What it feeds |
|---|---|
fuzz_ascii_replies |
Arbitrary bytes, split at fuzz-chosen points, as the reply to each ASCII command |
fuzz_ascii_session |
A random sequence of API calls, guarded ones with their tokens, against the model with random faults |
fuzz_modbus_replies |
Arbitrary bytes as the reply frame to each Modbus request |
fuzz_modbus_session |
The same session against the register model |
A wrapper port records every byte written and every clock read, and after every call the harness checks the
invariants marked "fuzz" below. A violation aborts the run. The others are proven by directed tests
(test_wire.cpp, test_decode.cpp) with a virtual clock: a
reply that arrives late, a clock that wraps, a decode that round-trips.
| ID | Checked by | Invariant |
|---|---|---|
| I1 | fuzz | Every Write() buffer passes the independent validator of the grammar and ends the line correctly |
| I2 | test_decode | A successful decode round-trips: encoding the decoded numbers again gives the numbers parsed |
| I3 | fuzz | Nothing is written while a transaction waits, and the gap to the previous byte is at least 3.5 characters |
| I4 | test_wire | After a failed transaction the next write comes only after the quarantine; a late reply never answers a later command |
| I5 | fuzz | The six guarded texts never appear unless the harness made the matching guarded call |
| I6 | fuzz | No setpoint above 102.5 % of full scale, none negative, no valve hold above 100, no saved setpoint too soon |
| I7 | fuzz | Decoded values stay in their bounds; no read beyond the line buffer (under the sanitizers) |
| I8 | fuzz | A failed call returns a zero-valued result; a frame is exposed only for the unit ID addressed |
| I9 | test_wire, test_bounds, test_busy, fuzz | The machine makes progress or ends within its deadlines: a request whose line is not quiet within one reply wait ends Busy unsent; the fuzz targets bound every call by ten seconds, a scan's polls and a hold-off still running |
| I10 | fuzz | Modbus writes use functions 6 and 16 only, on writable registers, never to address 0, with valid CRC and legal magic values |
| I11 | test_wire, test_clock, fuzz | A clock wrap, a start in the clock's upper half or more than 2^31 ms of quiet never causes an early send, a lost send or an endless wait |
| I12 | fuzz | StateUnknown is left only by a successful Recover(); asserted after every step with IsStateUnknown() |
| I14 | fuzz | While StateUnknown the wire carries only a read request of the grammar, a zero setpoint or hold, or P2 (Modbus: reads, a zero setpoint, register 56 = 0) |
| I13 | fuzz | A command sent into a reply still on the wire happens only in a session that chose a fault (ASCII) |
| I15 | fuzz | On a line declared to hold two instruments the unit ID, address, baud and factory-restore commands never reach the wire |
| I16 | fuzz | SetSetpoint(0.0) puts a frame on the wire unless it returns NotInitialized, Unsupported, FirmwareTooOld, or Busy while bytes were still arriving on the line |
Run a target locally with Clang:
cmake -S . -B build-fuzz -DCMAKE_CXX_COMPILER=clang++ -DALICAT_BASIS2_BUILD_TESTS=ON -DALICAT_BASIS2_BUILD_FUZZERS=ON
cmake --build build-fuzz
build-fuzz/tests/fuzz/fuzz_ascii_session tests/fuzz/corpus/fuzz_ascii_session -max_total_time=60
Quality gates¶
The Host quality job measures the library code, inc/ and src/ only (tests and examples are left out), and
fails when a value is over the limits of this repository's assurance level, high (set in
.kokeb/handbook.yml, defined by the handbook's ADR-0020):
| Gate | Tool and how it is measured | Limit at high |
|---|---|---|
| Complexity | lizard 1.24.1, modified cyclomatic complexity (-m: a switch counts once) per function |
CCN at most 15 |
| Function length | lizard, lines of code per function | at most 60 |
| Parameters | lizard, per function | at most 6 |
| Line coverage | GCC 14 --coverage -O0, every CTest, gcovr 8.7 --merge-lines (a template's lines count once across its instances), --exclude-unreachable-branches --exclude-throw-branches |
at least 85 % |
| Branch coverage | same run | at least 75 % |
| MC/DC | GCC -fcondition-coverage, same run |
measured, not gated at high (100 % at safety) |
| Sanitizers | Clang, -fsanitize=address,undefined -fno-sanitize-recover=all, every CTest |
no report |
| Fuzzing | libFuzzer with ASan and UBSan, 120 s per target | no broken promise |
The reports are in the run's host-quality-reports artifact and its step summary. To run the gates locally:
# Coverage (GCC 14 or later), then the complexity limits
cmake -S . -B build-cov -DCMAKE_CXX_COMPILER=g++-14 -DALICAT_BASIS2_BUILD_TESTS=ON \
-DCMAKE_CXX_FLAGS="--coverage -O0 -fcondition-coverage"
cmake --build build-cov && ctest --test-dir build-cov
gcovr --root . --filter inc/ --filter src/ --merge-lines --exclude-unreachable-branches --txt build-cov
lizard -m -C 15 -L 60 -a 6 inc src
# Sanitizers (Clang)
cmake -S . -B build-san -DCMAKE_CXX_COMPILER=clang++ -DALICAT_BASIS2_BUILD_TESTS=ON \
-DCMAKE_CXX_FLAGS="-fsanitize=address,undefined -fno-sanitize-recover=all -O1 -g"
cmake --build build-san && ctest --test-dir build-san --output-on-failure
The fuzz gate runs locally as shown in Fuzz tests.
clang-tidy 20 (_config/.clang-tidy: bugprone, CERT, clang-analyzer,
cppcoreguidelines, modernize, performance, portability and readability, each disabled check with its reason) and
cppcheck (strict) run in the C++ lint and Repository checks jobs and fail on any finding.
The MISRA job (ci-misra.yml) runs the shared ru-misra workflow of
ci-tools at level high: it traces a host build and the Cortex-M build (alicat_basis2_optimised_build) of every
translation unit, checks them against MISRA C++:2023 and the handbook's C++20 profile (the CodeQL coding-standards
and profile packs, plus the validated clang-tidy and cppcheck checks), and reports each finding against the
handbook's compliance matrix and deviation records. A finding is fixed in the code; a deliberate exception names
its deviation on the line before it (// codeql::misra_deviation_next_line(MD-CPP-009) above the .ipp include
of inc/alicat_basis2.hpp). Until the repository is public the job runs with open_source_prerelease: true, the
declaration that the code is released under the licence in LICENSE and is to be published.
Checks switched off for the tests¶
tests/.clang-tidy inherits the organisation's configuration and turns these checks off, because the code they
would change is test scaffolding that is never shipped. The driver in inc/ and src/ has no suppression and no
relaxation.
| Check | Reason |
|---|---|
readability-make-member-function-const |
The mock port hooks keep the port contract's non-const signature. |
clang-analyzer-optin.performance.Padding |
The mock chip lists its registers in the manual's order, not the best packing. |
clang-analyzer-optin.core.EnumCastOutOfRange |
The tests cast out-of-range values on purpose to reach the rejection paths. |
cppcoreguidelines-pro-type-reinterpret-cast |
The tests view a string's bytes as uint8_t to build frames. |
cppcoreguidelines-pro-type-member-init |
Golden-case structs are filled by aggregate initialisation. |
performance-inefficient-string-concatenation |
Test messages and expected wires are built once. |
The template's own relaxations (convert-member-functions-to-static, non-private-member-variables-in-classes,
macro-usage, the C-array and designated-initialiser checks for the test registry, and pro-type-vararg for the
report printing) stay as they were.
Documentation snippets¶
Every C++ block on this site is included from tests/docs/, which the host build
compiles (target alicat_basis2_doc_snippets) with the driver instantiated over the documented port. A change to
the API that breaks a page breaks the build first. The README's example is a generated copy of the same file;
kokeb-docs check fails when it drifts.
CI workflows¶
Everything runs from ci.yml, which calls one workflow file per check and ends in
the CI gate job, the single required check (CI gate / gate).
| Job | What it checks |
|---|---|
| Host tests | GCC, then Clang (one job), build and run every test above, then build and run examples/host on its own |
| Host quality | The gates above: complexity, coverage, sanitizers, fuzzing |
| Cross-compile | arm-none-eabi-g++ for Cortex-M0+, M4F and M7F with the strict flags, warnings as errors (nothing linked) |
| C++ lint | clang-format and clang-tidy over the sources |
| MISRA | MISRA C++:2023 and the C++20 profile on traced host and Cortex-M builds, at level high (see Quality gates) |
| Docs | Strict site build (_config/mkdocs.yml), figure and generated-region drift, Doxygen (_config/Doxyfile), offline link check, Lighthouse |
| Repository checks | One job: DCO on pull requests, the org standard, Markdown and YAML lint, the Kokeb Labs handbook rules for .kokeb/handbook.yml (by rule ID), template drift (warns only) and cppcheck over the driver sources; a second job runs the handbook's source checks (include guards, bus requirements, raw access) |
| CI gate | Passes only if every job above passed: the one required check |
The ESP32 examples job builds every app in examples/esp32/app_config.yml, for each listed ESP-IDF version and
build type, with the repository root as the driver's component, and lints the example sources. The gate needs it
too.
The workflows call the shared, versioned workflows of
kokeb-labs/ci-tools at @v1.
On hardware¶
Nothing has been run on an instrument. The MCU examples (Examples) are the bench test: the minimal one
polls the flow once a second, the full one scans, identifies and sets a setpoint. A bench session also settles the
assumptions listed on the Datasheet map: capture the replies of the setters,
DV with several items, GS *, VE, SN and the error reply, and the meter's data frame, with a terminal program,
and set the model's knobs to match. Over Modbus, read 4210 to 4215 after a measurement of a known average above
6553.5 (A25): the literal reading of M p.22 puts the low word in 4214, the driver decodes it high word first like every
other pair, and K-PREVAVG switches the model. Record the model, the firmware and the result in the pull request's bench-status
section.