Running the suite under sanitizers¶

A segfault is rarely where the bug is. Freed memory that something still points at is read successfully most of the time — whatever the allocator put in the block is read as if it were the object — so the symptom is "it sometimes crashes", often in a place that has nothing to do with the mistake. A sanitizer build turns that probability into a certainty: the stale read fails at the instruction that made it, and prints the stack that freed the memory beside the stack that read it.

This is how to build one and what to run in it.

1. A sanitizer build¶

cmake -S . -B build/asan -G Ninja \
      -DCMAKE_TOOLCHAIN_FILE=$PWD/build/clang-ninja/conan_toolchain.cmake \
      -DCMAKE_BUILD_TYPE=Release \
      -DULIB_SANITIZE=address,undefined \
      -DBUILD_TESTING=ON
cmake --build build/asan -j

CMAKE_BUILD_TYPE=Release is not a slip, and not negotiable: conan generated this project's dependency targets for exactly one configuration, so a tree configured as RelWithDebInfo gets those targets stripped of their include directories — which surfaces as flatbuffers/flatbuffers.h not being found, with nothing to connect it to the build type. -g and -fno-omit-frame-pointer are added by the sanitizer module regardless, so the stacks have names in them either way.

ULIB_SANITIZE takes whatever -fsanitize= takes. The useful ones here:

Value	Finds
`address,undefined`	use-after-free, buffer overrun, double free, leaks, plus signed overflow, bad shifts, misaligned loads, null dereference
`address`	the memory half alone, which is faster
`thread`	data races — a build of its own, since it cannot be combined with `address`

A separate build directory, not a reconfigure of the ordinary one: the flags change every object file, and you want both trees available.

The dependencies (Qt, VTK, Geant4, ROOT, Boost, Python) are not instrumented — they are whatever your environment provides. That is the normal arrangement and still catches our own bugs; it means a report whose frames are all inside libvtk is usually a limitation rather than a finding.

See CMake/uLibSanitizers.cmake for what the option does.

2. What the build has to do differently¶

Four things, all in CMake/uLibSanitizers.cmake, each learned by a test failing for a reason that had nothing to do with memory:

-Wl,--no-as-needed. Several classes reach ObjectFactory only through a static initializer, so a test that creates one by name references no symbol from the library that registers it. The ordinary build happens to keep that library; instrumenting changes what is referenced and the linker drops it — then Create("ContainerBox") returns null and a dozen tests fail for want of a class. Keeping the libraries is what makes a sanitizer run comparable with an ordinary one.

LD_PRELOAD for Python-driven tests only. A test that runs python3 is loading our instrumented libraries into an interpreter that is not instrumented, and the ASan runtime has to be in the process first, or the import fails with undefined symbol: __asan_option_detect_stack_use_after_return. Preloading it into our own executables is wrong — they already link it, and a second copy aborts with "application is linked against incompatible ASan runtimes". Hence two calls: ulib_sanitize_directory_tests() for a directory of executables and ulib_sanitize_python_tests(<names>) for the interpreter-driven ones.

Timeouts ×4, and ULIB_TEST_SLOWDOWN. Instrumented code runs several times slower. ulib_sanitize_directory_tests() scales each test's TIMEOUT property; a harness that waits on the wall clock itself (gcompose's console test waits for a startup banner) reads ULIB_TEST_SLOWDOWN from the environment instead. Scaled rather than removed: a hang should still fail.

Re-applying after an override. set_tests_properties(... ENVIRONMENT ...) replaces the property, so a directory that sets its own environment after uLib_add_tests drops the sanitizer options with it. Those directories call ulib_sanitize_directory_tests() at the end, which appends to whatever each test ended up with and skips the ones that already carry the options.

3. Running the tests¶

Every test registered through uLib_add_tests carries the label sanitizable and the sanitizer runtime options, so:

ctest --test-dir build/asan -L sanitizable -j 4          # the labelled suite
ctest --test-dir build/asan -j 4                         # everything
ctest --test-dir build/asan -R DeleteThenReopenTest --output-on-failure

Use fewer jobs than usual. ASan multiplies memory use by roughly three, and the Geant4 tests are not small to begin with.

The options each test runs with (CMake/uLibSanitizers.cmake):

abort_on_error=1 — a report is a failure, not a warning scrolled past;
detect_stack_use_after_return=1, strict_string_checks=1;
print_stacktrace=1, halt_on_error=1 for UBSan;
detect_leaks from -DULIB_SANITIZE_LEAKS=ON|OFF, with CMake/sanitizers/leaks.supp for the libraries that keep their state for the life of the process by design.

Leaks¶

Leak detection is off by default, and on with -DULIB_SANITIZE_LEAKS=ON. That is a judgement about what a run is usually for, not about leaks being unimportant: in the first full run of this suite, 121 of 251 tests reported at exit while the same run found four memory errors. The errors are what a segfault hunt needs, and 121 reports bury four. So the default build gates on errors; leaks are a pass you ask for.

When you do ask for it, read the reports by allocation site rather than count — most are retentions, objects still reachable when main returns because nothing tears the process down tidily, and they are not what makes a session grow. A leak worth fixing is one that repeats per item of work. Vtk::Prop3D's display properties were exactly that (one per field per prop, and Vtk::MuonScatter makes a prop per muon), which is why that one was fixed and the rest are recorded here rather than silenced.

CMake/sanitizers/leaks.supp covers the libraries that keep state for the life of the process by design. It is deliberately short and each entry says why it cannot be fixed from this side. Do not widen a pattern to silence a report — a broad leak:libQt6* hides our own leaks through Qt, which is the opposite of the point.

4. Running the application under it¶

QT_QPA_PLATFORM=offscreen \
ASAN_OPTIONS=abort_on_error=1:detect_leaks=0 \
  build/asan/app/gcompose/gcompose assets/examples/geant/20_driven_stream.gcx

Qt's own plugins are not instrumented, so a few frames will be opaque; the frame that matters — ours — will not be. For a sequence you can script, drive it through the remote port rather than clicking:

build/asan/app/gcompose/gcompose --port 0 scene.gcx &
# then, from Python: client.eval_python("ctx.remove(ctx['crate'])")

A GUI session under ASan starts slowly (tens of seconds) and uses several gigabytes. That is expected.

5. The first full run¶

251 tests, -DULIB_SANITIZE=address,undefined, leaks on:


memory errors	4 — 2 heap-use-after-free, 2 stack-use-after-scope
undefined behaviour	0
leak reports	121 tests, with leak detection on

All four errors are fixed (below). After the fixes and the build changes in section 2, the same suite reports 0 memory errors, 0 undefined behaviour and 0 leak reports, and the only tests that fail are the two that also fail in an ordinary build (PayloadShapeTest's recorded-shape drift and GComposeConfigTest). One test is disabled there on purpose — GcomposeConsoleModeTest, see ulib_sanitize_skip_tests.

The leak reports are a separate, larger piece of work. Read them by allocation site: the top ones were Vtk::Prop3D's display properties (fixed — see below), Geant::Scene and its world box, and property construction, and most are objects alive when main returns rather than growth during a run.

6. What this has found¶

DeleteThenReopenTest exists because of a report that reads exactly like the class of bug described at the top: "I open a file from the recent list, delete a logical or physical volume, and without saving I open the same file again — the app segfaults, and sometimes I see other segfaults too."

The cause was not in opening the file at all. gcompose::RemoveObject was given a raw uLib::Object* and immediately read it — its name for the journal, its parent to find the owning context. Deleting a parent destroys everything under it, so a caller working through a list of selected objects (the context tree's Delete with a parent and a child selected) handed the second call an object the first had already destroyed. Reading it worked often enough to look fine, and crashed when it did not — SmartPointer::release() on freed memory, which is what the stack showed.

Two changes came out of it, and the pair is the general lesson:

RemoveObject establishes that the object is still in the scene — by pointer comparison, before anything dereferences it — and reports an object that is already gone as removed, because that is what the caller asked for. Make the dangerous call safe.
RemoveObjects takes a whole selection and reduces it to its top-most members before deleting anything, which is what a multiple selection means; the context tree's two Delete paths use it. And do not generate the situation.

The other three¶

~PythonEngine waited on a destroyed recipe. The engine remembers which recipes are running as raw pointers; one destroyed in between (its context torn down, or removed and the last reference dropped) leaves an entry naming freed memory, and teardown called IsRunning() on every entry. It now waits only on recipes something still holds, and drops the rest. In a session this is "the app sometimes crashes on exit".

A test read an object it had just removed. vtkTransformModifierTest passed a destroyed TransformModifier* to a function taking Object*. Worth knowing in general: uLib::Object is a virtual base, so that conversion reads the object's vtable to find the base offset — a dangling pointer is unsafe to convert, never mind dereference, and nothing in the source looks like a dereference.

A signal handler outlived what it captured. vtkAssemblyTest connected a lambda capturing a stack counter by reference and never disconnected it; the property outlives the block, so a later announcement wrote to dead stack. Object::connect returns a Connection — keep it when the slot captures anything with a shorter life than the signal.

7. Writing a test that a sanitizer can judge¶

DeleteThenReopenTest does not try to crash. It performs the sequence through the real panels — open, select, delete, delete a parent with its child, open again — and asserts only what must be true (the object count after re-opening, that one removal covers both). The crash was never reproducible on demand; under ASan the stale read is a hard failure wherever it happens.

That is the shape to copy for this class of bug:

perform the real sequence, through the same entry points the UI uses, rather than calling the library the way no caller does;
assert the invariants, not the crash;
leave the memory judgement to the tool, and register the test with the sanitizable label so a sanitizer run includes it;
keep it fast enough to live in the ordinary suite too — it is a regression test either way, and the fix above makes it pass without a sanitizer.