How an out-of-tree Linux driver survives a kernel update

A kernel update rarely breaks an external driver in one obvious way. Sometimes the module no longer compiles. Sometimes it builds but cannot load because of symbols, configuration or signing. The dangerous case is a module that loads and passes a short smoke test, then corrupts memory during device removal, hangs while unloading, or loses performance under parallel load.

Maintaining an out-of-tree driver is therefore not a compiler-error exercise. It requires a controlled compatibility boundary, an explicit lifetime model, an automated kernel matrix and release criteria that can be verified.

1. Four distinct compatibility failures

Build incompatibility is visible first: a prototype, structure field, header arrangement or configuration dependency changed. It is inconvenient but comparatively safe because it is detected before execution.

Load incompatibility appears later: an exported symbol is unavailable, vermagic differs, the module is not trusted under the machine's Secure Boot policy, or it was built against a different kernel configuration.

Behavioural incompatibility is more dangerous. The code compiles, but a changed contract around object ownership, asynchronous cancellation or callback order opens a race. Finally, performance incompatibility keeps functionality intact while turning a fast path into a serial path through lock contention, extra mappings or incorrect NUMA placement.

Compilation proves only syntactic compatibility. A driver release starts after load, removal, suspend/resume, error and concurrent-load paths have been exercised.

2. A compatibility layer, not conditionals everywhere

The sustainable pattern is to keep core driver logic common and concentrate API differences in a small compatibility directory. A fast path covered in dozens of #if branches is difficult to read, profile and test.

/* compat/ks_device.h */
int ks_device_register(struct ks_device *dev);
void ks_device_unregister(struct ks_device *dev);

/* core/probe.c — shared code */
ret = ks_device_register(dev);
if (ret)
    goto err_stop_workers;

The compatibility implementation may contain two or three adapters, while the rest of the driver sees a stable local contract. Keep that layer thin: type conversion, selection of the correct API form and result normalization. Moving product logic into compatibility code creates a second driver implementation and makes the test matrix unmanageable.

3. Test capabilities instead of kernel version

A LINUX_VERSION_CODE check looks convenient but behaves poorly with distribution kernels. They may backport an API change while retaining an older branch number. A more robust configuration step compiles a tiny probe and generates a capability macro.

/* generated/ks_features.h */
#define KS_HAVE_DEVICE_API_V2 1

#ifdef KS_HAVE_DEVICE_API_V2
    return ks_register_device_v2(dev);
#else
    return ks_register_device_v1(dev);
#endif

A probe should test the exact contract in use: a field, callback shape or helper. Store probe results with the build artifact so that a failure on a customer system can be reproduced.

4. Lifetime and concurrency matter more than compilation

After adapting APIs, the hardest defects are usually in device shutdown. Removal can race with an IRQ, workqueue, timer, open file descriptor and user request. A pointer that was safe under the previous callback order can become a use-after-free.

A reliable design separates four actions: reject new operations, stop event producers, wait for in-flight work, and only then release memory. The primitives vary — refcount_t, kref, cancel_work_sync(), del_timer_sync(), synchronize_rcu() — but the order must be explicit and tested.

static void ks_remove(struct pci_dev *pdev)
{
    struct ks_device *dev = pci_get_drvdata(pdev);

    WRITE_ONCE(dev->stopping, true);
    ks_mask_interrupts(dev);
    synchronize_irq(dev->irq);
    cancel_work_sync(&dev->recovery_work);
    ks_wait_for_users(dev);
    ks_free_device(dev);
}

This path should be repeated deliberately in stress testing. A data-transfer test says little about correctness during unplug, rebinding or an application terminating with requests outstanding.

5. Build, load and stress-test matrix

A useful CI matrix represents risk classes rather than every kernel ever released:

• the minimum supported kernel;
• the current enterprise LTS branch;
• target distribution kernels;
• the latest stable kernel as early warning;
• at least a production-like and a diagnostic configuration.

Each row should run build, load/unload and functional stages. Risky paths add parallel open/close, reset, error injection, suspend/resume, driver rebinding and sustained traffic. Preserve kernel logs and exact configuration rather than only a green job status.

6. DKMS, module signing and Secure Boot

DKMS automates rebuilding; it does not prove compatibility. A successful build does not guarantee that a module will load or receive a trusted signature. Release engineering must control the key source, certificate enrollment, machine policy and signing after a local rebuild.

A closed driver should provide a diagnostic command that reports source version, commit, feature-probe set, vermagic, important configuration options and package identifier. That turns a vague field failure into a reproducible report.

7. Release acceptance criteria

Readiness should not be expressed as “builds on kernel 6.x”. It should be observable:

• clean builds for the declared matrix;
• load and unload without new kernel warnings;
• safe device removal, reset and cancellation of pending work;
• stress runs with available diagnostic facilities enabled;
• no regression against the agreed throughput and latency baseline;
• reproducible package, signature and installation procedure;
• compatibility changes isolated from core logic and code-reviewed.

This process turns a kernel update from emergency repair into planned maintenance. The driver becomes simpler as well: the unstable external boundary is isolated while the core remains readable and measurable.