WINDOWS KERNEL / DEBUGGING · 2001

Debugging Errors in Windows NT Drivers

A practical guide to driver-failure diagnosis: bugchecks, IRQL, map files, memory dumps, resource leaks, hangs and multiprocessor defects.

Windows NT is a stable operating system until a poorly tested kernel component — a driver or kernel-mode library — introduces memory corruption, an invalid IRQL transition or an unfinished I/O request. The visible result may be a hung application, a frozen system or a bugcheck.

No article can enumerate every driver defect, but the recurring categories and the methods used to isolate them are surprisingly consistent.

Hardware Problems

When a device and its driver are developed in parallel, the hardware itself may cause failures. Typical symptoms are an error status despite a command matching the specification, missing interrupts or an unexpected response to a request. The practical approach is to reproduce the problem and collect enough traces, register values and timing information for the hardware team.

System Crashes

A kernel driver can crash the operating system with a null or uninitialized pointer. Incorrect DMA addresses are even more dangerous because a device controller may overwrite memory belonging to unrelated components.

A blue screen is dramatic, but it is also a compact diagnostic report. Windows stops all threads, switches to text mode, prints the machine state and, when configured, writes a memory dump. An active kernel debugger can then inspect the failure immediately.

VOID KeBugCheck(ULONG Code);
VOID KeBugCheckEx(
    ULONG Code,
    ULONG Arg1,
    ULONG Arg2,
    ULONG Arg3,
    ULONG Arg4
);

Inside KeBugCheckEx

KeBugCheck forwards to KeBugCheckEx with zeroed additional arguments. The bugcheck path disables interrupts, displays the stop line, translates the code into text and runs callbacks registered through KeRegisterBugCheckCallback. A driver may use such a callback to place diagnostic information in the dump buffer or to put controlled hardware into a safe state.

KeDumpMachineState examines the four bugcheck arguments. If an argument points inside a loaded driver, the screen identifies the module and its base address. It then prints processor information, IRQL, build number, loaded modules and stack frames. A frame contains its address, return address and several values that may correspond to function arguments.

The driver name in the stack is a strong clue, but its absence does not clear the driver: an earlier write may have corrupted data and caused a later failure in a different module.

Decoding a Bugcheck

The examples below use the module base, the failing instruction address, a linker map file and compiler-generated assembly to map a production crash back to a source line. This method remains useful when the deployed driver was built without symbolic debugging information.

0xA — IRQL_NOT_LESS_OR_EQUAL

IRQL NOT LESS OR EQUAL bugcheck
Figure 1. IRQL_NOT_LESS_OR_EQUAL.
*** STOP: 0x0000000A
(0x00000003, 0x00000002, 0x00000001, 0xF91CA315)
IRQL_NOT_LESS_OR_EQUAL
Address F91CA315 has base at F91CA000 — testdrv.sys

The parameters say that the driver attempted to write to address 3 while running at DISPATCH_LEVEL. The failing instruction belongs to testdrv.sys. Subtracting the module base from the instruction address gives offset 0x315.

The map file identifies which function contains that offset; the generated .cod file identifies the instruction inside the function. In the example, offset 0x15 in Routine3 resolves to:

mov eax, DWORD PTR _uArg4$[ebp]
mov ecx, DWORD PTR _uArg3$[ebp]
mov DWORD PTR [eax], ecx

The corresponding source statement is *(ULONG*)uArg4 = uArg3;. uArg4 contains the invalid address and the code raises IRQL before accessing it.

0x1E — KMODE_EXCEPTION_NOT_HANDLED

KMODE EXCEPTION NOT HANDLED bugcheck
Figure 2. KMODE_EXCEPTION_NOT_HANDLED.
*** STOP: 0x0000001E
(0xC0000005, 0xF91CA2F9, 0x00000001, 0x00000002)
KMODE_EXCEPTION_NOT_HANDLED

The first argument is the exception code and the second is the instruction address. 0xC0000005 means access violation; the remaining arguments show a write to address 2. The same base-address and map-file calculation places the instruction in Routine2 and again resolves it to the invalid pointer assignment.

Where an operation can legitimately fault, structured exception handling can prevent a system-wide crash:

__try {
    *(ULONG*)uArg4 = uArg3;
}
__except (EXCEPTION_EXECUTE_HANDLER) {
    /* handle the failure */
}

0x50 — PAGE_FAULT_IN_NONPAGED_AREA

PAGE FAULT IN NONPAGED AREA bugcheck
Figure 3. PAGE_FAULT_IN_NONPAGED_AREA.

For this code the arguments may not identify the source. The stack becomes essential. A return address inside the driver is converted to a module-relative offset and then mapped to a function. In the sample the return address lands just after a call to ExFreePool. The stack shows that the routine received address 1, so the driver attempted to free an invalid pool block.

0x200 — A Driver-Defined Bugcheck

Driver-defined bugcheck 0x200
Figure 4. A bugcheck code unknown to the operating system.
*** STOP: 0x00000200
(0x00000001, 0x00000002, 0x00000003, 0x00000004)

Windows cannot provide a name because the code is driver-defined. The return address is still enough to locate the call site. In the example, the stack and map file lead to Routine0, whose source explicitly calls KeBugCheckEx(0x200, 1, 2, 3, 4).

When the screen is insufficient, analyze a full or mini memory dump. Later NT-based systems moved much of the module and stack information from the visible screen into the dump file.

Resource Leaks

User-mode process cleanup does not exist for a kernel driver. The driver must release every pool allocation, object reference and IRP it creates. Leaks gradually reduce available pool and may eventually make the system fail far from the original defect.

Pool Tags

Allocate pool with ExAllocatePoolWithTag and use meaningful four-character tags. Different structures or subsystems can use different tags. WinDbg commands such as !poolused and !poolfind then identify outstanding allocations; system-created IRPs use the tag 'Irp '.

WinDbg system-pool analysis
Figure 5. Locating tagged pool blocks and inspecting an IRP.

POOLMON, distributed with the NT DDK, displays live statistics by tag. Older Windows NT releases required the FLG_POOL_ENABLE_TAGGING flag in GlobalFlag.

Dynamic Pool Checks

Tail checking places a known pattern after each allocation and verifies it at free time, exposing writes beyond the requested size. Free checking fills released blocks and detects writes after free. In the historical Windows NT debugger these were enabled with FLG_POOL_ENABLE_TAIL_CHECK and FLG_POOL_ENABLE_FREE_CHECK.

Hung Threads

A common driver bug is failing to call IoCompleteRequest for every IRP. The requesting user thread remains blocked because the I/O manager still considers the request outstanding. Recursive acquisition of a Fast Mutex or Executive Resource can also deadlock a request path because these primitives are not recursively acquirable.

System Hangs

Recursively acquiring a spin lock can freeze the entire machine. The same is true of an infinite loop in an interrupt or timer callback, where execution blocks progress at elevated IRQL.

Intermittent Defects

The hardest defects appear only under heavy I/O, on a particular CPU speed, with little memory or on multiprocessor hardware. Stress the driver with parallel applications and a large request rate; historical NT systems allowed available physical memory to be limited through MAXMEM in boot.ini.

On multiprocessor systems an interrupt handler, timer callback and I/O dispatch routine may run concurrently, exposing synchronization defects hidden on a single processor.

Release-only failures often point to uninitialized variables, inline assembly that does not preserve registers, or different structure alignment between debug and optimized builds.

Driver defects are localizable when the investigation is systematic: preserve the failure evidence, calculate module-relative addresses, correlate them with map and assembly listings, inspect the stack and pool, and reproduce the problem under the conditions that make it most likely.

Related service

Drivers and system filters

Drivers and system filters → Discuss a project