In our years of experience debugging in the Windows environment, one of the most useful debugging techniques we’ve found involves the information we can collect from the pool allocator in the kernel environment. Over time, Microsoft has further improved the efficacy of this for debugging by including ever more information with both the debugger tools as well as by increasing the number of data structures described in the public symbols.
In this article, we will provide a basic explanation of how pool management in Windows works from the perspective of someone debugging on the platform. Thus, our goal is not to describe the nuances and details of exactly how the pool manager works, but rather to describe how pool allocation works vis-à-vis what we need to know to more effectively use the debugger.
Types of Pool Memory
In the Windows kernel environment, we have two basic types of pool: paged pool and non-paged pool.
Paged pool are blocks of memory where the virtual addresses have meaning, but the mapping from virtual to physical page may not be defined. In a case where the virtual address has no corresponding physical address, the data is stored in a paging file. Like any other virtual address, if a virtual page is accessed for which there is no corresponding physical page, a page fault occurs and the Memory Manager will allocate a new physical page, fetch the data from the paging file into the physical page and then update the page table so that it points to the correct physical page.
Paged pool is typically used for data structures that need not be always resident in memory for correct operation; there is generally more paged pool than non-paged pool and paged pool has less impact on overall machine performance because it does not lock down physical memory. For example, we do quite a bit of file systems work and we routinely work with structures that are only accessed at IRQL < DISPATCH_LEVEL. In such cases, the data structures may be placed in paged pool.
Non-paged pool are blocks of memory where the virtual addresses have meaning and the mapping of virtual to physical page is defined. Thus, if a “non-paged address” is accessed and the virtual-to-physical page is not defined, the Windows Memory Manager will terminate system operation with the bug check 0x50 (PAGE_FAULT_IN_NONPAGED_AREA).
Non-paged pool is typically used for data structures that must be resident in memory for correct operation. For example, any data that is needed to access the paging file (which is where paged pool data may be stored) must be in non-paged pool.
Windows Pool Allocator
Regardless of the type of pool, Windows uses the same pool allocation logic – it just maintains separate pools with different characteristics.
Dynamic memory allocation is an essential part of most kernel components as it provides a general framework for allocating resources on demand and thus permitting considerable flexibility in resource utilization. Indeed, the trend in Windows has been to encourage the use of dynamically allocated structures and discourage the use of statically allocated structures, as the former leads to greater flexibility in terms of implementation.
The core Windows OS provides two key functions for utilizing its dynamic memory allocator:
- ExAllocatePoolWithTag – this function allocates blocks of memory for use by a kernel component, including a driver.
- ExFreePool – this function frees a previously allocated block of memory to a kernel component.
The Windows pool allocator splits allocation requests into two categories – those that are small enough to use the small pool allocator (some value less than PAGE_SIZE so that it makes sense to fit multiple entries on a single page) and the large pool allocator, which allocates blocks of memory in integral multiples of PAGE_SIZE.
For large pool allocations, the actual allocation tracking information is stored separately from the block of memory and thus is handled completely differently. Most blocks of pool memory allocated by drivers are typically much smaller than a single page.
For allocations smaller than one page, the small pool allocator is used to satisfy the request. For allocations of one page or more they are allocated by the large pool allocator – in multiples of PAGE_SIZE.
Pool Block (Small Pool Allocations)
Memory allocated by the small pool allocator consists of a header, data region and special computed value (“canary”). See Figure 1.
Note that the “canary” value follows the data region and provides a mechanism for detecting buffer overwrite error. While special pool can also be used to detect buffer overruns, the canary technique is always active (beginning with Windows Server 2003) and does not require consumption of two pages of virtual address space to implement.
Aside: it is called a canary as an early warning sign of an imminent problem. Coal miners would use canaries as a form of “early warning sign” in coal mines, so when the canary stopped singing the miners knew something was wrong. http://www.wisegeek.org/what-does-it-mean-to-be-a-canary-in-a-coal-mine.htm
When a canary overwrite is detected, Windows will raise a bug check. This applies to all drivers in the system, not just those drivers that are running under Driver Verifier.
The first portion of any block of pool memory is the _POOL_HEADER structure, which defines the current layout of the header used by the small pool allocator. Here is a current version of it from a Windows 8 x64 box:
0: kd> dt nt!_POOL_HEADER +0x000 PreviousSize : Pos 0, 8 Bits +0x000 PoolIndex : Pos 8, 8 Bits +0x000 BlockSize : Pos 16, 8 Bits +0x000 PoolType : Pos 24, 8 Bits +0x000 Ulong1 : Uint4B +0x004 PoolTag : Uint4B +0x008 ProcessBilled : Ptr64 _EPROCESS +0x008 AllocatorBackTraceIndex : Uint2B +0x00a PoolTagHash : Uint2B
ck of the state of the each individual block of pool memory within a given page.
Debugger “!pool” Command
The debugger has a special extension command for looking at a given memory location to see if it exists within a block of pool. This command – the !pool – command – will examine the entire page of memory, looking to identify the pool block that contains the given address.
If the address given is not a valid pool address, it will typically display some sort of error condition, since the memory will not be laid out in the proper format.
0: kd> !pool fffffa8016990040 Pool page fffffa8016990040 region is Nonpaged pool *fffffa8016990000 size: 540 previous size: 0 (Allocated) *Thre Pooltag Thre : Thread objects, Binary : nt!ps fffffa8016990540 size: 30 previous size: 540 (Allocated) AlSe fffffa8016990570 size: 210 previous size: 30 (Allocated) ALPC fffffa8016990780 size: 880 previous size: 210 (Allocated) LSfR
Note that the allocated block in which the address given to !pool was given is indicated by the * at the beginning of the line.
The debugger extension exploits its knowledge of the pool block layout to analyze the page. Each header contains information that is sufficient to “walk” the list of pool allocations for that entire page, with each byte on the page either being part of the pool manager meta-data (header or canary) or the data region as well as some padding.
Note that the Windows small pool allocator guarantees the memory returned will be on an eight byte boundary. There are parts of Windows that still exploit this knowledge and sometimes they use the low two bits of an address for storing information – but those bits are not really part of the memory address.
When a driver allocates a block of pool, the pool allocator will attempt to find an existing block of pool of the correct size. This is done by maintaining lists of free pool regions. Typically, this has been implemented by using the data region as a doubly- linked list pointer so that the memory can be stored on a list of the relevant size – though we note that nothing requires this implementation and the details of the pool allocator do change from release to release.
Indeed, as Windows has evolved, the pool allocator has become increasingly sophisticated in its management of pool. Thus, pool is now organized by NUMA Node as well as by individual CPU. The idea is that in order to minimize contention, each CPU will use its own pool region first and when it needs more memory it can get it from the NUMA node specific allocation. Some versions of Windows will level out kernel memory allocations in order to ensure uniform performance across the entire array of CPUs, while application specific memory is typically allocated from the local CPU’s pool cache (it is not restricted to that CPU, but rather preferred by a given CPU in order to optimize performance).
Pool Tags
When memory is allocated, a four-byte tag value can be specified. By convention this is usually just an array of four characters; in some circumstances the pool allocator will interpret the high bit of the tag to have special meaning.
For those performing crash analysis, this pool tag can help in figuring out what a given pool block represents based upon that value. For those writing drivers, picking unique pool tags is invaluable in tracking down and finding memory leaks as well as aiding in forensic analysis of crashes reported by test teams and maybe even customers.
The debugger uses a file called pooltag.txt in order to display diagnostic information about these tag values. On my development system this file is found in either C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\triage or C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x86\triage. On my system these two files are identical.
This file consists of a series of lines that include the tag value and then the corresponding “hint” that should be displayed by the debugger:
Irp – <unknown> – Io, IRP packets
Note: if you were to look at the original code you would see a string like ‘ prI’ in the call to ExAllocatePool WithTag. That’s because of the little endian nature of Windows platforms, so that the bytes appear in memory in the “correct” order.
For someone using the debugger, this allows the use of this information for inferring the type of a given pool region. For example, if something is a device object, typically it will consist of an object header (nt!_OBJECT_HEADER) followed by the actual device object. Using this knowledge allows us to compute the address of the object and then supply that result to the !devobj command.
Conclusion
A basic understanding of the Windows memory allocator can be helpful in understanding both the function of this critical OS feature as well as in simplifying debugging, particularly of OS components.
OSR Open Systems Resources, Inc.