Fixing Broken Debugger Extensions

As I’m sure you’re all aware at this point, there are two different sets of PDB files generated for Microsoft provided components: private and public. The private symbols are used internally for source level debugging of the operating system. The public symbols are the private symbols with interesting bits stripped from them. These interesting bits include:

• Data structure types
• Local variable names
• Function parameters
• Global variable data types
• Source line information

Around the XP timeframe, it was determined that the public PDBs weren’t as useful as they could be. Notably, the absence of the data structure type information made it difficult to debug OS level issues and created a maintenance burden for the public debugger release (anyone else remember when the debugger extensions had hard coded versions of the internal data structures?).

It was then decided that some of the interesting information lost during the stripping process should be put back into the public PDBs. Note that I did say some of the interesting information, not all. Thus, only data type information is added back into the public PDBs. To restrict this even further, only those data types considered to be necessary or appropriate for public use are added back into the PDBs.

Inevitably, this results in problems with the debugging tools. These tools are developed internally using private PDBs, therefore they don’t always function properly when run against the public PDBs due to missing types. Luckily there is a way to fix this in some cases, as it is possible to add type information to an existing PDB.

The trick is to simply compile a C source file containing the types of interest and specify an existing PDB as the location of the debug information. Instead of overwriting this PDB, the compiler will add the missing types to the file. What’s pretty neat about this solution is that you can even do this to override existing types in the PDB, which you could use to fix broken data structures present in the PDB (and, yes, this does happen too!).

Invoking CL through MSBuild

To do this, we’ll need to invoke the Microsoft C/C++ compiler directly via CL.EXE and carefully supply the debug parameters of interest. But, you didn’t think that we’d wimp out and invoke CL directly from a command window, did you? Of course not! It’s a new year and we have a new build system, so why wouldn’twe do this through an MSBuild project file? The beginning of PDBFix.proj can be seen in Figure 1.

In our project file, we include the properties necessary for a kernel mode driver. This allows us to easily add the WDK include path to our compilation step. For this project, we want complete control over the parameters passed to CL.EXE, thus we import the CPP properties and targets to obtain access to the CL Task.

Minimally, compilation requires that we set the pre-processor define to indicate the processor architecture that we are compiling for. In this project, we rely on the Platformproperty to indicate which pre-processor define should be set. Lastly, we use the custom PdbParameterproperty to determine the location of the PDB file. This allows the user to supply the path to the PDB on the command line via the /property (/p) switch.

Figure 2 shows the remainder of the project file, which is responsible for executing the CL Task.

Adding Types to the Source File

Now that we have our project file, the next step is to add data types to our source file, PDBFix.c. This is pretty straightforward; just add the data structure definition as you would in any other driver. The only thing to remember is unused types do not go in the PDB, thus you’ll need to create a global variable that actually uses the type you define. An example PDBFix.c is shown in its entirety in Figure 3.

Compiling the Code and Adding the Types

Let’s now finally put this together and see how we can add types to an existing PDB using MSBuild. All we need to do is to build our PDBFix project, pointing the output PDB path of that project to an existing PDB file.  This results in the linker adding the symbol definitions we create in our PDBFix project to the already existing PDB file.  Pretty cool, eh?

Let’s walk through the steps:  First, we need to find the location of the PDB that has the missing type information. Once you have the path of the PDB of interest, you’ll need to unload the symbols for the module so that CL can modify the PDB. You can see an example of the steps necessary in Figure 4.

The next step is to open a VS2012 Tools Command Prompt of the appropriate architecture for the target, in this case x86. Once there, navigate to the location of the PDBFix project and execute MSBuild, passing the full path to the PDB as the PdbNameproperty.  If all goes well, you should see output resembling that in Figure 5.

We can now return to WinDBG, reload our now fixed PDB, and see that the type information has indeed been added (Figure 6).

Applying our Technique: Fixing !exqueue on Windows 8 x86 Targets

Now that we know how to add information to PDBs, we can use what we’ve learned to fix the !exqueue command on Windows 8 x86 targets. This is a command that we at OSR use all the time, thus the fact that it’s not working in some cases is an annoyance. We filed a bug report on the problem when we discovered it a while back, but we’re impatient so we set out to fix it ourselves in the meantime.

First, let’s see what happens when we run the command against the public symbols, as shown in Figure 7.

As we discussed earlier, in creating the public PDBs, the data types associated with global variables are stripped out. Thus, as we see from Figure 7’s x and dt command outputs, we have no type information to go with the global and therefore the debugger cannot query its size.

Note that, up to this point, we have only discussed adding missing data types to an existing PDB and have said nothing about associating a global with a type. This isn’t by accident, after several attempts we were unsuccessful in getting this information wired back into the PDB. This is unfortunate, as it would have been an elegant solution to an otherwise ugly problem. All is not lost, however, as we discovered an ugly solution instead.

As it turns out, it’s possible to trick the debugger into getting a size for the global by creating an appropriately sized data type of the same name. In the case of KeNumberNodes, dumping the memory location with the d command shows that the global is likely at most two bytes long. In order to account for this, we created a data type KeNumberNodes with a single USHORT member (Figure 8).

While the x command will still not be able to resolve a type for the global variable, dt will pick up the type definition and correctly interpret the size of that structure as two bytes. This trick is enough to fool the !exqueue command into getting a size for the global, resulting in it happily dumping out the worker queues (Figure 9).

It’s definitely a hack, but the usefulness of the extension makes it worth it until the command is fixed in a future release.

Conclusion

Sources for this article are provided for download, giving you a method to fix most categories of public symbol issues you may come across. If you use it to resolve any problems let us know, one less broken command for us to stumble over when we need it!

 

Code associated with this article:

http://insider.osr.com/2013/code/fix.zip