SOSEX 4.6.0.377

Over the course of the last few months, I've really put lots of effort into understanding and utilizing WinDbg. As a primarily C# developer, this meant also becoming intimately familiar with the SOS extension. Though a bit tedious, this exercise has already paid rich dividends in my debugging experience. As powerful and handy as SOS is, however, it has some annoying limitations and quirks. My personal peeves with SOS, combined with my desire to learn to write a WinDbg extension, led me to develop SOSEX, a debugging extension for managed code that begins to alleviate some of my frustrations with SOS. SOSEX (available in x86 and x64 versions) provides 8 easy-to-use commands: !dumpgen (dumps the contents of a GC generation), !gcgen (indicates the GC generation of a given object), !refs (lists all references to and from the specified object), !bpsc (breakpoint, source code), !bpmo (breakpoint, method offset), !vars (dump all args and local variables), !isf (inspect static field) and !dlk (deadlock detection).

The rest of this post will provide a bit more detail about each command and how they can save you time. Use the !help command for a list of commands and !help <command name> for the syntax and usage of each command.

!dumpgen and !gcgen

With SOS, you can dump the contents of the heap like so:

0:000> !dumpheap -short
00000642787c7370
00000642787c7388
00000642787c73b0
00000642787c7410
00000642787c7440
00000642787c7498
00000642787c74f0...

The problem with this is that there is no easy way to tell from the output which generation each object belongs to. You can follow up the call to !dumpheap with !eeheap -gc, which will provide the necessary information to determine generations. However, determining the contents of, say, generation 2 using this method is very tedious. Here's the output of !eeheap -gc for a dual-processor system in server GC mode:

0:000> !eeheap -gc
Number of GC Heaps: 2
------------------------------
Heap 0 (0000000002264180)
generation 0 starts at 0x000000007fff0098
generation 1 starts at 0x000000007fff0080
generation 2 starts at 0x000000007fff0068
ephemeral segment allocation context: none
         segment            begin         allocated             size
0000000002271b80 00000642787c7370  0000064278809088 0x0000000000041d18(269592)
000000007fff0000 000000007fff0068  0000000080002fe8 0x0000000000012f80(77696)
Large object heap starts at 0x00000000ffff0068
         segment            begin         allocated             size
00000000ffff0000 00000000ffff0068  00000000ffff80c8 0x0000000000008060(32864)
Heap Size           0x5ccf8(380152)
------------------------------
Heap 1 (0000000002264e00)
generation 0 starts at 0x00000000bfff0098
generation 1 starts at 0x00000000bfff0080
generation 2 starts at 0x00000000bfff0068
ephemeral segment allocation context: none
         segment            begin         allocated             size
00000000bfff0000 00000000bfff0068  00000000bfff00b0 0x0000000000000048(72)
Large object heap starts at 0x000000010fff0068
         segment            begin         allocated             size
000000010fff0000 000000010fff0068  000000010fff0080 0x0000000000000018(24)
Heap Size              0x60(96)
------------------------------
GC Heap Size           0x5cd58(380248)

As you can see, you have a lot of work to do in order to pick through the output of !dumpheap and compare object addresses to the segment addresses provided by !eeheap -gc. Enter SOSEX's !dumpgen command. Using !dumpgen, you can easily determine the contents of a given generation simply by providing the number of the generation you wish to examine:

0:000> !dumpgen 2
00000642787c7370         24    System.Object
00000642787c7388         38    System.String    STRVAL=length
00000642787c73b0         94    System.String    STRVAL=ArgumentOutOfRange_MustBeNonNegNum
00000642787c7410         46    System.String    STRVAL=startIndex
00000642787c7440         84    System.String    STRVAL=ArgumentOutOfRange_StartIndex
00000642787c7498         86    System.String    STRVAL=ArgumentOutOfRange_IndexLength
00000642787c74f0         42    System.String    STRVAL=capacity...

Conversely, if you know an object's address and you wish to know what generation it belongs to, you can simply provide the address of the object to the !gcgen command.

!refs

Though not a replacement, the !refs command supplements SOS's !gcroot command by allowing you to view  the immediate references from and to a given object.

0:000> !refs 0000000080000db8
Objects referenced by 0000000080000db8 (System.Threading.Mutex):
0000000080000ef0         32    Microsoft.Win32.SafeHandles.SafeWaitHandle

Objects referencing 0000000080000db8 (System.Threading.Mutex):
0000000080000e08         72    System.Threading.Mutex+<>c__DisplayClass3
0000000080000e50         64    System.Runtime.CompilerServices.RuntimeHelpers+CleanupCode

The sample output above shows only heap references, but !refs will also list all references from handles, stacks, registers and the freachable queues.

!dlk

The !dlk command allows you to easily spot deadlocks in your application if you suspect deadlock to be the cause of an application hang. If !dlk detects deadlock, the output will list the sync blocks that are held as well as the sync blocks for which each thread is waiting, as well as the type, method, IL offset, and, if symbols are available, the source code and line number at which each thread is waiting:

0:010> !dlk
Deadlock detected:
CLR thread 4 holds sync block 00000000024c6970 OBJ:000000007fff0f80[System.String] STRVAL=SYNC1
             waits sync block 00000000024c6928 OBJ:000000007fff0fa8[System.String] STRVAL=SYNC2
CLR thread 5 holds sync block 00000000024c6928 OBJ:000000007fff0fa8[System.String] STRVAL=SYNC2
             waits sync block 00000000024c6970 OBJ:000000007fff0f80[System.String] STRVAL=SYNC1
CLR Thread 4 is waiting at ConsoleTestApp.ConsoleTestApp.MonitorDeadlockThreadProc()+0xa4(IL) [C:\dev\ConsoleTestApp\ConsoleTestApp.cs, line 195]
CLR Thread 5 is waiting at ConsoleTestApp.ConsoleTestApp.MonitorDeadlockThreadProc()+0xa4(IL) [C:\dev\ConsoleTestApp\ConsoleTestApp.cs, line 195]

1 deadlock detected.

As you can see, !dlk makes it dead simple to troubleshoot this common kind of deadlock in managed code. One important caveat holds true for !dlk: it only works for deadlocks on "sync blocks". Put more simply, it will only spot locks created by Monitor.Enter (which is used by the C# lock keyword). !dlk will not catch deadlock on other types of synchronization objects, such as mutexes and semaphores.

!bpsc and !bpmo

Although managed debugging with WinDbg has become more supported in later builds, it is still difficult to get managed source code breakpoints to work correctly. With the !bpsc command, it is now simple to set source code breakpoints. !bpsc allows you to set a breakpoint in a source code file at a specified line number and, optionally, column number. I've often wanted to stop the execution of my application either deep inside a method in the BCL or in one of my own methods in a build for which I have no symbols. Though SOS provides the handy !bpmd command, !bpmd only allows you to set a breakpoint on the first instruction of a method. This is often not effective for more difficult debugging scenarios. The !bpmo command now provides a means for you to set a breakpoint at specific IL offset within a given type and method. 

!vars and !isf

Once you are stopped at a breakpoint, you need an easy means to inspect the contents of method arguments, local variables and instance and static fields. !vars dumps the values of arguments and local variables. If symbols are available, !vars will display the names of the local variables. Names of arguments are (with rare exception) provided even without symbols. In order to inspect member variables, first call !vars and then provide the value of "this" to the sos!do command. Using SOS, it is a bit of a cumbersome process to view the contents of static fields. The !isf command makes it simple by allowing you to simply specify the type and field name to inspect.