How to debug dump multiple dumps

how-to-debug-dump.md

Get the Helix payload

Runfo helps get information about helix test runs and azure devops builds. We will use it to download the payload and symbols (recommended version 0.6.4 or later):

dotnet tool install --global runfo
dotnet tool update --global runfo

If prompted, open a new command prompt to pick up the updated PATH.

# On Windows
# assumes c:\helix_payload\Microsoft.Extensions.Hosting.Functional.Tests does not exist
runfo get-helix-payload -j 39c497dd-2d87-4609-8881-9f5f4753c2be -w Microsoft.Extensions.Hosting.Functional.Tests -o c:\helix_payload\Microsoft.Extensions.Hosting.Functional.Tests
# On Linux and macOS
# assumes ~/helix_payload/Microsoft.Extensions.Hosting.Functional.Tests does not exist
runfo get-helix-payload -j 39c497dd-2d87-4609-8881-9f5f4753c2be -w Microsoft.Extensions.Hosting.Functional.Tests -o ~/helix_payload/Microsoft.Extensions.Hosting.Functional.Tests

Any dump files published by helix will be downloaded.

NOTE: if the helix job is an internal job, you need to pass down a helix authentication token using the --helix-token argument.

Now extract the files:

# On Windows
for /f %i in ('dir /s/b c:\helix_payload\Microsoft.Extensions.Hosting.Functional.Tests\*zip') do tar -xf %i -C c:\helix_payload\Microsoft.Extensions.Hosting.Functional.Tests

# On Linux and macOS
# obtain `unzip` if necessary; eg `sudo apt-get install unzip` or `sudo dnf install unzip`
find ~/helix_payload/Microsoft.Extensions.Hosting.Functional.Tests -name '*zip' -exec unzip -d ~/helix_payload/Microsoft.Extensions.Hosting.Functional.Tests {} \;

Now use the dotnet-sos global tool to install the SOS debugging extension.

dotnet tool install --global dotnet-sos
dotnet tool update --global dotnet-sos

If prompted, open a new command prompt to pick up the updated PATH.

# Install only one: the one matching your dump
dotnet sos install --architecture Arm
dotnet sos install --architecture Arm64
dotnet sos install --architecture x86
dotnet sos install --architecture x64

Now choose a section below based on your OS.

If it's a Windows dump on Windows...

... and you want to debug with WinDbg

Install or update WinDbg if necessary (external, internal). If you don't have a recent WinDbg you may have to do .update sos.

Open WinDbg and open the dump with File>Open Dump.

!setclrpath c:\helix_payload\Microsoft.Extensions.Hosting.Functional.Tests\shared\Microsoft.NETCore.App\6.0.0
.sympath+ c:\helix_payload\Microsoft.Extensions.Hosting.Functional.Tests\shared\Microsoft.NETCore.App\6.0.0

Now you can use regular SOS commands like !dumpstack, !pe, etc.

... and you want to debug with Visual Studio

Currently this is not possible because mscordbi.dll is not signed.

... and you want to debug with dotnet-dump

Install the dotnet-dump global tool.

dotnet tool install --global dotnet-dump
dotnet tool update --global dotnet-dump

If prompted, open a new command prompt to pick up the updated PATH.

dotnet-dump analyze c:\helix_payload\Microsoft.Extensions.Hosting.Functional.Tests\workitems\Microsoft.Extensions.Hosting.Functional.Tests\crashdump_31967.dmp
dotnet-dump analyze c:\helix_payload\Microsoft.Extensions.Hosting.Functional.Tests\workitems\Microsoft.Extensions.Hosting.Functional.Tests\crashdump_31970.dmp
dotnet-dump analyze c:\helix_payload\Microsoft.Extensions.Hosting.Functional.Tests\workitems\Microsoft.Extensions.Hosting.Functional.Tests\crashdump_7164.dmp
dotnet-dump analyze c:\helix_payload\Microsoft.Extensions.Hosting.Functional.Tests\workitems\Microsoft.Extensions.Hosting.Functional.Tests\crashdump_51193.dmp
dotnet-dump analyze c:\helix_payload\Microsoft.Extensions.Hosting.Functional.Tests\workitems\Microsoft.Extensions.Hosting.Functional.Tests\crashdump_51187.dmp

Within dotnet-dump:

setclrpath c:\helix_payload\Microsoft.Extensions.Hosting.Functional.Tests\shared\Microsoft.NETCore.App\6.0.0
setsymbolserver -directory c:\helix_payload\Microsoft.Extensions.Hosting.Functional.Tests\shared\Microsoft.NETCore.App\6.0.0

Now you can use regular SOS commands like dumpstack, pe, etc. If you are debugging a 32 bit dump using 64 bit dotnet, you will get an error SOS does not support the current target architecture. In that case replace dotnet-dump with the 32 bit version:

dotnet tool uninstall --global dotnet-dump
"C:\Program Files (x86)\dotnet\dotnet.exe" tool install --global dotnet-dump

---

If it's a Linux dump on Windows...

Download the Cross DAC Binaries, open it and choose the flavor that matches the dump you are to debug, and copy those files to c:\helix_payload\Microsoft.Extensions.Hosting.Functional.Tests\shared\Microsoft.NETCore.App\6.0.0.

Now you can debug with WinDbg or dotnet-dump as if it was a Windows dump. See above.

---

If it's a Linux dump on Linux...

... and you want to debug with LLDB

Install or update LLDB if necessary (instructions here)

Load the dump:

lldb --core ~/helix_payload/Microsoft.Extensions.Hosting.Functional.Tests/workitems/Microsoft.Extensions.Hosting.Functional.Tests/crashdump_31967.dmp ~/helix_payload/Microsoft.Extensions.Hosting.Functional.Tests/shared/Microsoft.NETCore.App/6.0.0/dotnet
lldb --core ~/helix_payload/Microsoft.Extensions.Hosting.Functional.Tests/workitems/Microsoft.Extensions.Hosting.Functional.Tests/crashdump_31970.dmp ~/helix_payload/Microsoft.Extensions.Hosting.Functional.Tests/shared/Microsoft.NETCore.App/6.0.0/dotnet
lldb --core ~/helix_payload/Microsoft.Extensions.Hosting.Functional.Tests/workitems/Microsoft.Extensions.Hosting.Functional.Tests/crashdump_7164.dmp ~/helix_payload/Microsoft.Extensions.Hosting.Functional.Tests/shared/Microsoft.NETCore.App/6.0.0/dotnet
lldb --core ~/helix_payload/Microsoft.Extensions.Hosting.Functional.Tests/workitems/Microsoft.Extensions.Hosting.Functional.Tests/crashdump_51193.dmp ~/helix_payload/Microsoft.Extensions.Hosting.Functional.Tests/shared/Microsoft.NETCore.App/6.0.0/dotnet
lldb --core ~/helix_payload/Microsoft.Extensions.Hosting.Functional.Tests/workitems/Microsoft.Extensions.Hosting.Functional.Tests/crashdump_51187.dmp ~/helix_payload/Microsoft.Extensions.Hosting.Functional.Tests/shared/Microsoft.NETCore.App/6.0.0/dotnet

Within lldb:

setclrpath ~/helix_payload/Microsoft.Extensions.Hosting.Functional.Tests/shared/Microsoft.NETCore.App/6.0.0
sethostruntime /usr/bin/dotnet
setsymbolserver -directory ~/helix_payload/Microsoft.Extensions.Hosting.Functional.Tests/shared/Microsoft.NETCore.App/6.0.0

If you want to load native symbols

loadsymbols

... and you want to debug with dotnet-dump

Install the dotnet-dump global tool.

dotnet tool install --global dotnet-dump
dotnet tool update --global dotnet-dump

If prompted, open a new command prompt to pick up the updated PATH.

dotnet-dump analyze ~/helix_payload/Microsoft.Extensions.Hosting.Functional.Tests/workitems/Microsoft.Extensions.Hosting.Functional.Tests/crashdump_31967.dmp
dotnet-dump analyze ~/helix_payload/Microsoft.Extensions.Hosting.Functional.Tests/workitems/Microsoft.Extensions.Hosting.Functional.Tests/crashdump_31970.dmp
dotnet-dump analyze ~/helix_payload/Microsoft.Extensions.Hosting.Functional.Tests/workitems/Microsoft.Extensions.Hosting.Functional.Tests/crashdump_7164.dmp
dotnet-dump analyze ~/helix_payload/Microsoft.Extensions.Hosting.Functional.Tests/workitems/Microsoft.Extensions.Hosting.Functional.Tests/crashdump_51193.dmp
dotnet-dump analyze ~/helix_payload/Microsoft.Extensions.Hosting.Functional.Tests/workitems/Microsoft.Extensions.Hosting.Functional.Tests/crashdump_51187.dmp

Within dotnet-dump:

setclrpath ~/helix_payload/Microsoft.Extensions.Hosting.Functional.Tests/shared/Microsoft.NETCore.App/6.0.0
setsymbolserver -directory ~/helix_payload/Microsoft.Extensions.Hosting.Functional.Tests/shared/Microsoft.NETCore.App/6.0.0

---

If it's a macOS dump

Instructions for debugging dumps on macOS are essentially the same as Linux with one exception: dotnet-dump cannot analyze macOS system dumps: you must use lldb for those. dotnet-dump can only analyze dumps created by dotnet-dump or createdump, by the runtime on crashes when the appropriate environment variables are set, or the blame-hang setting of dotnet test.

---

Other Helpful Information

• How to debug a Linux core dump with SOS