Pattern-Oriented

Sample Training Exercises

Version 2.0

Facebook LinkedIn Twitter Software Diagnostics Services

Training Courses
 Accelerated Windows Memory Dump Analysis

 Accelerated .NET Memory Dump Analysis

 Accelerated Mac OS X Core Dump Analysis

 Accelerated Linux Core Dump Analysis

 Accelerated Windows Debugging3

 Accelerated Windows Malware Analysis with Memory

Dumps

 Practical Foundations of Windows Debugging,

Disassembling, Reversing

 Accelerated Disassembly, Reconstruction and

Reversing

 Accelerated Windows Software Trace Analysis

 Advanced Windows Memory Dump Analysis with Data

Structures

© 2018 Software Diagnostics Services

Training Packs
 Pattern-Oriented Trace and Log Analysis

 Pattern-Oriented Malware Analysis

 Pattern-Oriented Unix Memory Dump Analysis

 Pattern-Oriented Memory Dump Analysis

 Pattern-Oriented Windows Crash Dump Analysis

 Pattern-Oriented Windows Debugging

 Pattern-Oriented Windows Memory Forensics

 Pattern-Oriented Complete Windows Memory

Dump Analysis

 Complete Pattern-Oriented Software Diagnostics

© 2018 Software Diagnostics Services

Training Roadmap
Intermediate Crash and hang Windows Intermediate Live and source code Windows
diagnostics and analysis? debugging via WinDbg?

Beginner/ Beginner/
Intermediate Intermediate

Accelerated Windows Memory Dump Analysis Accelerated Windows Debugging3

Intermediate/
Advanced Kernel space

Accelerated Windows Malware Analysis User space

Managed .NET space

Intermediate
Advanced Windows Memory Dump Analysis

Accelerated .NET Memory Dump Analysis Intermediate/

Advanced

Accelerated Disassembly, Reconstruction and Reversing

Process Monitor log and/or Assembly language for

CDF/ETW trace analysis? Windows debugging?

Beginner/ Intermediate/ Beginner/

Intermediate Advanced Intermediate

Practical Foundations of Windows Debugging,

Accelerated Windows Software Trace Analysis
Disassembling, Reversing

Mac OX X and GDB/LLDB core Linux and GDB core dump

dump analysis and diagnostics? analysis and diagnostics?

Beginner/ Beginner/
Intermediate Intermediate

Accelerated Mac OS X Core Dump Analysis Accelerated Linux Core Dump Analysis

© 2018 Software Diagnostics Services

 Dmitry Vostokov is an internationally recognized expert, speaker,
educator, scientist, and author. He is the founder of pattern-oriented
software diagnostics, forensics, and prognostics discipline and Software
Diagnostics Institute (DA+TA: DumpAnalysis.org + TraceAnalysis.org).
Vostokov has also authored more than 30 books on software diagnostics,
forensics and problem-solving, memory dump analysis, debugging,
software trace and log analysis, reverse engineering, and malware
analysis. He has more than 20 years of experience in software
architecture, design, development, and maintenance in a variety of
industries including leadership, technical and people management roles.
Dmitry also founded DiaThings, Logtellect, OpenTask Iterative and Incremental Publishing
(OpenTask.com), Software Diagnostics Services (former Memory Dump Analysis Services)
PatternDiagnostics.com and Software Prognostics. In his spare time, he presents various topics on
Debugging.TV and explores Software Narratology, an applied science of software stories that he
pioneered, and its further development as Narratology of Things and Diagnostics of Things (DoT). His
current areas of interest are theoretical software diagnostics and its mathematical and computer
science foundations, software diagnostics engineering and diagnostics-driven development.

6
Published by OpenTask, Republic of Ireland

Copyright © 2016 by OpenTask

Copyright © 2016 by Software Diagnostics Services

Copyright © 2016 by Dmitry Vostokov

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or
transmitted, in any form or by any means, without the prior written permission of the
publisher.

You must not circulate this book in any other binding or cover, and you must impose the same
condition on any acquirer.

Product and company names mentioned in this book may be trademarks of their owners.

OpenTask books and magazines are available through booksellers and distributors worldwide.
For further information or comments send requests to [email protected].

A CIP catalog record for this book is available from the British Library.

ISBN-l3: 978-1-908043-46-7 (Paperback)

Version 4, 2016

2
Contents

About the Author ...........................................................................................................................................................7

Presentation Slides and Transcript .................................................................................................................................9
Practice Exercises .........................................................................................................................................................35
Exercise 0: Download, setup and verify your WinDbg installation ............................................................................40
Exercise P1: Analysis of a normal application process dump (32-bit notepad) .........................................................47
Exercise P2: Analysis of a normal application process dump (64-bit notepad) .........................................................72
Exercise P3: Analysis of a normal application process dump (64-bit Microsoft Edge) ..............................................84
Exercise P4: Analysis of an application process dump (64-bit ApplicationK, no symbols).......................................113
Exercise P5: Analysis of an application process dump (64-bit ApplicationK, with application symbols) .................126
Exercise P6: Analysis of application process dump (ApplicationL, 32-bit) ...............................................................131
Exercise P7: Analysis of an application process dump (ApplicationL, 64-bit) ..........................................................140
Exercise P8: Analysis of an application process dump (ApplicationM, 64-bit) ........................................................148
Exercise P9: Analysis of an application process dump (ApplicationN, 64-bit) .........................................................162
Exercise P10: Analysis of an application process dump (ApplicationO, 64-bit) .......................................................174
Exercise P11: Analysis of an application process dump (ApplicationP, 64-bit) .......................................................184
Exercise P12: Analysis of an application process dump (ApplicationR, 32-bit) .......................................................199
Exercise P13: Analysis of an application process dump (ApplicationA, 64-bit) .......................................................217
Exercise P14: Analysis of an application process dump (ApplicationS, 64-bit) ........................................................225
Exercise P15: Analysis of an application process dump (notepad, 32-bit) ..............................................................238
Exercise P16: Analysis of an application process dump (notepad, 64-bit) ..............................................................242
Exercise P17: Analysis of an application process dump (ApplicationQ, 32-bit) .......................................................249
Exercise K1: Analysis of a normal kernel dump (64-bit) ..........................................................................................262
Exercise K2: Analysis of a kernel dump with pool leak (64-bit) ...............................................................................308
Exercise K3: Analysis of a kernel dump with pool corruption (64-bit) ....................................................................326
Exercise K4: Analysis of a kernel dump with code corruption (64-bit) ....................................................................335
Exercise K5: Analysis of a kernel dump with hang I/O (64-bit) ...............................................................................359
Exercise C1: Analysis of a normal complete dump (64-bit) .....................................................................................379
Exercise C2: Analysis of a problem complete dump (64-bit) ...................................................................................400
Exercise C3: Analysis of a problem complete dump (64-bit) ...................................................................................424
Exercise C4: Analysis of a problem complete dump (64-bit) ...................................................................................441
Exercise A1: Analysis of a problem active dump (64-bit) ........................................................................................463
Legacy Exercises .........................................................................................................................................................485
Exercise Legacy.0 ....................................................................................................................................................487
3
 Exercise Legacy.P1: Analysis of a normal application process dump (32-bit notepad) ...........................................492
Exercise Legacy.P2: Analysis of a normal application process dump (64-bit notepad) ...........................................513
Exercise Legacy.P3: Analysis of a normal application process dump (32-bit IE) ......................................................522
Exercise Legacy.P4: Analysis of an application process dump (32-bit ApplicationK, no symbols) ...........................537
Exercise Legacy.P5: Analysis of an application process dump (32-bit ApplicationK, with application symbols) .....547
Exercise Legacy.P6: Analysis of application process dump (ApplicationL, 32-bit) ...................................................551
Exercise Legacy.P7: Analysis of an application process dump (ApplicationL, 64-bit) ..............................................558
Exercise Legacy.P8: Analysis of an application process dump (ApplicationM, 32-bit) ............................................562
Exercise Legacy.P9: Analysis of an application process dump (ApplicationN, 64-bit) .............................................572
Exercise Legacy.P10: Analysis of an application process dump (ApplicationO, 64-bit) ...........................................580
Exercise Legacy.P11: Analysis of an application process dump (ApplicationP, 32-bit) ............................................586
Exercise Legacy.P13: Analysis of an application process dump (ApplicationA, 32-bit) ...........................................597
Exercise Legacy.P14: Analysis of an application process dump (ApplicationS, 32-bit) ............................................605
Exercise Legacy.P15: Analysis of an application process dump (notepad, 32-bit) ..................................................614
Exercise Legacy.P16: Analysis of an application process dump (notepad, 64-bit) ..................................................618
Exercise Legacy.P17: Analysis of an application process dump (ApplicationQ, 32-bit) ...........................................624
Exercise Legacy.K1: Analysis of a normal kernel dump (32-bit) ..............................................................................633
Exercise Legacy.K2: Analysis of a kernel dump with pool leak (32-bit) ...................................................................670
Exercise Legacy.K3: Analysis of a kernel dump with pool corruption (32-bit) .........................................................689
Exercise Legacy.K4: Analysis of a kernel dump with code corruption (32-bit) ........................................................701
Exercise Legacy.K5: Analysis of a kernel dump with hang I/O (32-bit) ....................................................................715
Exercise Legacy.C1: Analysis of a normal complete dump (32-bit) .........................................................................728
Exercise Legacy.C2: Analysis of a problem complete dump (32-bit) .......................................................................748
Application Source Code ............................................................................................................................................783
ApplicationA ...........................................................................................................................................................785
ApplicationB ...........................................................................................................................................................787
ApplicationC ...........................................................................................................................................................789
ApplicationE ...........................................................................................................................................................791
ApplicationK ...........................................................................................................................................................793
ApplicationL ............................................................................................................................................................794
ApplicationM ..........................................................................................................................................................795
ApplicationN ...........................................................................................................................................................796
ApplicationO ...........................................................................................................................................................797
ApplicationP ...........................................................................................................................................................798
ApplicationR ...........................................................................................................................................................799
4
 ApplicationS............................................................................................................................................................800
ApplicationQ ...........................................................................................................................................................801
Selected Q&A .............................................................................................................................................................805
Minidump Analysis .....................................................................................................................................................849
Scripts and WinDbg Commands .............................................................................................................................849
Component Identification.......................................................................................................................................852
Raw Stack Data Analysis .........................................................................................................................................857
Symbols and Images ...............................................................................................................................................866
Wait Chain (Executive Resources) ..............................................................................................................................869

5
 Exercise P1: Analysis of a normal application process dump (32-bit notepad)

Goal: Learn how to see dump file type and version, get a stack trace, check its correctness, perform default analysis,
list modules, check their version information, check process environment.

Patterns: Manual Dump; Stack Trace; Not My Version; Environment Hint.

1. Launch WinDbg from Windows Kits \ WinDbg (X64).

2. Open \AWMDA-Dumps\x86\Processes\notepad.DMP.

3. We get the dump file loaded:

47
4. Open a log file to save all future output using .logopen command:

Note: You can type any comment by using the * command.

5. Type the command .symfix c:\mss to set a path to download symbol files from Microsoft symbol file server:

48
6. Type .reload command to download symbols if necessary:

7. Type k command to verify the correctness of the stack trace:

49
8. Type version command to get OS version, system and process uptimes, the dump file timestamp and its type:

50
Note: This is the full output:

0:000> version
Windows 10 Version 10240 MP (4 procs) Free x86 compatible
Product: WinNt, suite: SingleUserTS Personal
kernel32.dll version: 10.0.10240.16384 (th1.150709-1700)
Machine Name:
Debug session time: Sun May 1 16:07:18.000 2016 (UTC + 1:00)
System Uptime: 1 days 2:47:47.329
Process Uptime: 0 days 0:00:31.000
Kernel time: 0 days 0:00:00.000
User time: 0 days 0:00:00.000
Full memory user mini dump: C:\AWMDA-Dumps\x86\Processes\notepad.DMP

Microsoft (R) Windows Debugger Version 10.0.10586.567 AMD64

Copyright (c) Microsoft Corporation. All rights reserved.

command line: '"C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\windbg.exe" ' Debugger

Process 0x2B54
dbgeng: image 10.0.10586.15, built Fri Nov 20 04:56:41 2015
[path: C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\dbgeng.dll]
dbghelp: image 10.0.10586.15, built Fri Nov 20 04:55:01 2015
[path: C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\dbghelp.dll]
DIA version: 40116
Extension DLL search Path:
C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\WINXP;C:\Program Files (x86)\Windows
Kits\10\Debuggers\x64\winext;C:\Program Files (x86)\Windows
Kits\10\Debuggers\x64\winext\arcade;C:\Program Files (x86)\Windows
Kits\10\Debuggers\x64\pri;C:\Program Files (x86)\Windows Kits\10\Debuggers\x64;C:\Program Files
(x86)\Windows
Kits\10\Debuggers\x64\winext\arcade;C:\ProgramData\Oracle\Java\javapath;C:\Program Files\Common
Files\Microsoft Shared\Windows Live;C:\Program Files (x86)\Common Files\Microsoft
Shared\Windows Live;C:\Program Files (x86)\Intel\iCLS Client\;C:\Program Files\Intel\iCLS
51
Client\;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowe
rShell\v1.0\;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program
Files\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files (x86)\Intel\Intel(R)
Management Engine Components\DAL;C:\Program Files (x86)\Intel\Intel(R) Management Engine
Components\IPT;C:\Program Files (x86)\Intel\OpenCL SDK\2.0\bin\x86;C:\Program Files
(x86)\Intel\OpenCL SDK\2.0\bin\x64;C:\Program Files\Intel\WiFi\bin\;C:\Program Files\Common
Files\Intel\WirelessCommon\;C:\Program Files (x86)\Windows Live\Shared;C:\Program
Files\Microsoft\Web Platform Installer\;C:\Program Files (x86)\Microsoft ASP.NET\ASP.NET Web
Pages\v1.0\;C:\Program Files (x86)\Symantec\VIP Access
Client\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowe
rShell\v1.0\;C:\Program Files (x86)\Skype\Phone\;C:\Program Files (x86)\Windows
Kits\8.1\Windows Performance Toolkit\
Extension DLL chain:
dbghelp: image 10.0.10586.15, API 10.0.6, built Fri Nov 20 04:55:01 2015
[path: C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\dbghelp.dll]
ext: image 10.0.10586.15, API 1.0.0, built Fri Nov 20 04:55:08 2015
[path: C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\winext\ext.dll]
exts: image 10.0.10586.15, API 1.0.0, built Fri Nov 20 04:54:07 2015
[path: C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\WINXP\exts.dll]
uext: image 10.0.10586.15, API 1.0.0, built Fri Nov 20 04:54:02 2015
[path: C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\winext\uext.dll]
ntsdexts: image 10.0.10586.15, API 1.0.0, built Fri Nov 20 05:28:14 2015
[path: C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\WINXP\ntsdexts.dll]

Note: Debug session time is when the dump was generated. Although the dump is called “mini dump” it is a full
memory user dump with all process memory included.

9. Type the default analysis command !analyze -v:

52
53
Note: This (or.reload command) may take some time initially as symbols are downloaded from the symbol server:

54
10. Let’s now look at the output in more detail:

0:000> !analyze -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************

DUMP_CLASS: 2

DUMP_QUALIFIER: 400

FAULTING_IP:
+0
00000000 ?? ???

EXCEPTION_RECORD: (.exr -1)

ExceptionAddress: 00000000
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 0

FAULTING_THREAD: 00003078

DEFAULT_BUCKET_ID: STATUS_BREAKPOINT

PROCESS_NAME: notepad.exe

ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION} Breakpoint A breakpoint has been reached.

EXCEPTION_CODE: (HRESULT) 0x80000003 (2147483651) - One or more arguments are invalid

EXCEPTION_CODE_STR: 80000003

WATSON_BKT_PROCSTAMP: 55bebe90

WATSON_BKT_PROCVER: 10.0.10240.16425

PROCESS_VER_PRODUCT: Microsoft® Windows® Operating System

WATSON_BKT_MODULE: unknown

WATSON_BKT_MODVER: 0.0.0.0

WATSON_BKT_MODOFFSET: 0

WATSON_BKT_MODSTAMP: bbbbbbb4

BUILD_VERSION_STRING: 10.0.10240.16384 (th1.150709-1700)

MODLIST_WITH_TSCHKSUM_HASH: 409dc00a3b07a0619d19699aaf2ad34995696fba

MODLIST_SHA1_HASH: a2b8dbdc12e291e73566ab6765f5a7461a85a26b

NTGLOBALFLAG: 400

APPLICATION_VERIFIER_FLAGS: 0

55
PRODUCT_TYPE: 1

SUITE_MASK: 784

DUMP_FLAGS: 8000c07

DUMP_TYPE: 0

APP: notepad.exe

ANALYSIS_SESSION_HOST: TRAINING-PC

ANALYSIS_SESSION_TIME: 05-01-2016 19:08:54.0766

ANALYSIS_VERSION: 10.0.10586.567 amd64fre

THREAD_ATTRIBUTES:
OS_LOCALE: ENU

PROBLEM_CLASSES:

Tid [0x0]
Frame [0x00]
String [STATUS_BREAKPOINT]
Data Bucketing

BUGCHECK_STR: STATUS_BREAKPOINT

LAST_CONTROL_TRANSFER: from 74d7325a to 74d74d9c

STACK_TEXT:
04ebf8e0 74d7325a 04ebf920 00000000 00000000 user32!NtUserGetMessage+0xc
04ebf8fc 009e5eb6 04ebf920 00000000 00000000 user32!GetMessageW+0x2a
04ebf93c 009f5b41 009e0000 00000000 05134032 notepad!WinMain+0xe6
04ebf9d0 749e3744 7e3da000 749e3720 0b053f62 notepad!WinMainCRTStartup+0x151
04ebf9e4 773e9e54 7e3da000 1c64488a 00000000 kernel32!BaseThreadInitThunk+0x24
04ebfa2c 773e9e1f ffffffff 7740d6d6 00000000 ntdll!__RtlUserThreadStart+0x2f
04ebfa3c 00000000 009f59f0 7e3da000 00000000 ntdll!_RtlUserThreadStart+0x1b

STACK_COMMAND: ~0s; .ecxr ; kb

THREAD_SHA1_HASH_MOD_FUNC: 938dec2050a1e4605831341df0b0049900cc489a

THREAD_SHA1_HASH_MOD_FUNC_OFFSET: 48302f2507a707f990bbcb69a94480fc874178b2

THREAD_SHA1_HASH_MOD: 77973f77be56c743a9806c895e818a3dc0c6b5f2

FOLLOWUP_IP:
notepad!WinMain+e6
009e5eb6 85c0 test eax,eax

FAULT_INSTR_CODE: 9075c085

SYMBOL_STACK_INDEX: 2

56
SYMBOL_NAME: notepad!WinMain+e6

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: notepad

IMAGE_NAME: notepad.exe

DEBUG_FLR_IMAGE_TIMESTAMP: 55bebe90

BUCKET_ID: STATUS_BREAKPOINT_notepad!WinMain+e6

PRIMARY_PROBLEM_CLASS: STATUS_BREAKPOINT_notepad!WinMain+e6

BUCKET_ID_OFFSET: e6

BUCKET_ID_MODULE_STR: notepad

BUCKET_ID_MODTIMEDATESTAMP: 55bebe90

BUCKET_ID_MODCHECKSUM: 37c17

BUCKET_ID_MODVER_STR: 10.0.10240.16425

BUCKET_ID_PREFIX_STR: STATUS_BREAKPOINT_

FAILURE_PROBLEM_CLASS: STATUS_BREAKPOINT

FAILURE_EXCEPTION_CODE: 80000003

FAILURE_IMAGE_NAME: notepad.exe

FAILURE_FUNCTION_NAME: WinMain

BUCKET_ID_FUNCTION_STR: WinMain

FAILURE_SYMBOL_NAME: notepad.exe!WinMain

FAILURE_BUCKET_ID: STATUS_BREAKPOINT_80000003_notepad.exe!WinMain

WATSON_STAGEONE_URL:
http://watson.microsoft.com/StageOne/notepad.exe/10.0.10240.16425/55bebe90/unknown/0.0.0.0/bbbb
bbb4/80000003/00000000.htm?Retriage=1

TARGET_TIME: 2016-05-01T15:07:18.000Z

OSBUILD: 10240

OSSERVICEPACK: 16384

SERVICEPACK_NUMBER: 0

OS_REVISION: 0

OSPLATFORM_TYPE: x86

OSNAME: Windows 10

OSEDITION: Windows 10 WinNt SingleUserTS Personal

57
USER_LCID: 0

OSBUILD_TIMESTAMP: 2015-07-10 04:25:21

BUILDDATESTAMP_STR: 150709-1700

BUILDLAB_STR: th1

BUILDOSVER_STR: 10.0.10240.16384

ANALYSIS_SESSION_ELAPSED_TIME: 1e4

ANALYSIS_SOURCE: UM

FAILURE_ID_HASH_STRING: um:status_breakpoint_80000003_notepad.exe!winmain

FAILURE_ID_HASH: {39352512-8c1c-b033-4491-409b6d85420b}

Followup: MachineOwner
---------

Note: “Break instruction exception“ can be the sign of Manual Dump pattern but often WinDbg is not able to figure
out an exception which may be on another thread or hidden.

11. Now we check how many threads by using ~ command:

58
12. Now we dump a stack trace using kc command (only modules and symbols):

59
13. Now we dump the stack trace of the current thread using k command (with symbols, return addresses, and
function offsets):

60
0:000> k
# ChildEBP RetAddr
00 04ebf8e0 74d7325a user32!NtUserGetMessage+0xc
01 04ebf8fc 009e5eb6 user32!GetMessageW+0x2a
02 04ebf93c 009f5b41 notepad!WinMain+0xe6
03 04ebf9d0 749e3744 notepad!WinMainCRTStartup+0x151
04 04ebf9e4 773e9e54 kernel32!BaseThreadInitThunk+0x24
05 04ebfa2c 773e9e1f ntdll!__RtlUserThreadStart+0x2f
06 04ebfa3c 00000000 ntdll!_RtlUserThreadStart+0x1b

Hint: How to check that the stack trace is correct. Use ub command (unassemble backwards) to check if there is a
call instruction. We check that GetMessageW function was called from WinMain function:

0:000> k
# ChildEBP RetAddr
00 04ebf8e0 74d7325a user32!NtUserGetMessage+0xc
01 04ebf8fc 009e5eb6 user32!GetMessageW+0x2a
02 04ebf93c 009f5b41 notepad!WinMain+0xe6
03 04ebf9d0 749e3744 notepad!WinMainCRTStartup+0x151
04 04ebf9e4 773e9e54 kernel32!BaseThreadInitThunk+0x24
05 04ebfa2c 773e9e1f ntdll!__RtlUserThreadStart+0x2f
06 04ebfa3c 00000000 ntdll!_RtlUserThreadStart+0x1b3

61
0:000> ub 009e5eb6
notepad!WinMain+0xd2:
009e5ea2 50 push eax
009e5ea3 ff15b8a19f00 call dword ptr [notepad!_imp__DispatchMessageW (009fa1b8)]
009e5ea9 53 push ebx
009e5eaa 53 push ebx
009e5eab 53 push ebx
009e5eac 8d45e4 lea eax,[ebp-1Ch]
009e5eaf 50 push eax
009e5eb0 ff15a8a19f00 call dword ptr [notepad!_imp__GetMessageW (009fa1a8)]

Then we check that NtUserGetMessage function was called from GetMessageW function:

0:000> k
# ChildEBP RetAddr
00 04ebf8e0 74d7325a user32!NtUserGetMessage+0xc
01 04ebf8fc 009e5eb6 user32!GetMessageW+0x2a
02 04ebf93c 009f5b41 notepad!WinMain+0xe6
03 04ebf9d0 749e3744 notepad!WinMainCRTStartup+0x151
04 04ebf9e4 773e9e54 kernel32!BaseThreadInitThunk+0x24
05 04ebfa2c 773e9e1f ntdll!__RtlUserThreadStart+0x2f
06 04ebfa3c 00000000 ntdll!_RtlUserThreadStart+0x1b

0:000> ub 74d7325a
user32!GetMessageW+0x15:
74d73245 0f85c7cc0100 jne user32!GetMessageW+0x1cce2 (74d8ff12)
74d7324b 56 push esi
74d7324c 8b7508 mov esi,dword ptr [ebp+8]
74d7324f 50 push eax
74d73250 52 push edx
74d73251 ff750c push dword ptr [ebp+0Ch]
74d73254 56 push esi
74d73255 e8361b0000 call user32!NtUserGetMessage (74d74d90)

62
14. Now we dump the stack trace using verbose kv command (includes the first possible function parameters):

Note: Remember the functions call each other from bottom to top. The topmost function is the last one that was
called. ExceptionAddress or FAULTING_IP may point to the last one. We would come to this in the real exception
process dumps later. Here in another example below I would like to point out that the top function call func1 has a
return address already (to func2), and the function was being executed somewhere in its code at 0x20 offset:

63
0:000> k
ChildEBP RetAddr
0024f9a0 772c199a ModuleA!func1+0x20
0024f9a4 772c19cd ModuleA!func2+0x16
[...]
0024fa9c 776fa9bd kernel32!BaseThreadInitThunk+0xe
0024fadc 00000000 ntdll!_RtlUserThreadStart+0x23

15. Now we check the list of loaded modules using lm command:

64
16. We can check verbose module information using lmv command or use lmv m <module name> to check an
individual module (Not My Version pattern):

65
17. Sometimes lmv command doesn’t show much and !lmi command might give extra information:

66
Note: We can also use lmt command variant if we are interested in timestamps only.

18. Sometimes Environment Hint pattern can give troubleshooting suggestions related to environment variables
and DLL paths. !peb command (Process Environment Block):

0:000> !peb
PEB at 7e3da000
InheritedAddressSpace: No
ReadImageFileExecOptions: No
BeingDebugged: No
ImageBaseAddress: 009e0000
Ldr 77498b40
Ldr.Initialized: Yes
Ldr.InInitializationOrderModuleList: 051337b0 . 0513adf8
Ldr.InLoadOrderModuleList: 05133880 . 0513ade8
Ldr.InMemoryOrderModuleList: 05133888 . 0513adf0
Base TimeStamp Module
9e0000 55bebe90 Aug 03 02:06:24 2015 C:\Windows\SysWOW64\notepad.exe
77390000 56ad9358 Jan 31 04:53:44 2016 C:\WINDOWS\SYSTEM32\ntdll.dll
749d0000 559f3b21 Jul 10 04:25:21 2015 C:\WINDOWS\SYSTEM32\KERNEL32.DLL
758a0000 56e8cf1c Mar 16 03:12:28 2016 C:\WINDOWS\SYSTEM32\KERNELBASE.dll
75770000 568b1dff Jan 05 01:35:59 2016 C:\WINDOWS\SYSTEM32\ADVAPI32.dll
75460000 559f3e0e Jul 10 04:37:50 2015 C:\WINDOWS\SYSTEM32\msvcrt.dll
75850000 559f3afd Jul 10 04:24:45 2015 C:\WINDOWS\SYSTEM32\sechost.dll
75b10000 55b992ea Jul 30 03:58:50 2015 C:\WINDOWS\SYSTEM32\RPCRT4.dll
74440000 559f3af4 Jul 10 04:24:36 2015 C:\WINDOWS\SYSTEM32\SspiCli.dll
74430000 559f3af8 Jul 10 04:24:40 2015 C:\WINDOWS\SYSTEM32\CRYPTBASE.dll
743d0000 559f3c0f Jul 10 04:29:19 2015 C:\WINDOWS\SYSTEM32\bcryptPrimitives.dll
771f0000 568b1b15 Jan 05 01:23:33 2016 C:\WINDOWS\SYSTEM32\GDI32.dll
74d40000 56553339 Nov 25 04:04:09 2015 C:\WINDOWS\SYSTEM32\USER32.dll
75bc0000 56ad9664 Jan 31 05:06:44 2016 C:\WINDOWS\SYSTEM32\combase.dll
75530000 559f3b0b Jul 10 04:24:59 2015 C:\WINDOWS\SYSTEM32\OLEAUT32.dll
67
 745d0000 5655342b Nov 25 04:08:11 2015 C:\WINDOWS\SYSTEM32\COMDLG32.dll
74cb0000 559f3d59 Jul 10 04:34:49 2015 C:\WINDOWS\SYSTEM32\shcore.dll
72b80000 559f3e45 Jul 10 04:38:45 2015 C:\WINDOWS\WinSxS\x86_microsoft.windows.common-
controls_6595b64144ccf1df_6.0.10240.16384_none_3bccb1ff6bcd1849\COMCTL32.dll
75720000 559f3c42 Jul 10 04:30:10 2015 C:\WINDOWS\SYSTEM32\SHLWAPI.dll
75df0000 56e8d63b Mar 16 03:42:51 2016 C:\WINDOWS\SYSTEM32\SHELL32.dll
74f80000 55fa574f Sep 17 07:01:51 2015 C:\WINDOWS\SYSTEM32\windows.storage.dll
757f0000 559f3aff Jul 10 04:24:47 2015 C:\WINDOWS\SYSTEM32\kernel.appcore.dll
75800000 559f3aff Jul 10 04:24:47 2015 C:\WINDOWS\SYSTEM32\powrprof.dll
74690000 559f3af5 Jul 10 04:24:37 2015 C:\WINDOWS\SYSTEM32\profapi.dll
730d0000 559f3c05 Jul 10 04:29:09 2015 C:\Windows\SYSTEM32\WINSPOOL.DRV
73d90000 559f3c18 Jul 10 04:29:28 2015 C:\Windows\SYSTEM32\bcrypt.dll
756f0000 559f3b8d Jul 10 04:27:09 2015 C:\WINDOWS\SYSTEM32\IMM32.DLL
74850000 56ad94ab Jan 31 04:59:23 2016 C:\WINDOWS\SYSTEM32\MSCTF.dll
72dc0000 55af08da Jul 22 04:07:06 2015 C:\WINDOWS\system32\uxtheme.dll
10000000 4c31b72f Jul 05 11:42:55 2010 C:\Program Files (x86)\Samsung\Easy
Settings\WinCRT.dll
71e70000 55a862ea Jul 17 03:05:30 2015 C:\WINDOWS\system32\dwmapi.dll
75a20000 56cc3889 Feb 23 10:46:33 2016 C:\WINDOWS\SYSTEM32\ole32.dll
755d0000 559f3cb0 Jul 10 04:32:00 2015 C:\WINDOWS\SYSTEM32\clbcatq.dll
SubSystemData: 00000000
ProcessHeap: 05130000
ProcessParameters: 05131b98
CurrentDirectory: 'C:\Windows\SysWOW64\'
WindowTitle: 'C:\Windows\SysWOW64\notepad.exe'
ImageFile: 'C:\Windows\SysWOW64\notepad.exe'
CommandLine: '"C:\Windows\SysWOW64\notepad.exe" '
DllPath: '< Name not readable >'
Environment: 051305c8
=::=::\
ALLUSERSPROFILE=C:\ProgramData
APPDATA=C:\Users\Training\AppData\Roaming
asl.log=Destination=file
CommonProgramFiles=C:\Program Files (x86)\Common Files
CommonProgramFiles(x86)=C:\Program Files (x86)\Common Files
CommonProgramW6432=C:\Program Files\Common Files
COMPUTERNAME=TRAINING-PC
ComSpec=C:\WINDOWS\system32\cmd.exe
FPS_BROWSER_APP_PROFILE_STRING=Internet Explorer
FPS_BROWSER_USER_PROFILE_STRING=Default
FP_NO_HOST_CHECK=NO
HOMEDRIVE=C:
HOMEPATH=\Users\Training
LOCALAPPDATA=C:\Users\Training\AppData\Local
LOGONSERVER=\\TRAINING-PC
NUMBER_OF_PROCESSORS=4
OS=Windows_NT
Path=C:\ProgramData\Oracle\Java\javapath;C:\Program Files\Common Files\Microsoft
Shared\Windows Live;C:\Program Files (x86)\Common Files\Microsoft Shared\Windows
Live;C:\Program Files (x86)\Intel\iCLS Client\;C:\Program Files\Intel\iCLS
Client\;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowe
rShell\v1.0\;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program
Files\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files (x86)\Intel\Intel(R)
Management Engine Components\DAL;C:\Program Files (x86)\Intel\Intel(R) Management Engine
Components\IPT;C:\Program Files (x86)\Intel\OpenCL SDK\2.0\bin\x86;C:\Program Files
(x86)\Intel\OpenCL SDK\2.0\bin\x64;C:\Program Files\Intel\WiFi\bin\;C:\Program Files\Common
Files\Intel\WirelessCommon\;C:\Program Files (x86)\Windows Live\Shared;C:\Program
Files\Microsoft\Web Platform Installer\;C:\Program Files (x86)\Microsoft ASP.NET\ASP.NET Web
Pages\v1.0\;C:\Program Files (x86)\Symantec\VIP Access
Client\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowe

68
rShell\v1.0\;C:\Program Files (x86)\Skype\Phone\;C:\Program Files (x86)\Windows
Kits\8.1\Windows Performance Toolkit\
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
PROCESSOR_ARCHITECTURE=x86
PROCESSOR_ARCHITEW6432=AMD64
PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 58 Stepping 9, GenuineIntel
PROCESSOR_LEVEL=6
PROCESSOR_REVISION=3a09
ProgramData=C:\ProgramData
ProgramFiles=C:\Program Files (x86)
ProgramFiles(x86)=C:\Program Files (x86)
ProgramW6432=C:\Program Files
PSModulePath=C:\WINDOWS\system32\WindowsPowerShell\v1.0\Modules\
PUBLIC=C:\Users\Public
SESSIONNAME=Console
SystemDrive=C:
SystemRoot=C:\WINDOWS
TEMP=C:\Users\Training\AppData\Local\Temp
TMP=C:\Users\Training\AppData\Local\Temp
USERDOMAIN=TRAINING-PC
USERDOMAIN_ROAMINGPROFILE=TRAINING-PC
USERNAME=Training
USERPROFILE=C:\Users\Training
VS110COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Tools\
VS140COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\Tools\
windir=C:\WINDOWS
windows_tracing_flags=3
windows_tracing_logfile=C:\BVTBin\Tests\installpackage\csilogfile.log

19. We close logging before exiting WinDbg:

69
Note: To avoid possible confusion and glitches we recommend exiting WinDbg after each exercise.

70
Published by OpenTask, Republic of Ireland

Copyright © 2018 by OpenTask

Copyright © 2018 by Software Diagnostics Services

Copyright © 2018 by Dmitry Vostokov

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or
transmitted, in any form or by any means, without the prior written permission of the
publisher.

Product and company names mentioned in this book may be trademarks of their owners.

OpenTask books and magazines are available through booksellers and distributors worldwide.
For further information or comments send requests to [email protected].

A CIP catalog record for this book is available from the British Library.

ISBN-l3: 978-1-908043-87-0 (Paperback)

Revision 3.0 (August 2018)

2
Contents

About the Author.............................................................................................................................................................. 5

Introduction ...................................................................................................................................................................... 7
Practice Exercises ........................................................................................................................................................... 23
Exercise 0: Download, setup and verify your WinDbg installation ............................................................................ 28
Exercise PN1: Analysis of an application process dump (ApplicationA, 64-bit) ......................................................... 37
Exercise PN2: Analysis of an application process dump (ApplicationA, 32-bit) ......................................................... 56
Exercise PN3: Analysis of an application process dump (LINQPadB, 64-bit).............................................................. 72
Exercise PN4: Analysis of an application process dump (LINQPadB, 32-bit).............................................................. 95
Exercise PN5: Analysis of an application process dump (LINQPadC, 64-bit)............................................................ 118
Exercise PN6: Analysis of an application process dump (LINQPadC, 32-bit)............................................................ 133
Exercise PN7: Analysis of an application process dump (ApplicationD, 64-bit) ....................................................... 152
Exercise PN8: Analysis of an application process dump (ApplicationD, 32-bit) ....................................................... 179
Exercise PN9: Analysis of an application process dump (LINQPadD, 64-bit) ........................................................... 194
Exercise PN10: Analysis of an application process dump (LINQPadD, 32-bit) ......................................................... 210
Exercise PN11: Analysis of an application process dump (LINQPadE, 64-bit) .......................................................... 227
Exercise PN12: Analysis of an application process dump (LINQPadE, 32-bit) .......................................................... 237
Legacy Exercises ........................................................................................................................................................... 253
Exercise Legacy.0: Download, setup and verify your WinDbg installation .............................................................. 255
Exercise Legacy.PN1: Analysis of an application process dump (ApplicationA, 32-bit, CLR2) ................................. 260
Exercise Legacy.PN2: Analysis of an application process dump (ApplicationA, 32-bit, CLR4) ................................. 270
Exercise Legacy.PN3: Analysis of an application process dump (LINQPadB, 64-bit, CLR4) ...................................... 284
Exercise Legacy.PN4: Analysis of an application process dump (LINQPadB, 32-bit, CLR2) ...................................... 306
Exercise Legacy.PN5: Analysis of an application process dump (LINQPadC, 64-bit, CLR4) ...................................... 324
Exercise Legacy.PN6: Analysis of an application process dump (LINQPadC, 32-bit, CLR4) ...................................... 344
Exercise Legacy.PN7: Analysis of an application process dump (LINQPadD, 32-bit, CLR4)...................................... 364
Exercise Legacy.PN8: Analysis of an application process dump (LINQPadE, 32-bit, CLR4) ...................................... 403
Application Source Code .............................................................................................................................................. 413
ApplicationA ............................................................................................................................................................. 415
LinqB ......................................................................................................................................................................... 416
LinqC ......................................................................................................................................................................... 417
ApplicationD ............................................................................................................................................................. 419
LinqD ......................................................................................................................................................................... 421
LinqE ......................................................................................................................................................................... 423
3
Selected Q&A................................................................................................................................................................ 425

4
 Exercise PN1: Analysis of an application process dump (ApplicationA, 64-bit)

Goal: Learn how to load the correct .NET SOS WinDbg extension and analyze managed space.

Patterns: Stack Trace Collection; CLR Thread; Version-Specific Extension; Software Exception, Exception Stack Trace,
Managed Code Exception; Managed Stack Trace.

Commands: .logopen, .symfix, .reload, ~*k, .load, !pe, ~*e, lmv, .chain, .unload, !analyze -v, !CLRStack, .logclose

1. Launch WinDbg from Windows Kits \ WinDbg (X64).

2. Open \ANETMDA-Dumps\Processes\ApplicationA.DMP

3. We get the dump file loaded:

Note: ApplicationA shows this dialog when launched:

37
When we click on a button it shows the following exception dialog:

At this point, we saved a process memory dump on a Windows 10 x64 system using Task Manager.

4. Open a log file using .logopen command and load symbols (.symfix and .reload commands):

0:000> .logopen C:\ANETMDA-Dumps\Processes\ApplicationA.log

Opened log file 'C:\ANETMDA-Dumps\Processes\ApplicationA.log'

0:000> .symfix c:\mss

0:000> .reload
............................................................
Loading unloaded module list
.
*** WARNING: Unable to verify checksum for System.Windows.Forms.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for
System.Windows.Forms.ni.dll

************* Symbol Loading Error Summary **************

Module name Error
System.Windows.Forms.n 0x80190194 - Not found (404). :
SRV*c:\mss*https://msdl.microsoft.com/download/symbols

You can troubleshoot most symbol related issues by turning on symbol loading diagnostics (!sym
noisy) and repeating the command that caused symbols to be loaded.
You should also verify that your symbol search path (.sympath) is correct.

Note: The results may be slightly different on your system if you don’t have .NET Framework 4.0.30319 installed or
you have a version different from 4.7.3120.0 that was on a virtual machine where all the dumps were saved.

38
5. Type ~*k command to verify the correctness of all stack traces (the command execution time may be longer
for the first time because symbol files need to be downloaded from Microsoft symbol server):

39
0:000> ~*k

. 0 Id: 7f0.22e0 Suspend: 0 Teb: 00000000`00fcc000 Unfrozen

# Child-SP RetAddr Call Site
00 00000000`0113bbc8 00007ffc`d8b933f8 win32u!NtUserWaitMessage+0x14
01 00000000`0113bbd0 00007ffc`d8b2f452 System_Windows_Forms_ni+0x2d33f8
02 00000000`0113bc80 00007ffc`d8b2ebd2 System_Windows_Forms_ni+0x26f452
03 00000000`0113bd70 00007ffc`d8b2e9df System_Windows_Forms_ni+0x26ebd2
04 00000000`0113be10 00007ffc`d9226bfd System_Windows_Forms_ni+0x26e9df
05 00000000`0113be70 00007ffc`d91f72f3 System_Windows_Forms_ni+0x966bfd
06 00000000`0113bf70 00007ffc`d920494a System_Windows_Forms_ni+0x9372f3
07 00000000`0113bfe0 00007ffc`d8b1a413 System_Windows_Forms_ni+0x94494a
08 00000000`0113c010 00007ffc`ef378a6d System_Windows_Forms_ni+0x25a413
09 00000000`0113c060 00007ffc`ef378934 clr!ExceptionTracker::CallHandler+0xfd
0a 00000000`0113c150 00007ffc`ef378848 clr!ExceptionTracker::CallCatchHandler+0x90
0b 00000000`0113c1f0 00007ffd`1918ed6d clr!ProcessCLRException+0x31c
0c 00000000`0113c2d0 00007ffd`190f7670 ntdll!RtlpExecuteHandlerForUnwind+0xd
0d 00000000`0113c300 00007ffc`ef379550 ntdll!RtlUnwindEx+0x3a0
0e 00000000`0113c9e0 00007ffc`ef37950b clr!ClrUnwindEx+0x40
0f 00000000`0113cf00 00007ffd`1918eced clr!ProcessCLRException+0x2e9
10 00000000`0113cfe0 00007ffd`190f6c86 ntdll!RtlpExecuteHandlerForException+0xd
11 00000000`0113d010 00007ffd`190f52ca ntdll!RtlDispatchException+0x3c6
12 00000000`0113d710 00007ffd`15d8a388 ntdll!RtlRaiseException+0x31a
13 00000000`0113df70 00007ffc`ef2b1209 KERNELBASE!RaiseException+0x68
14 00000000`0113e050 00007ffc`ef2b123b clr!NakedThrowHelper2+0x9
15 00000000`0113e080 00007ffc`ef2b1245 clr!NakedThrowHelper_RspAligned+0x1e
16 00000000`0113e5a8 00007ffc`8fcb0829 clr!NakedThrowHelper_FixRsp+0x5
17 00000000`0113e5b0 00007ffc`d8b060b2 0x00007ffc`8fcb0829
18 00000000`0113e5f0 00007ffc`d8b094cc System_Windows_Forms_ni+0x2460b2
19 00000000`0113e630 00007ffc`d92579cc System_Windows_Forms_ni+0x2494cc
1a 00000000`0113e680 00007ffc`d9204602 System_Windows_Forms_ni+0x9979cc
1b 00000000`0113e740 00007ffc`d8b1aebb System_Windows_Forms_ni+0x944602
1c 00000000`0113e7c0 00007ffc`d8b10234 System_Windows_Forms_ni+0x25aebb
1d 00000000`0113e880 00007ffc`d8b10184 System_Windows_Forms_ni+0x250234
1e 00000000`0113e900 00007ffc`d8b1a3c3 System_Windows_Forms_ni+0x250184
1f 00000000`0113e930 00007ffc`d91911f1 System_Windows_Forms_ni+0x25a3c3
20 00000000`0113e9d0 00007ffc`ef2b221e System_Windows_Forms_ni+0x8d11f1
21 00000000`0113ea40 00007ffd`17646cc1 clr!UMThunkStub+0x6e
22 00000000`0113ead0 00007ffd`17646693 user32!UserCallWinProcCheckWow+0x2c1
23 00000000`0113ec60 00007ffc`d8b9a378 user32!DispatchMessageWorker+0x1c3
24 00000000`0113ecf0 00007ffc`d8b2f23e System_Windows_Forms_ni+0x2da378
25 00000000`0113edb0 00007ffc`d8b2ebd2 System_Windows_Forms_ni+0x26f23e
26 00000000`0113eea0 00007ffc`d8b2e9df System_Windows_Forms_ni+0x26ebd2
27 00000000`0113ef40 00007ffc`8fcb04d2 System_Windows_Forms_ni+0x26e9df
28 00000000`0113efa0 00007ffc`ef2b6bb3 0x00007ffc`8fcb04d2
29 00000000`0113efe0 00007ffc`ef2b6a70 clr!CallDescrWorkerInternal+0x83
2a 00000000`0113f020 00007ffc`ef2b735d clr!CallDescrWorkerWithHandler+0x4e
2b 00000000`0113f060 00007ffc`ef30ec1c clr!MethodDescCallSite::CallTargetWorker+0xf8
2c 00000000`0113f160 00007ffc`ef30ee06 clr!RunMain+0x1e7
2d 00000000`0113f340 00007ffc`ef30ecfb clr!Assembly::ExecuteMainMethod+0xb6
2e 00000000`0113f630 00007ffc`ef30eaf4 clr!SystemDomain::ExecuteMainMethod+0x57c
2f 00000000`0113fc40 00007ffc`ef30ea72 clr!ExecuteEXE+0x3f
30 00000000`0113fcb0 00007ffc`ef30ef34 clr!_CorExeMainInternal+0xb2
31 00000000`0113fd40 00007ffc`efca7b2d clr!CorExeMain+0x14
32 00000000`0113fd80 00007ffc`f52ba4cc mscoreei!CorExeMain+0x112
33 00000000`0113fde0 00007ffd`165c3034 mscoree!CorExeMain_Exported+0x6c
34 00000000`0113fe10 00007ffd`19161431 kernel32!BaseThreadInitThunk+0x14
35 00000000`0113fe40 00000000`00000000 ntdll!RtlUserThreadStart+0x21

40
 1 Id: 7f0.2038 Suspend: 0 Teb: 00000000`00fce000 Unfrozen
# Child-SP RetAddr Call Site
00 00000000`0133f858 00007ffd`1910f856 ntdll!NtWaitForWorkViaWorkerFactory+0x14
01 00000000`0133f860 00007ffd`165c3034 ntdll!TppWorkerThread+0x536
02 00000000`0133fb50 00007ffd`19161431 kernel32!BaseThreadInitThunk+0x14
03 00000000`0133fb80 00000000`00000000 ntdll!RtlUserThreadStart+0x21

2 Id: 7f0.203c Suspend: 0 Teb: 00000000`00fd0000 Unfrozen

# Child-SP RetAddr Call Site
00 00000000`0154f538 00007ffd`1910f856 ntdll!NtWaitForWorkViaWorkerFactory+0x14
01 00000000`0154f540 00007ffd`165c3034 ntdll!TppWorkerThread+0x536
02 00000000`0154f830 00007ffd`19161431 kernel32!BaseThreadInitThunk+0x14
03 00000000`0154f860 00000000`00000000 ntdll!RtlUserThreadStart+0x21

3 Id: 7f0.2040 Suspend: 0 Teb: 00000000`00fd2000 Unfrozen

# Child-SP RetAddr Call Site
00 00000000`02ebf438 00007ffd`15d96099 ntdll!NtWaitForMultipleObjects+0x14
01 00000000`02ebf440 00007ffc`ef346a42 KERNELBASE!WaitForMultipleObjectsEx+0xf9
02 00000000`02ebf740 00007ffc`ef34696d clr!DebuggerRCThread::MainLoop+0xce
03 00000000`02ebf800 00007ffc`ef346880 clr!DebuggerRCThread::ThreadProc+0xd2
04 00000000`02ebf850 00007ffd`165c3034 clr!DebuggerRCThread::ThreadProcStatic+0x41
05 00000000`02ebf8a0 00007ffd`19161431 kernel32!BaseThreadInitThunk+0x14
06 00000000`02ebf8d0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

4 Id: 7f0.2058 Suspend: 0 Teb: 00000000`00fd4000 Unfrozen

# Child-SP RetAddr Call Site
00 00000000`1b4af7b8 00007ffd`15d96099 ntdll!NtWaitForMultipleObjects+0x14
01 00000000`1b4af7c0 00007ffc`ef372a36 KERNELBASE!WaitForMultipleObjectsEx+0xf9
02 00000000`1b4afac0 00007ffc`ef443b84 clr!FinalizerThread::WaitForFinalizerEvent+0xb6
03 00000000`1b4afb00 00007ffc`ef2b7b21 clr!FinalizerThread::FinalizerThreadWorker+0x54
04 00000000`1b4afb40 00007ffc`ef2b7a90 clr!ManagedThreadBase_DispatchInner+0x39
05 00000000`1b4afb80 00007ffc`ef2b79cd clr!ManagedThreadBase_DispatchMiddle+0x6c
06 00000000`1b4afc80 00007ffc`ef3374fa clr!ManagedThreadBase_DispatchOuter+0x75
07 00000000`1b4afd10 00007ffc`ef362e8f clr!FinalizerThread::FinalizerThreadStart+0x10a
08 00000000`1b4afdb0 00007ffd`165c3034 clr!Thread::intermediateThreadProc+0x86
09 00000000`1b4afe70 00007ffd`19161431 kernel32!BaseThreadInitThunk+0x14
0a 00000000`1b4afea0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

5 Id: 7f0.2030 Suspend: 0 Teb: 00000000`00fd6000 Unfrozen

# Child-SP RetAddr Call Site
00 00000000`1bb5fa48 00007ffd`1765029d win32u!NtUserMsgWaitForMultipleObjectsEx+0x14
01 00000000`1bb5fa50 00007ffd`021f5cf3 user32!RealMsgWaitForMultipleObjectsEx+0x1d
02 00000000`1bb5fa90 00007ffd`021f5c6f GdiPlus!BackgroundThreadProc+0x63
03 00000000`1bb5fb00 00007ffd`165c3034 GdiPlus!DllRefCountSafeThreadThunk+0x1f
04 00000000`1bb5fb30 00007ffd`19161431 kernel32!BaseThreadInitThunk+0x14
05 00000000`1bb5fb60 00000000`00000000 ntdll!RtlUserThreadStart+0x21

6 Id: 7f0.205c Suspend: 0 Teb: 00000000`00fd8000 Unfrozen

# Child-SP RetAddr Call Site
00 00000000`1dc7fb98 00007ffd`1910f856 ntdll!NtWaitForWorkViaWorkerFactory+0x14
01 00000000`1dc7fba0 00007ffd`165c3034 ntdll!TppWorkerThread+0x536
02 00000000`1dc7fe90 00007ffd`19161431 kernel32!BaseThreadInitThunk+0x14
03 00000000`1dc7fec0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

7 Id: 7f0.2184 Suspend: 0 Teb: 00000000`00fda000 Unfrozen

# Child-SP RetAddr Call Site
00 00000000`1f9cfb98 00007ffd`1910f856 ntdll!NtWaitForWorkViaWorkerFactory+0x14
01 00000000`1f9cfba0 00007ffd`165c3034 ntdll!TppWorkerThread+0x536
02 00000000`1f9cfe90 00007ffd`19161431 kernel32!BaseThreadInitThunk+0x14
03 00000000`1f9cfec0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

41
 8 Id: 7f0.2098 Suspend: 0 Teb: 00000000`00fdc000 Unfrozen
# Child-SP RetAddr Call Site
00 00000000`1facf9f8 00007ffd`1910f856 ntdll!NtWaitForWorkViaWorkerFactory+0x14
01 00000000`1facfa00 00007ffd`165c3034 ntdll!TppWorkerThread+0x536
02 00000000`1facfcf0 00007ffd`19161431 kernel32!BaseThreadInitThunk+0x14
03 00000000`1facfd20 00000000`00000000 ntdll!RtlUserThreadStart+0x21

9 Id: 7f0.113c Suspend: 0 Teb: 00000000`00fde000 Unfrozen

# Child-SP RetAddr Call Site
00 00000000`1fbcf1b8 00007ffd`15d96099 ntdll!NtWaitForMultipleObjects+0x14
01 00000000`1fbcf1c0 00007ffd`17382ab7 KERNELBASE!WaitForMultipleObjectsEx+0xf9
02 00000000`1fbcf4c0 00007ffd`1737ce40 combase!WaitCoalesced+0xb3
[onecore\com\published\comutils\coalescedwait.cxx @ 72]
03 00000000`1fbcf750 00007ffd`1737ff11 combase!CROIDTable::WorkerThreadLoop+0x50
[onecore\com\combase\dcomrem\refcache.cxx @ 1650]
04 00000000`1fbcf7a0 00007ffd`173c75dc combase!CRpcThread::WorkerLoop+0x169
[onecore\com\combase\dcomrem\threads.cxx @ 269]
05 00000000`1fbcf800 00007ffd`165c3034 combase!CRpcThreadCache::RpcWorkerThreadEntry+0x7c
[onecore\com\combase\dcomrem\threads.cxx @ 76]
06 00000000`1fbcf830 00007ffd`19161431 kernel32!BaseThreadInitThunk+0x14
07 00000000`1fbcf860 00000000`00000000 ntdll!RtlUserThreadStart+0x21

Note: We see that threads #0, #3, #4 have clr module on their stack traces (old version of .NET 2.x used mscorwks
module as can be seen in exercise Legacy.PN1). We also see signs of software exception (in red) and exception stack
trace #0 which has signs of managed code exception processing (in yellow).

6. Since .NET Framework version can be different on a machine where the dump file was saved we need to
load the corresponding WinDbg SOS extension version. In the folder C:\ANETMDA-Dumps\Framework64\v4.0.30319
we have the correct version of .NET Framework copied from the machine the memory dump came from. So we load
SOS WinDbg extension (.load command):

0:000> .load C:\ANETMDA-Dumps\Framework64\v4.0.30319\SOS

7. We check if there is a .NET exception on the current thread 0:

0:000> !pe
Exception object: 0000000002fa3cb0
Exception type: System.NullReferenceException
Message: Object reference not set to an instance of an object.
InnerException: <none>
StackTrace (generated):
SP IP Function
000000000113E5B0 00007FFC8FCB0829
ApplicationA!ApplicationA.Form1.button1_Click_1(System.Object, System.EventArgs)+0x39
000000000113E5F0 00007FFCD8B060B2
System_Windows_Forms_ni!System.Windows.Forms.Control.OnClick(System.EventArgs)+0x82
000000000113E630 00007FFCD8B094CC
System_Windows_Forms_ni!System.Windows.Forms.Button.OnClick(System.EventArgs)+0xbc
000000000113E680 00007FFCD92579CC
System_Windows_Forms_ni!System.Windows.Forms.Button.OnMouseUp(System.Windows.Forms.MouseEventAr
gs)+0x14c
000000000113E740 00007FFCD9204602
System_Windows_Forms_ni!System.Windows.Forms.Control.WmMouseUp(System.Windows.Forms.Message
ByRef, System.Windows.Forms.MouseButtons, Int32)+0x2d2

42
 000000000113E7C0 00007FFCD8B1AEBB
System_Windows_Forms_ni!System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message
ByRef)+0x97b
000000000113E880 00007FFCD8B10234
System_Windows_Forms_ni!System.Windows.Forms.ButtonBase.WndProc(System.Windows.Forms.Message
ByRef)+0x84
000000000113E900 00007FFCD8B10184
System_Windows_Forms_ni!System.Windows.Forms.Button.WndProc(System.Windows.Forms.Message
ByRef)+0x24
000000000113E930 00007FFCD8B1A3C3
System_Windows_Forms_ni!System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr,
IntPtr)+0xc3

StackTraceString: <none>
HResult: 80004003

Note: We also double check that no other threads have exceptions by executing !pe command for each thread using
~*e command:

0:000> ~*e !pe

Exception object: 0000000002fa3cb0
Exception type: System.NullReferenceException
Message: Object reference not set to an instance of an object.
InnerException: <none>
StackTrace (generated):
SP IP Function
000000000113E5B0 00007FFC8FCB0829
ApplicationA!ApplicationA.Form1.button1_Click_1(System.Object, System.EventArgs)+0x39
000000000113E5F0 00007FFCD8B060B2
System_Windows_Forms_ni!System.Windows.Forms.Control.OnClick(System.EventArgs)+0x82
000000000113E630 00007FFCD8B094CC
System_Windows_Forms_ni!System.Windows.Forms.Button.OnClick(System.EventArgs)+0xbc
000000000113E680 00007FFCD92579CC
System_Windows_Forms_ni!System.Windows.Forms.Button.OnMouseUp(System.Windows.Forms.MouseEventAr
gs)+0x14c
000000000113E740 00007FFCD9204602
System_Windows_Forms_ni!System.Windows.Forms.Control.WmMouseUp(System.Windows.Forms.Message
ByRef, System.Windows.Forms.MouseButtons, Int32)+0x2d2
000000000113E7C0 00007FFCD8B1AEBB
System_Windows_Forms_ni!System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message
ByRef)+0x97b
000000000113E880 00007FFCD8B10234
System_Windows_Forms_ni!System.Windows.Forms.ButtonBase.WndProc(System.Windows.Forms.Message
ByRef)+0x84
000000000113E900 00007FFCD8B10184
System_Windows_Forms_ni!System.Windows.Forms.Button.WndProc(System.Windows.Forms.Message
ByRef)+0x24
000000000113E930 00007FFCD8B1A3C3
System_Windows_Forms_ni!System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr,
IntPtr)+0xc3

StackTraceString: <none>
HResult: 80004003
The current thread is unmanaged
The current thread is unmanaged
The current thread is unmanaged
There is no current managed exception on this thread
The current thread is unmanaged
The current thread is unmanaged
43
The current thread is unmanaged
The current thread is unmanaged
The current thread is unmanaged

8. We now check the version of .NET used when ApplicationA was running:

0:000> lmv m clr

Browse full module list
start end module name
00007ffc`ef2b0000 00007ffc`efc9c000 clr (pdb symbols)
c:\mss\clr.pdb\89AF76D6C0C841F8884C33E9CD93C8FF2\clr.pdb
Loaded symbol image file: clr.dll
Image path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
Image name: clr.dll
Browse all global symbols functions data
Timestamp: Fri May 25 18:28:01 2018 (5B08B821)
CheckSum: 009E96E0
ImageSize: 009EC000
File version: 4.7.3120.0
Product version: 4.0.30319.0
File flags: 8 (Mask 3F) Private
File OS: 4 Unknown Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04b0
Information from resource tables:
CompanyName: Microsoft Corporation
ProductName: Microsoft® .NET Framework
InternalName: clr.dll
OriginalFilename: clr.dll
ProductVersion: 4.7.3120.0
FileVersion: 4.7.3120.0 built by: NET472REL1LAST
PrivateBuild: DDBLD413
FileDescription: Microsoft .NET Runtime Common Language Runtime - WorkStation
LegalCopyright: © Microsoft Corporation. All rights reserved.
Comments: Flavor=Retail

44
Note: On my analysis system the version is slightly different:

It has a different .3131 version suffix. The version can also be checked by listing all loaded WinDbg extensions
(sos.dll is used for .NET analysis):

0:000> .chain
Extension DLL search Path:
[...]
Extension DLL chain:
c:\mss\SOS_AMD64_AMD64_4.7.3120.00.dll\5B08B8219ec000\SOS_AMD64_AMD64_4.7.3120.00.dll:
image 4.7.3120.0, API 1.0.0, built Fri May 25 18:20:07 2018
[path:
c:\mss\SOS_AMD64_AMD64_4.7.3120.00.dll\5B08B8219ec000\SOS_AMD64_AMD64_4.7.3120.00.dll]
C:\ANETMDA-Dumps\Framework64\v4.0.30319\SOS: image 4.7.3120.0, API 1.0.0, built Fri May 25
18:20:07 2018
[path: C:\ANETMDA-Dumps\Framework64\v4.0.30319\SOS.dll]
dbghelp: image 10.0.17134.12, API 10.0.6,
[path: C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\dbghelp.dll]
ext: image 10.0.17134.12, API 1.0.0,
[path: C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\winext\ext.dll]
exts: image 10.0.17134.12, API 1.0.0,
[path: C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\WINXP\exts.dll]
uext: image 10.0.17134.12, API 1.0.0,
[path: C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\winext\uext.dll]
ntsdexts: image 10.0.17134.12, API 1.0.0,
[path: C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\WINXP\ntsdexts.dll]
45
Note: We see two SOS extension DLLs loaded having the same timestamp but different paths. The top one was
probably downloaded from Microsoft symbol server and loaded as the resut of !pe command. We can unload them
one after another and check !pe command again (which shouldn’t be available):

0:000> .unload SOS_AMD64_AMD64_4.7.3120.00

Unloading c:\mss\SOS_AMD64_AMD64_4.7.3120.00.dll\5B08B8219ec000\SOS_AMD64_AMD64_4.7.3120.00.dll
extension DLL

0:000> .chain
Extension DLL search Path:
[...]
Extension DLL chain:
C:\ANETMDA-Dumps\Framework64\v4.0.30319\SOS: image 4.7.3120.0, API 1.0.0, built Fri May 25
18:20:07 2018
[path: C:\ANETMDA-Dumps\Framework64\v4.0.30319\SOS.dll]
dbghelp: image 10.0.17134.12, API 10.0.6,
[path: C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\dbghelp.dll]
ext: image 10.0.17134.12, API 1.0.0,
[path: C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\winext\ext.dll]
exts: image 10.0.17134.12, API 1.0.0,
[path: C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\WINXP\exts.dll]
uext: image 10.0.17134.12, API 1.0.0,
[path: C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\winext\uext.dll]
ntsdexts: image 10.0.17134.12, API 1.0.0,
[path: C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\WINXP\ntsdexts.dll]

0:000> .unload SOS

Unloading C:\ANETMDA-Dumps\Framework64\v4.0.30319\SOS extension DLL

0:000> .chain
Extension DLL search Path:
[...]
Extension DLL chain:
dbghelp: image 10.0.17134.12, API 10.0.6,
[path: C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\dbghelp.dll]
ext: image 10.0.17134.12, API 1.0.0,
[path: C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\winext\ext.dll]
exts: image 10.0.17134.12, API 1.0.0,
[path: C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\WINXP\exts.dll]
uext: image 10.0.17134.12, API 1.0.0,
[path: C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\winext\uext.dll]
ntsdexts: image 10.0.17134.12, API 1.0.0,
[path: C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\WINXP\ntsdexts.dll]

0:000> !pe
No export pe found

46
9. Let’s see what !analyze -v command says:

0:000> !analyze -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************

*** ERROR: Module load completed but symbols could not be loaded for mscorlib.ni.dll
*** WARNING: Unable to verify checksum for ApplicationA.exe
GetUrlPageData2 (WinHttp) failed: 12002.

KEY_VALUES_STRING: 1

TIMELINE_ANALYSIS: 1

Timeline: !analyze.Start
Name: <blank>
Time: 2018-07-27T23:53:37.297Z
Diff: 1569881297 mSec

Timeline: Dump.Current
Name: <blank>
Time: 2018-07-09T19:48:56.0Z
Diff: 0 mSec

Timeline: Process.Start
Name: <blank>
Time: 2018-07-09T19:48:20.0Z
Diff: 36000 mSec

Timeline: OS.Boot
Name: <blank>
Time: 2018-07-08T16:43:01.0Z
Diff: 97555000 mSec

DUMP_CLASS: 2

DUMP_QUALIFIER: 400

FAULTING_IP:
+0
00000000`00000000 ?? ???

EXCEPTION_RECORD: 000000001e58d400 -- (.exr 0x1e58d400)

ExceptionAddress: 00050001ffff0006
ExceptionCode: 00010000
ExceptionFlags: 00050003
NumberParameters: 131071
Parameter[0]: 0005000300010000
Parameter[1]: 000300010000ffff
Parameter[2]: 00010000ffff0006
Parameter[3]: 0000ffff00050003
Parameter[4]: ffff000600030001
Parameter[5]: 0005000300010000
Parameter[6]: 000300010000ffff
Parameter[7]: 00010000ffff0006

47
 Parameter[8]: 0026ffff00050003
Parameter[9]: ffff003a0039002c
Parameter[10]: 0005000300010000
Parameter[11]: 000300010000ffff
Parameter[12]: 00010000ffff0006
Parameter[13]: 0000ffff00050003
Parameter[14]: ffff000600030001

FAULTING_THREAD: 000022e0

DEFAULT_BUCKET_ID: BREAKPOINT_NOSOS

PROCESS_NAME: ApplicationA.exe

ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION} Breakpoint A breakpoint has been reached.

EXCEPTION_CODE: (HRESULT) 0x80000003 (2147483651) - One or more arguments are invalid

EXCEPTION_CODE_STR: 80000003

WATSON_BKT_PROCSTAMP: 5b43b8ae

WATSON_BKT_PROCVER: 1.0.0.0

PROCESS_VER_PRODUCT: ApplicationA

WATSON_BKT_MODULE: unknown

WATSON_BKT_MODVER: 0.0.0.0

WATSON_BKT_MODOFFSET: 0

WATSON_BKT_MODSTAMP: bbbbbbb4

BUILD_VERSION_STRING: 17134.1.amd64fre.rs4_release.180410-1804

MODLIST_WITH_TSCHKSUM_HASH: a035b8758813cf1c8d02cba3f73b17e1bf0cb64f

MODLIST_SHA1_HASH: cfe07c3c7dceb6b7fc873c4345687f87357309a6

NTGLOBALFLAG: 0

PROCESS_BAM_CURRENT_THROTTLED: 0

PROCESS_BAM_PREVIOUS_THROTTLED: 0

APPLICATION_VERIFIER_FLAGS: 0

PRODUCT_TYPE: 1

SUITE_MASK: 784

DUMP_FLAGS: 8000c07

DUMP_TYPE: 3

MISSING_CLR_SYMBOL: 0

ANALYSIS_SESSION_HOST: DESKTOP-IS6V2L0

48
ANALYSIS_SESSION_TIME: 07-28-2018 00:53:37.0297

ANALYSIS_VERSION: 10.0.17134.12 amd64fre

MANAGED_CODE: 1

MANAGED_ENGINE_MODULE: clr

CONTEXT: 0000000051661bf8 -- (.cxr 0x51661bf8)

Unable to read context, HRESULT 0x80004002

THREAD_ATTRIBUTES:
OS_LOCALE: ENI

ADDITIONAL_DEBUG_TEXT: SOS.DLL is not loaded for managed code. Analysis might be incomplete

PROBLEM_CLASSES:

ID: [0n317]
Type: [@APPLICATION_FAULT_STRING]
Class: Primary
Scope: DEFAULT_BUCKET_ID (Failure Bucket ID prefix)
BUCKET_ID
Name: Omit
Data: Add
String: [BREAKPOINT]
PID: [Unspecified]
TID: [Unspecified]
Frame: [0]

ID: [0n247]
Type: [NOSOS]
Class: Addendum
Scope: DEFAULT_BUCKET_ID (Failure Bucket ID prefix)
BUCKET_ID
Name: Add
Data: Omit
PID: [Unspecified]
TID: [Unspecified]
Frame: [0]

BUGCHECK_STR: BREAKPOINT_NOSOS

PRIMARY_PROBLEM_CLASS: BREAKPOINT

LAST_CONTROL_TRANSFER: from 00007ffcd8b933f8 to 00007ffd16171204

STACK_TEXT:
00000000`0113bbc8 00007ffc`d8b933f8 : 00000000`02f87908 00007ffc`d8b2f6d9 00000000`00000000
0000cb83`56ecf3c2 : win32u!NtUserWaitMessage+0x14
00000000`0113bbd0 00007ffc`d8b2f452 : 00000000`02f87908 00000000`0113bce0 00000000`0113bcf0
00000000`00000000 : System_Windows_Forms_ni+0x2d33f8
00000000`0113bc80 00007ffc`d8b2ebd2 : 00000000`02f920a8 00000000`00000001 0000cb83`56ecf3c2
00007ffc`d8b68996 : System_Windows_Forms_ni+0x26f452
00000000`0113bd70 00007ffc`d8b2e9df : 00000000`02f87908 00000000`00000004 00000000`02fd4070
00007ffc`d91f629c : System_Windows_Forms_ni+0x26ebd2
00000000`0113be10 00007ffc`d9226bfd : 00000000`01390e50 00000000`0113bea0 00000000`00001000
00000000`0113be60 : System_Windows_Forms_ni+0x26e9df
00000000`0113be70 00007ffc`d91f72f3 : 00000000`02faacb8 00000000`00000000 00000000`02fd34d8
00000000`00000000 : System_Windows_Forms_ni+0x966bfd

49
00000000`0113bf70 00007ffc`d920494a : 00000000`02f87908 00000000`02fa3cb0 00000000`01390e50
00000000`01390e50 : System_Windows_Forms_ni+0x9372f3
00000000`0113bfe0 00007ffc`d8b1a413 : 00000000`02f8a598 00000000`02fa3cb0 00000000`01390e50
00007ffc`d88f7ea0 : System_Windows_Forms_ni+0x94494a
00000000`0113c010 00007ffc`ef378a6d : 00000000`00000004 00000000`01390e50 00000000`01390e50
00007ffc`d8b1a3ea : System_Windows_Forms_ni+0x25a413
00000000`0113c060 00007ffc`ef378934 : 00000000`0138e9d0 00007ffc`d8b1a3ea 00000000`0113e930
00000000`0138ea68 : clr!ExceptionTracker::CallHandler+0xfd
00000000`0113c150 00007ffc`ef378848 : 00000000`0113e930 00000000`0113ca10 00000000`0113c269
00000000`00000001 : clr!ExceptionTracker::CallCatchHandler+0x90
00000000`0113c1f0 00007ffd`1918ed6d : 00007ffc`d8bbac00 00000000`0113e930 00000000`00000000
00000000`0113c3c0 : clr!ProcessCLRException+0x31c
00000000`0113c2d0 00007ffd`190f7670 : 00000000`0113c400 00000000`0113e930 00000000`00000000
00000000`0113ca10 : ntdll!RtlpExecuteHandlerForUnwind+0xd
00000000`0113c300 00007ffc`ef379550 : 00000000`0113d0c0 00000000`00000000 00000000`00000000
00000000`00000000 : ntdll!RtlUnwindEx+0x3a0
00000000`0113c9e0 00007ffc`ef37950b : 00000000`00000000 00000000`0113d0c0 00000000`00000001
00000000`00000000 : clr!ClrUnwindEx+0x40
00000000`0113cf00 00007ffd`1918eced : 00007ffc`d8bbaca4 00000000`0113e930 00000000`00000000
00000000`0113d0c0 : clr!ProcessCLRException+0x2e9
00000000`0113cfe0 00007ffd`190f6c86 : 00000000`0113d110 00000000`0113d960 00000000`00000000
00000000`00000000 : ntdll!RtlpExecuteHandlerForException+0xd
00000000`0113d010 00007ffd`190f52ca : 00000000`1e58d400 00000000`51661bf8 00000000`0113d780
00000000`00000000 : ntdll!RtlDispatchException+0x3c6
00000000`0113d710 00007ffd`15d8a388 : 00000000`00000000 00000000`02f8b8a0 00000000`02f8a598
00000000`02fa1028 : ntdll!RtlRaiseException+0x31a
00000000`0113df70 00007ffc`ef2b1209 : 00000000`00000000 00000000`00000000 00000000`00000000
00000000`00000000 : KERNELBASE!RaiseException+0x68
00000000`0113e050 00007ffc`ef2b123b : 00000000`00000000 00000000`00000000 00000000`00000000
00000000`00000000 : clr!NakedThrowHelper2+0x9
00000000`0113e080 00007ffc`ef2b1245 : 00007ffc`8fcb0829 00000000`02f8a598 00000000`02f8b8a0
00000000`0113e730 : clr!NakedThrowHelper_RspAligned+0x1e
00000000`0113e5a8 00007ffc`8fcb0829 : 00000000`02f8a598 00000000`02f8b8a0 00000000`0113e730
00000000`00000002 : clr!NakedThrowHelper_FixRsp+0x5
00000000`0113e5b0 00007ffc`d8b060b2 : 00000000`02f27ee8 00000000`02f8a598 00000000`02fa1028
00000000`00000000 : 0x00007ffc`8fcb0829
00000000`0113e5f0 00007ffc`d8b094cc : 00000000`02f27ee8 00000000`00000000 00000000`0113e678
00000000`0113e730 : System_Windows_Forms_ni+0x2460b2
00000000`0113e630 00007ffc`d92579cc : 00000000`02f27ee8 00000000`00000155 00000000`0113e678
00000000`0113e730 : System_Windows_Forms_ni+0x2494cc
00000000`0113e680 00007ffc`d9204602 : 00000000`02f8a598 00000000`02fa1028 0000c9a6`c076a0d7
000000a2`00000103 : System_Windows_Forms_ni+0x9979cc
00000000`0113e740 00007ffc`d8b1aebb : 00000000`02f8a598 00000000`0113e860 00000003`00000000
00000000`00000004 : System_Windows_Forms_ni+0x944602
00000000`0113e7c0 00007ffc`d8b10234 : 00000000`00000000 00007ffd`13ac369f 00000000`0000000f
00000000`00000000 : System_Windows_Forms_ni+0x25aebb
00000000`0113e880 00007ffc`d8b10184 : 00000000`02f8a598 00000000`00000000 00000000`00000000
00000000`02f8a6e0 : System_Windows_Forms_ni+0x250234
00000000`0113e900 00007ffc`d8b1a3c3 : 00000000`00000000 00000000`00000000 00000103`00000001
00000000`0000000f : System_Windows_Forms_ni+0x250184
00000000`0113e930 00007ffc`d91911f1 : 00000000`02f8a6e0 00000000`00000000 00000000`00000202
00007ffd`19123f93 : System_Windows_Forms_ni+0x25a3c3
00000000`0113e9d0 00007ffc`ef2b221e : 00000000`00000070 ffffffff`febd718f ffffffff`febffe97
00007ffd`17646b37 : System_Windows_Forms_ni+0x8d11f1
00000000`0113ea40 00007ffd`17646cc1 : 00000000`80006010 00000000`00000000 00000000`00000000
00000000`00000000 : clr!UMThunkStub+0x6e
00000000`0113ead0 00007ffd`17646693 : 00000000`0113ed00 00000000`1b990c2c 00000000`001c040c
00000000`00000202 : user32!UserCallWinProcCheckWow+0x2c1
00000000`0113ec60 00007ffc`d8b9a378 : 00000000`0113ee10 00000000`00000000 00000000`0113eda0
00007ffc`d8b2f6d9 : user32!DispatchMessageWorker+0x1c3

50
00000000`0113ecf0 00007ffc`d8b2f23e : 00000000`02f87908 00000000`0113ee10 00000000`00000000
00000000`00000000 : System_Windows_Forms_ni+0x2da378
00000000`0113edb0 00007ffc`d8b2ebd2 : 00000000`02f920a8 00000000`00000001 00000000`ffffffff
00000000`00000000 : System_Windows_Forms_ni+0x26f23e
00000000`0113eea0 00007ffc`d8b2e9df : 00000000`02f87908 00000000`ffffffff 00000000`02f8ccd8
00000000`0113f210 : System_Windows_Forms_ni+0x26ebd2
00000000`0113ef40 00007ffc`8fcb04d2 : 00000000`02f87908 00000000`ffffffff 00000000`02f8ccd8
00000000`01390e50 : System_Windows_Forms_ni+0x26e9df
00000000`0113efa0 00007ffc`ef2b6bb3 : 00007ffc`ef2b72e9 00007ffc`8fba4118 00000000`00000000
00007ffc`00000000 : 0x00007ffc`8fcb04d2
00000000`0113efe0 00007ffc`ef2b6a70 : 00000000`00df3067 00007ffc`ef2b78b9 00000000`0113f390
00007ffc`ef2c4570 : clr!CallDescrWorkerInternal+0x83
00000000`0113f020 00007ffc`ef2b735d : 00000000`00000000 00000000`0113f188 00000000`0113f210
00000000`0113f2c8 : clr!CallDescrWorkerWithHandler+0x4e
00000000`0113f060 00007ffc`ef30ec1c : 00000000`0113f110 00000000`00000000 00000000`00000000
00000000`00000000 : clr!MethodDescCallSite::CallTargetWorker+0xf8
00000000`0113f160 00007ffc`ef30ee06 : 00000000`00000000 00000000`00000001 00000000`00000000
00000000`00000000 : clr!RunMain+0x1e7
00000000`0113f340 00007ffc`ef30ecfb : 00007ffc`ef394a40 00000000`01384dd0 00007ffc`ef394a40
00000000`01384dd0 : clr!Assembly::ExecuteMainMethod+0xb6
00000000`0113f630 00007ffc`ef30eaf4 : 00000000`00000000 00000000`00df0000 00000000`00000000
00000000`00000000 : clr!SystemDomain::ExecuteMainMethod+0x57c
00000000`0113fc40 00007ffc`ef30ea72 : 00000000`00df0000 00007ffc`ef30ef20 00000000`00000000
00000000`00000000 : clr!ExecuteEXE+0x3f
00000000`0113fcb0 00007ffc`ef30ef34 : ffffffff`ffffffff 00007ffc`ef30ef20 00000000`00000000
00000000`00000000 : clr!_CorExeMainInternal+0xb2
00000000`0113fd40 00007ffc`efca7b2d : 00000000`00000000 00007ffd`00000091 00000000`00000000
00000000`0113fd18 : clr!CorExeMain+0x14
00000000`0113fd80 00007ffc`f52ba4cc : 00000000`00000000 00007ffc`ef30ef20 00000000`00000000
00000000`00000000 : mscoreei!CorExeMain+0x112
00000000`0113fde0 00007ffd`165c3034 : 00007ffc`efca0000 00000000`00000000 00000000`00000000
00000000`00000000 : mscoree!CorExeMain_Exported+0x6c
00000000`0113fe10 00007ffd`19161431 : 00000000`00000000 00000000`00000000 00000000`00000000
00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
00000000`0113fe40 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000
00000000`00000000 : ntdll!RtlUserThreadStart+0x21

STACK_COMMAND: dt ntdll!LdrpLastDllInitializer BaseDllName ; dt ntdll!LdrpFailureData ; .cxr

0x51661bf8 ; kb

THREAD_SHA1_HASH_MOD_FUNC: 887d086448f96d24f3b65f66fc60a3e4bdb1e4a7

THREAD_SHA1_HASH_MOD_FUNC_OFFSET: fe2edf247b80cd0b68ce89d015c32bb1c5fd1220

THREAD_SHA1_HASH_MOD: af8bef11d1bf76b3e133b20a1a20ebffc06a9385

FOLLOWUP_IP:
win32u!NtUserWaitMessage+14
00007ffd`16171204 c3 ret

FAULT_INSTR_CODE: c32ecdc3

SYMBOL_STACK_INDEX: 0

SYMBOL_NAME: win32u!NtUserWaitMessage+14

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: win32u

51
IMAGE_NAME: win32u.dll

DEBUG_FLR_IMAGE_TIMESTAMP: 0

BUCKET_ID: BREAKPOINT_NOSOS_win32u!NtUserWaitMessage+14

FAILURE_EXCEPTION_CODE: 80000003

FAILURE_IMAGE_NAME: win32u.dll

BUCKET_ID_IMAGE_STR: win32u.dll

FAILURE_MODULE_NAME: win32u

BUCKET_ID_MODULE_STR: win32u

FAILURE_FUNCTION_NAME: NtUserWaitMessage

BUCKET_ID_FUNCTION_STR: NtUserWaitMessage

BUCKET_ID_OFFSET: 14

BUCKET_ID_MODTIMEDATESTAMP: 0

BUCKET_ID_MODCHECKSUM: 27b98

BUCKET_ID_MODVER_STR: 10.0.17134.1

BUCKET_ID_PREFIX_STR: BREAKPOINT_NOSOS_

FAILURE_PROBLEM_CLASS: BREAKPOINT

FAILURE_SYMBOL_NAME: win32u.dll!NtUserWaitMessage

FAILURE_BUCKET_ID: BREAKPOINT_NOSOS_80000003_win32u.dll!NtUserWaitMessage

WATSON_STAGEONE_URL:
http://watson.microsoft.com/StageOne/ApplicationA.exe/1.0.0.0/5b43b8ae/unknown/0.0.0.0/bbbbbbb4
/80000003/00000000.htm?Retriage=1

TARGET_TIME: 2018-07-09T19:48:56.000Z

OSBUILD: 17134

OSSERVICEPACK: 1

SERVICEPACK_NUMBER: 0

OS_REVISION: 0

OSPLATFORM_TYPE: x64

OSNAME: Windows 10

OSEDITION: Windows 10 WinNt SingleUserTS Personal

USER_LCID: 0

OSBUILD_TIMESTAMP: 2020-08-28 05:38:41

52
BUILDDATESTAMP_STR: 180410-1804

BUILDLAB_STR: rs4_release

BUILDOSVER_STR: 10.0.17134.1.amd64fre.rs4_release.180410-1804

ANALYSIS_SESSION_ELAPSED_TIME: 70e3

ANALYSIS_SOURCE: UM

FAILURE_ID_HASH_STRING: um:breakpoint_nosos_80000003_win32u.dll!ntuserwaitmessage

FAILURE_ID_HASH: {c13a261a-1261-0b6a-f27a-a40bf396360c}

Followup: MachineOwner
---------

Note: We see normal manual dump breakpoint error (in blue) but no .NET diagnostics (in red).

10. Finally, we get managed stack trace of the current thread:

0:000> !CLRStack
OS Thread Id: 0x22e0 (0)
Child SP IP Call Site
000000000113bbf8 00007ffd16171204 [InlinedCallFrame: 000000000113bbf8]
System.Windows.Forms.UnsafeNativeMethods.WaitMessage()
000000000113bbf8 00007ffcd8b933f8 [InlinedCallFrame: 000000000113bbf8]
System.Windows.Forms.UnsafeNativeMethods.WaitMessage()
000000000113bbd0 00007ffcd8b933f8 DomainBoundILStubClass.IL_STUB_PInvoke()
000000000113bc80 00007ffcd8b2f452
System.Windows.Forms.Application+ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMso
ComponentManager.FPushMessageLoop(IntPtr, Int32, Int32)
000000000113bd70 00007ffcd8b2ebd2
System.Windows.Forms.Application+ThreadContext.RunMessageLoopInner(Int32,
System.Windows.Forms.ApplicationContext)
000000000113be10 00007ffcd8b2e9df
System.Windows.Forms.Application+ThreadContext.RunMessageLoop(Int32,
System.Windows.Forms.ApplicationContext)
000000000113be70 00007ffcd9226bfd
System.Windows.Forms.Form.ShowDialog(System.Windows.Forms.IWin32Window)
000000000113bf70 00007ffcd91f72f3
System.Windows.Forms.Application+ThreadContext.OnThreadException(System.Exception)
000000000113bfe0 00007ffcd920494a
System.Windows.Forms.Control.WndProcException(System.Exception)
000000000113c010 00007ffcd8b1a413 System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32,
IntPtr, IntPtr)
000000000113e0b0 00007ffcef378a6d [FaultingExceptionFrame: 000000000113e0b0]
000000000113e5b0 00007ffc8fcb0829 ApplicationA.Form1.button1_Click_1(System.Object,
System.EventArgs)
000000000113e5f0 00007ffcd8b060b2 System.Windows.Forms.Control.OnClick(System.EventArgs)
000000000113e630 00007ffcd8b094cc System.Windows.Forms.Button.OnClick(System.EventArgs)
000000000113e680 00007ffcd92579cc
System.Windows.Forms.Button.OnMouseUp(System.Windows.Forms.MouseEventArgs)
000000000113e740 00007ffcd9204602
System.Windows.Forms.Control.WmMouseUp(System.Windows.Forms.Message ByRef,
System.Windows.Forms.MouseButtons, Int32)
000000000113e7c0 00007ffcd8b1aebb
System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)
53
000000000113e880 00007ffcd8b10234
System.Windows.Forms.ButtonBase.WndProc(System.Windows.Forms.Message ByRef)
000000000113e900 00007ffcd8b10184
System.Windows.Forms.Button.WndProc(System.Windows.Forms.Message ByRef)
000000000113e930 00007ffcd8b1a3c3 System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32,
IntPtr, IntPtr)
000000000113e9d0 00007ffcd91911f1 DomainBoundILStubClass.IL_STUB_ReversePInvoke(Int64, Int32,
Int64, Int64)
000000000113ed20 00007ffcef2b221e [InlinedCallFrame: 000000000113ed20]
System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG ByRef)
000000000113ed20 00007ffcd8b9a378 [InlinedCallFrame: 000000000113ed20]
System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG ByRef)
000000000113ecf0 00007ffcd8b9a378 DomainBoundILStubClass.IL_STUB_PInvoke(MSG ByRef)
000000000113edb0 00007ffcd8b2f23e
System.Windows.Forms.Application+ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMso
ComponentManager.FPushMessageLoop(IntPtr, Int32, Int32)
000000000113eea0 00007ffcd8b2ebd2
System.Windows.Forms.Application+ThreadContext.RunMessageLoopInner(Int32,
System.Windows.Forms.ApplicationContext)
000000000113ef40 00007ffcd8b2e9df
System.Windows.Forms.Application+ThreadContext.RunMessageLoop(Int32,
System.Windows.Forms.ApplicationContext)
000000000113efa0 00007ffc8fcb04d2 ApplicationA.Program.Main()
000000000113f210 00007ffcef2b6bb3 [GCFrame: 000000000113f210]

11. We close logging before exiting WinDbg:

0:000> .logclose
Closing open log file C:\ANETMDA-Dumps\Processes\ApplicationA.log

Note: To avoid possible confusion and glitches we recommend exiting WinDbg after each exercise.

54
Published by OpenTask, Republic of Ireland

Copyright © 2014 by OpenTask

Copyright © 2014 by Software Diagnostics Services

Copyright © 2014 by Dmitry Vostokov

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or
transmitted, in any form or by any means, without the prior written permission of the
publisher.

You must not circulate this book in any other binding or cover and you must impose the same
condition on any acquirer.

Product and company names mentioned in this book may be trademarks of their owners.

OpenTask books and magazines are available through booksellers and distributors worldwide.
For further information or comments send requests to [email protected].

A CIP catalogue record for this book is available from the British Library.

ISBN-l3: 978-1-908043-71-9 (Paperback)

1st printing, 2014

2
Contents

Presentation Slides and Transcript ................................................................................................................................... 5

Core Dump Collection..................................................................................................................................................... 25
Practice Exercises ........................................................................................................................................................... 31
Exercise 0 (GDB) ......................................................................................................................................................... 36
Exercise 0 (LLDB)......................................................................................................................................................... 39
Exercise A1 (GDB) ....................................................................................................................................................... 42
Exercise A1 (LLDB) ...................................................................................................................................................... 54
Exercise A2 (GDB) ....................................................................................................................................................... 66
Exercise A2 (LLDB) ...................................................................................................................................................... 74
Exercise A3 (GDB) ....................................................................................................................................................... 83
Exercise A3 (LLDB) ...................................................................................................................................................... 88
Exercise A4 (GDB) ....................................................................................................................................................... 94
Exercise A4 (LLDB) .................................................................................................................................................... 105
Exercise A5 (GDB) ..................................................................................................................................................... 115
Exercise A5 (LLDB) .................................................................................................................................................... 121
Exercise A6 (GDB) ..................................................................................................................................................... 129
Exercise A6 (LLDB) .................................................................................................................................................... 155
Exercise A7 (GDB) ..................................................................................................................................................... 176
Exercise A7 (LLDB) .................................................................................................................................................... 184
Exercise A8 (GDB) ..................................................................................................................................................... 192
Exercise A8 (LLDB) .................................................................................................................................................... 207
Exercise A9 (GDB) ..................................................................................................................................................... 222
Exercise A9 (LLDB) .................................................................................................................................................... 249
Exercise A10 (GDB) ................................................................................................................................................... 277
Exercise A10 (LLDB) .................................................................................................................................................. 290
Exercise A11 (GDB) ................................................................................................................................................... 305
Exercise A11 (LLDB) .................................................................................................................................................. 312
Exercise A12 (GDB) ................................................................................................................................................... 321
Exercise A12 (LLDB) .................................................................................................................................................. 344
App Source Code .......................................................................................................................................................... 353
App0 ......................................................................................................................................................................... 354
App1 ......................................................................................................................................................................... 355
App2 ......................................................................................................................................................................... 356
3
 App3 ......................................................................................................................................................................... 358
App4 ......................................................................................................................................................................... 360
App5 ......................................................................................................................................................................... 362
App6 ......................................................................................................................................................................... 364
App7 ......................................................................................................................................................................... 366
App8 ......................................................................................................................................................................... 368
App9 ......................................................................................................................................................................... 370
App10 ....................................................................................................................................................................... 372
App11 ....................................................................................................................................................................... 374
Selected Patterns.......................................................................................................................................................... 377
NULL Pointer (data) .................................................................................................................................................. 378
Incomplete Stack Trace ............................................................................................................................................ 379
Stack Trace................................................................................................................................................................ 380
Multiple Exceptions .................................................................................................................................................. 381
Shared Buffer Overwrite........................................................................................................................................... 382
Incorrect Stack Trace ................................................................................................................................................ 386
NULL Pointer (code).................................................................................................................................................. 387
Spiking Thread .......................................................................................................................................................... 389
Dynamic Memory Corruption (process heap) .......................................................................................................... 391
Double Free (process heap)...................................................................................................................................... 392
Execution Residue .................................................................................................................................................... 393
Coincidental Symbolic Information .......................................................................................................................... 395
Stack Overflow (user mode) ..................................................................................................................................... 397
Divide by Zero (user mode) ...................................................................................................................................... 400
Local Buffer Overflow ............................................................................................................................................... 401
C++ Exception ........................................................................................................................................................... 403
Truncated Dump ....................................................................................................................................................... 404
Paratext .................................................................................................................................................................... 405

4
 Exercise A1 (GDB)

Goal: Learn how to list stack traces, disassemble functions, check their correctness, dump data, compare core
dumps with diagnostic reports, get environment

Patterns: Manual Dump, Stack Trace, Stack Trace Collection, Annotated Disassembly, Paratext, Not My Version,
Environment Hint

1. Load a core dump core.1394 and App1 executable:

$ gdb -c ~/Documents/AMCDA-Dumps/core.1394 -e ~/Documents/AMCDA-

Dumps/Apps/App1/Build/Products/Release/App1
GNU gdb 6.3.50-20050815 (Apple version gdb-1820) (Sat Jun 16 02:40:11 UTC 2012)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin".
Reading symbols for shared libraries . done
Reading symbols for shared libraries .......................... done
#0 0x00007fff8a10ce42 in __semwait_signal ()

2. List all threads:

(gdb) info threads

6 0x00007fff8a10ce42 in __semwait_signal ()
5 0x00007fff8a10ce42 in __semwait_signal ()
4 0x00007fff8a10ce42 in __semwait_signal ()
3 0x00007fff8a10ce42 in __semwait_signal ()
2 0x00007fff8a10ce42 in __semwait_signal ()
* 1 0x00007fff8a10ce42 in __semwait_signal ()

3. Get all thread stack traces:

(gdb) thread apply all bt

Thread 6 (core thread 5):

#0 0x00007fff8a10ce42 in __semwait_signal ()
#1 0x00007fff84d6edea in nanosleep ()
#2 0x00007fff84d6ec2c in sleep ()
#3 0x00007fff84d6ec08 in sleep ()
#4 0x000000010390bbb2 in bar_five ()
#5 0x000000010390bbc9 in foo_five ()
#6 0x000000010390bbe1 in thread_five ()
#7 0x00007fff84db88bf in _pthread_start ()
#8 0x00007fff84dbbb75 in thread_start ()

42
Thread 5 (core thread 4):
#0 0x00007fff8a10ce42 in __semwait_signal ()
#1 0x00007fff84d6edea in nanosleep ()
#2 0x00007fff84d6ec2c in sleep ()
#3 0x00007fff84d6ec08 in sleep ()
#4 0x000000010390bb52 in bar_four ()
#5 0x000000010390bb69 in foo_four ()
#6 0x000000010390bb81 in thread_four ()
#7 0x00007fff84db88bf in _pthread_start ()
#8 0x00007fff84dbbb75 in thread_start ()

Thread 4 (core thread 3):

#0 0x00007fff8a10ce42 in __semwait_signal ()
#1 0x00007fff84d6edea in nanosleep ()
#2 0x00007fff84d6ec2c in sleep ()
#3 0x00007fff84d6ec08 in sleep ()
#4 0x000000010390baf2 in bar_three ()
#5 0x000000010390bb09 in foo_three ()
#6 0x000000010390bb21 in thread_three ()
#7 0x00007fff84db88bf in _pthread_start ()
#8 0x00007fff84dbbb75 in thread_start ()

Thread 3 (core thread 2):

#0 0x00007fff8a10ce42 in __semwait_signal ()
#1 0x00007fff84d6edea in nanosleep ()
#2 0x00007fff84d6ec2c in sleep ()
#3 0x00007fff84d6ec08 in sleep ()
#4 0x000000010390ba92 in bar_two ()
#5 0x000000010390baa9 in foo_two ()
#6 0x000000010390bac1 in thread_two ()
---Type <return> to continue, or q <return> to quit---
#7 0x00007fff84db88bf in _pthread_start ()
#8 0x00007fff84dbbb75 in thread_start ()

Thread 2 (core thread 1):

#0 0x00007fff8a10ce42 in __semwait_signal ()
#1 0x00007fff84d6edea in nanosleep ()
#2 0x00007fff84d6ec2c in sleep ()
#3 0x00007fff84d6ec08 in sleep ()
#4 0x000000010390ba32 in bar_one ()
#5 0x000000010390ba49 in foo_one ()
#6 0x000000010390ba61 in thread_one ()
#7 0x00007fff84db88bf in _pthread_start ()
#8 0x00007fff84dbbb75 in thread_start ()

Thread 1 (core thread 0):

#0 0x00007fff8a10ce42 in __semwait_signal ()
#1 0x00007fff84d6edea in nanosleep ()
#2 0x00007fff84d6ec2c in sleep ()
#3 0x00007fff84d6ec08 in sleep ()
#4 0x000000010390bcc3 in main ()

4. Switch to the thread #3 and get its stack trace:

(gdb) thread 3
[Switching to thread 3 (core thread 2)]
0x00007fff8a10ce42 in __semwait_signal ()

43
(gdb) bt
#0 0x00007fff8a10ce42 in __semwait_signal ()
#1 0x00007fff84d6edea in nanosleep ()
#2 0x00007fff84d6ec2c in sleep ()
#3 0x00007fff84d6ec08 in sleep ()
#4 0x000000010390ba92 in bar_two ()
#5 0x000000010390baa9 in foo_two ()
#6 0x000000010390bac1 in thread_two ()
#7 0x00007fff84db88bf in _pthread_start ()
#8 0x00007fff84dbbb75 in thread_start ()

5. Check that bar_two called sleep function:

(gdb) disassemble bar_two

Dump of assembler code for function bar_two:
0x000000010390ba80 <bar_two+0>: push %rbp
0x000000010390ba81 <bar_two+1>: mov %rsp,%rbp
0x000000010390ba84 <bar_two+4>: sub $0x10,%rsp
0x000000010390ba88 <bar_two+8>: mov $0xffffffff,%edi
0x000000010390ba8d <bar_two+13>: callq 0x10390bce0 <dyld_stub_sleep>
0x000000010390ba92 <bar_two+18>: mov %eax,-0x4(%rbp)
0x000000010390ba95 <bar_two+21>: add $0x10,%rsp
0x000000010390ba99 <bar_two+25>: pop %rbp
0x000000010390ba9a <bar_two+26>: retq
0x000000010390ba9b <bar_two+27>: nopl 0x0(%rax,%rax,1)
End of assembler dump.

6. Compare with intel disassembly flavor:

(gdb) set disassembly-flavor intel

(gdb) disassemble bar_two

Dump of assembler code for function bar_two:
0x000000010390ba80 <bar_two+0>: push rbp
0x000000010390ba81 <bar_two+1>: mov rbp,rsp
0x000000010390ba84 <bar_two+4>: sub rsp,0x10
0x000000010390ba88 <bar_two+8>: mov edi,0xffffffff
0x000000010390ba8d <bar_two+13>: call 0x10390bce0 <dyld_stub_sleep>
0x000000010390ba92 <bar_two+18>: mov DWORD PTR [rbp-0x4],eax
0x000000010390ba95 <bar_two+21>: add rsp,0x10
0x000000010390ba99 <bar_two+25>: pop rbp
0x000000010390ba9a <bar_two+26>: ret
0x000000010390ba9b <bar_two+27>: nop DWORD PTR [rax+rax+0x0]
End of assembler dump.

(gdb) set disassembly-flavor att

44
7. Follow bar_two to sleep function code:

(gdb) disassemble bar_two

Dump of assembler code for function bar_two:
0x000000010390ba80 <bar_two+0>: push %rbp
0x000000010390ba81 <bar_two+1>: mov %rsp,%rbp
0x000000010390ba84 <bar_two+4>: sub $0x10,%rsp
0x000000010390ba88 <bar_two+8>: mov $0xffffffff,%edi
0x000000010390ba8d <bar_two+13>: callq 0x10390bce0 <dyld_stub_sleep>
0x000000010390ba92 <bar_two+18>: mov %eax,-0x4(%rbp)
0x000000010390ba95 <bar_two+21>: add $0x10,%rsp
0x000000010390ba99 <bar_two+25>: pop %rbp
0x000000010390ba9a <bar_two+26>: retq
0x000000010390ba9b <bar_two+27>: nopl 0x0(%rax,%rax,1)
End of assembler dump.

(gdb) disassemble dyld_stub_sleep

Dump of assembler code for function dyld_stub_sleep:
0x000000010390bce0 <dyld_stub_sleep+0>: jmpq *0x362(%rip) # 0x10390c048
End of assembler dump.

8. Dump the annotated value as a memory address interpreting its contents as a symbol and then disassemble
it:

(gdb) x/a 0x10390c048

0x10390c048: 0x7fff84d6ebef <sleep>

(gdb) disassemble 0x7fff84d6ebef

Dump of assembler code for function sleep:
0x00007fff84d6ebef <sleep+0>: push %rbp
0x00007fff84d6ebf0 <sleep+1>: mov %rsp,%rbp
0x00007fff84d6ebf3 <sleep+4>: push %rbx
0x00007fff84d6ebf4 <sleep+5>: sub $0x28,%rsp
0x00007fff84d6ebf8 <sleep+9>: test %edi,%edi
0x00007fff84d6ebfa <sleep+11>: mov %edi,%ebx
0x00007fff84d6ebfc <sleep+13>: jns 0x7fff84d6ec11 <sleep+34>
0x00007fff84d6ebfe <sleep+15>: mov $0x7fffffff,%edi
0x00007fff84d6ec03 <sleep+20>: callq 0x7fff84d6ebef <sleep>
0x00007fff84d6ec08 <sleep+25>: lea -0x7fffffff(%rbx,%rax,1),%eax
0x00007fff84d6ec0f <sleep+32>: jmp 0x7fff84d6ec4f <sleep+96>
0x00007fff84d6ec11 <sleep+34>: mov %ebx,%eax
0x00007fff84d6ec13 <sleep+36>: mov %rax,-0x18(%rbp)
0x00007fff84d6ec17 <sleep+40>: movq $0x0,-0x10(%rbp)
0x00007fff84d6ec1f <sleep+48>: lea -0x18(%rbp),%rdi
0x00007fff84d6ec23 <sleep+52>: lea -0x28(%rbp),%rsi
0x00007fff84d6ec27 <sleep+56>: callq 0x7fff84d6ed46 <nanosleep>
0x00007fff84d6ec2c <sleep+61>: cmp $0xffffffffffffffff,%eax
0x00007fff84d6ec2f <sleep+64>: je 0x7fff84d6ec37 <sleep+72>
0x00007fff84d6ec31 <sleep+66>: xor %ebx,%ebx
0x00007fff84d6ec33 <sleep+68>: mov %ebx,%eax
0x00007fff84d6ec35 <sleep+70>: jmp 0x7fff84d6ec4f <sleep+96>
0x00007fff84d6ec37 <sleep+72>: callq 0x7fff84e0cc88 <__error>
0x00007fff84d6ec3c <sleep+77>: cmpl $0x4,(%rax)
0x00007fff84d6ec3f <sleep+80>: jne 0x7fff84d6ec33 <sleep+68>
0x00007fff84d6ec41 <sleep+82>: cmpq $0x0,-0x20(%rbp)
0x00007fff84d6ec46 <sleep+87>: setne %al
0x00007fff84d6ec49 <sleep+90>: movzbl %al,%eax
0x00007fff84d6ec4c <sleep+93>: add -0x28(%rbp),%eax
0x00007fff84d6ec4f <sleep+96>: add $0x28,%rsp
45
0x00007fff84d6ec53 <sleep+100>: pop %rbx
0x00007fff84d6ec54 <sleep+101>: pop %rbp
0x00007fff84d6ec55 <sleep+102>: retq
End of assembler dump.

9. Repeat the same with resolving DYLD trampoline stub command:

(gdb) disassemble bar_two

Dump of assembler code for function bar_two:
0x000000010390ba80 <bar_two+0>: push %rbp
0x000000010390ba81 <bar_two+1>: mov %rsp,%rbp
0x000000010390ba84 <bar_two+4>: sub $0x10,%rsp
0x000000010390ba88 <bar_two+8>: mov $0xffffffff,%edi
0x000000010390ba8d <bar_two+13>: callq 0x10390bce0 <dyld_stub_sleep>
0x000000010390ba92 <bar_two+18>: mov %eax,-0x4(%rbp)
0x000000010390ba95 <bar_two+21>: add $0x10,%rsp
0x000000010390ba99 <bar_two+25>: pop %rbp
0x000000010390ba9a <bar_two+26>: retq
0x000000010390ba9b <bar_two+27>: nopl 0x0(%rax,%rax,1)
End of assembler dump.

(gdb) info trampoline 0x10390bce0

Function at 0x10390bce0 becomes 0x7fff84d6ebef becomes 0x0

10. Compare stack trace for thread #3 (core thread 2) and its module info with the diagnostic report
App1_1394.crash:

Process: App1 [1394]

Path: /Users/USER/Documents/*/App1
Identifier: App1
Version: ??? (???)
Code Type: X86-64 (Native)
Parent Process: bash [661]

Date/Time: 2012-07-24 00:20:26.078 +0100

OS Version: Mac OS X 10.7.4 (11E53)
Report Version: 9

Crashed Thread: 0 Dispatch queue: com.apple.main-thread

Exception Type: EXC_CRASH (SIGABRT)

Exception Codes: 0x0000000000000000, 0x0000000000000000

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread

0 libsystem_kernel.dylib 0x00007fff8a10ce42 __semwait_signal + 10
1 libsystem_c.dylib 0x00007fff84d6edea nanosleep + 164
2 libsystem_c.dylib 0x00007fff84d6ec2c sleep + 61
3 libsystem_c.dylib 0x00007fff84d6ec08 sleep + 25
4 App1 0x000000010390bcc3 main + 195
5 App1 0x000000010390ba14 start + 52

Thread 1:
0 libsystem_kernel.dylib 0x00007fff8a10ce42 __semwait_signal + 10
1 libsystem_c.dylib 0x00007fff84d6edea nanosleep + 164
2 libsystem_c.dylib 0x00007fff84d6ec2c sleep + 61
3 libsystem_c.dylib 0x00007fff84d6ec08 sleep + 25
4 App1 0x000000010390ba32 bar_one + 18
5 App1 0x000000010390ba49 foo_one + 9
6 App1 0x000000010390ba61 thread_one + 17
7 libsystem_c.dylib 0x00007fff84db88bf _pthread_start + 335
8 libsystem_c.dylib 0x00007fff84dbbb75 thread_start + 13

46
Thread 2:
0 libsystem_kernel.dylib 0x00007fff8a10ce42 __semwait_signal + 10
1 libsystem_c.dylib 0x00007fff84d6edea nanosleep + 164
2 libsystem_c.dylib 0x00007fff84d6ec2c sleep + 61
3 libsystem_c.dylib 0x00007fff84d6ec08 sleep + 25
4 App1 0x000000010390ba92 bar_two + 18
5 App1 0x000000010390baa9 foo_two + 9
6 App1 0x000000010390bac1 thread_two + 17
7 libsystem_c.dylib 0x00007fff84db88bf _pthread_start + 335
8 libsystem_c.dylib 0x00007fff84dbbb75 thread_start + 13

Thread 3:
0 libsystem_kernel.dylib 0x00007fff8a10ce42 __semwait_signal + 10
1 libsystem_c.dylib 0x00007fff84d6edea nanosleep + 164
2 libsystem_c.dylib 0x00007fff84d6ec2c sleep + 61
3 libsystem_c.dylib 0x00007fff84d6ec08 sleep + 25
4 App1 0x000000010390baf2 bar_three + 18
5 App1 0x000000010390bb09 foo_three + 9
6 App1 0x000000010390bb21 thread_three + 17
7 libsystem_c.dylib 0x00007fff84db88bf _pthread_start + 335
8 libsystem_c.dylib 0x00007fff84dbbb75 thread_start + 13

Thread 4:
0 libsystem_kernel.dylib 0x00007fff8a10ce42 __semwait_signal + 10
1 libsystem_c.dylib 0x00007fff84d6edea nanosleep + 164
2 libsystem_c.dylib 0x00007fff84d6ec2c sleep + 61
3 libsystem_c.dylib 0x00007fff84d6ec08 sleep + 25
4 App1 0x000000010390bb52 bar_four + 18
5 App1 0x000000010390bb69 foo_four + 9
6 App1 0x000000010390bb81 thread_four + 17
7 libsystem_c.dylib 0x00007fff84db88bf _pthread_start + 335
8 libsystem_c.dylib 0x00007fff84dbbb75 thread_start + 13

Thread 5:
0 libsystem_kernel.dylib 0x00007fff8a10ce42 __semwait_signal + 10
1 libsystem_c.dylib 0x00007fff84d6edea nanosleep + 164
2 libsystem_c.dylib 0x00007fff84d6ec2c sleep + 61
3 libsystem_c.dylib 0x00007fff84d6ec08 sleep + 25
4 App1 0x000000010390bbb2 bar_five + 18
5 App1 0x000000010390bbc9 foo_five + 9
6 App1 0x000000010390bbe1 thread_five + 17
7 libsystem_c.dylib 0x00007fff84db88bf _pthread_start + 335
8 libsystem_c.dylib 0x00007fff84dbbb75 thread_start + 13

Thread 0 crashed with X86 Thread State (64-bit):

rax: 0x0000000000000004 rbx: 0x00007fff6350aa08 rcx: 0x00007fff6350a9c8 rdx: 0x0000000000000001
rdi: 0x0000000000000c03 rsi: 0x0000000000000000 rbp: 0x00007fff6350a9f0 rsp: 0x00007fff6350a9c8
r8: 0x000000007fffffff r9: 0x0000000000000000 r10: 0x0000000000000001 r11: 0xffffff80002da8d0
r12: 0x0000000000000000 r13: 0x0000000000000000 r14: 0x00007fff6350aa18 r15: 0x0000000000000000
rip: 0x00007fff8a10ce42 rfl: 0x0000000000000247 cr2: 0x0000000103d0b880
Logical CPU: 0

Binary Images:
0x10390b000 - 0x10390bfff +App1 (??? - ???) <5BC0342F-7E97-3A7D-8EA6-75A0468021EA>
/Users/USER/Documents/*/App1
0x7fff6350b000 - 0x7fff6353fbaf dyld (195.6 - ???) <0CD1B35B-A28F-32DA-B72E-452EAD609613> /usr/lib/dyld
0x7fff849f2000 - 0x7fff84a0ffff libxpc.dylib (77.19.0 - compatibility 1.0.0) <9F57891B-D7EF-3050-BEDD-
21E7C6668248> /usr/lib/system/libxpc.dylib
0x7fff84d68000 - 0x7fff84d69ff7 libsystem_blocks.dylib (53.0.0 - compatibility 1.0.0) <8BCA214A-8992-34B2-
A8B9-B74DEACA1869> /usr/lib/system/libsystem_blocks.dylib
0x7fff84d6a000 - 0x7fff84e47fef libsystem_c.dylib (763.13.0 - compatibility 1.0.0) <41B43515-2806-3FBC-ACF1-
A16F35B7E290> /usr/lib/system/libsystem_c.dylib
0x7fff85022000 - 0x7fff85030fff libdispatch.dylib (187.9.0 - compatibility 1.0.0) <1D5BE322-A9B9-3BCE-8FAC-
076FB07CF54A> /usr/lib/system/libdispatch.dylib
0x7fff855f0000 - 0x7fff855f1fff libunc.dylib (24.0.0 - compatibility 1.0.0) <337960EE-0A85-3DD0-A760-
7134CF4C0AFF> /usr/lib/system/libunc.dylib
0x7fff85ae3000 - 0x7fff85ae4ff7 libremovefile.dylib (21.1.0 - compatibility 1.0.0) <739E6C83-AA52-3C6C-A680-
B37FE2888A04> /usr/lib/system/libremovefile.dylib
0x7fff89114000 - 0x7fff89118fff libmathCommon.A.dylib (2026.0.0 - compatibility 1.0.0) <FF83AFF7-42B2-306E-
90AF-D539C51A4542> /usr/lib/system/libmathCommon.A.dylib
0x7fff89119000 - 0x7fff8911dfff libdyld.dylib (195.5.0 - compatibility 1.0.0) <380C3F44-0CA7-3514-8080-
46D1C9DF4FCD> /usr/lib/system/libdyld.dylib
0x7fff89740000 - 0x7fff89741ff7 libsystem_sandbox.dylib (??? - ???) <96D38E74-F18F-3CCB-A20B-E8E3ADC4E166>
/usr/lib/system/libsystem_sandbox.dylib
0x7fff8a0ef000 - 0x7fff8a0f5fff libmacho.dylib (800.0.0 - compatibility 1.0.0) <165514D7-1BFA-38EF-A151-
676DCD21FB64> /usr/lib/system/libmacho.dylib
47
 0x7fff8a0f6000 - 0x7fff8a116fff libsystem_kernel.dylib (1699.26.8 - compatibility 1.0.0) <1DDC0B0F-DB2A-34D6-
895D-E5B2B5618946> /usr/lib/system/libsystem_kernel.dylib
0x7fff8a2ac000 - 0x7fff8a2b4fff libsystem_dnssd.dylib (??? - ???) <D9BB1F87-A42B-3CBC-9DC2-FC07FCEF0016>
/usr/lib/system/libsystem_dnssd.dylib
0x7fff8ae26000 - 0x7fff8ae61fff libsystem_info.dylib (??? - ???) <35F90252-2AE1-32C5-8D34-782C614D9639>
/usr/lib/system/libsystem_info.dylib
0x7fff8b248000 - 0x7fff8b24afff libquarantine.dylib (36.6.0 - compatibility 1.0.0) <0EBF714B-4B69-3E1F-9A7D-
6BBC2AACB310> /usr/lib/system/libquarantine.dylib
0x7fff8b3b4000 - 0x7fff8b3b4fff libkeymgr.dylib (23.0.0 - compatibility 1.0.0) <61EFED6A-A407-301E-B454-
CD18314F0075> /usr/lib/system/libkeymgr.dylib
0x7fff8b3dd000 - 0x7fff8b3e2fff libcompiler_rt.dylib (6.0.0 - compatibility 1.0.0) <98ECD5F6-E85C-32A5-98CD-
8911230CB66A> /usr/lib/system/libcompiler_rt.dylib
0x7fff8bd1a000 - 0x7fff8bd1bfff libdnsinfo.dylib (395.11.0 - compatibility 1.0.0) <853BAAA5-270F-3FDC-B025-
D448DB72E1C3> /usr/lib/system/libdnsinfo.dylib
0x7fff8c528000 - 0x7fff8c52dff7 libsystem_network.dylib (??? - ???) <5DE7024E-1D2D-34A2-80F4-08326331A75B>
/usr/lib/system/libsystem_network.dylib
0x7fff8cfa3000 - 0x7fff8cfadff7 liblaunch.dylib (392.38.0 - compatibility 1.0.0) <6ECB7F19-B384-32C1-8652-
2463C1CF4815> /usr/lib/system/liblaunch.dylib
0x7fff8fe02000 - 0x7fff8fe09fff libcopyfile.dylib (85.1.0 - compatibility 1.0.0) <0AB51EE2-E914-358C-AC19-
47BC024BDAE7> /usr/lib/system/libcopyfile.dylib
0x7fff8fe4b000 - 0x7fff8fe8dff7 libcommonCrypto.dylib (55010.0.0 - compatibility 1.0.0) <BB770C22-8C57-365A-
8716-4A3C36AE7BFB> /usr/lib/system/libcommonCrypto.dylib
0x7fff90c0f000 - 0x7fff90c18ff7 libsystem_notify.dylib (80.1.0 - compatibility 1.0.0) <A4D651E3-D1C6-3934-
AD49-7A104FD14596> /usr/lib/system/libsystem_notify.dylib
0x7fff91376000 - 0x7fff913a3fe7 libSystem.B.dylib (159.1.0 - compatibility 1.0.0) <7BEBB139-50BB-3112-947A-
F4AA168F991C> /usr/lib/libSystem.B.dylib
0x7fff91489000 - 0x7fff9148fff7 libunwind.dylib (30.0.0 - compatibility 1.0.0) <1E9C6C8C-CBE8-3F4B-A5B5-
E03E3AB53231> /usr/lib/system/libunwind.dylib
0x7fff91a22000 - 0x7fff91a27fff libcache.dylib (47.0.0 - compatibility 1.0.0) <1571C3AB-BCB2-38CD-B3B2-
C5FC3F927C6A> /usr/lib/system/libcache.dylib

External Modification Summary:

Calls made by other processes targeting this process:
task_for_pid: 2
thread_create: 0
thread_set_state: 0
Calls made by this process:
task_for_pid: 0
thread_create: 0
thread_set_state: 0
Calls made by all processes on this machine:
task_for_pid: 2696
thread_create: 0
thread_set_state: 0

VM Region Summary:
ReadOnly portion of Libraries: Total=50.2M resident=50.2M(100%) swapped_out_or_unallocated=0K(0%)
Writable regions: Total=38.9M written=10.8M(28%) resident=42.6M(110%) swapped_out=0K(0%)
unallocated=16777216.0T(45221404475392%)

REGION TYPE VIRTUAL

=========== =======
MALLOC 1220K
Stack 66.6M
__DATA 464K
__LINKEDIT 47.7M
__TEXT 2484K
shared memory 12K
=========== =======
TOTAL 118.4M

48
11. Get App1 data section from the output of vmmap_1394.log:

Virtual Memory Map of process 1394 (App1)

Output report format: 2.2 -- 64-bit process

==== Non-writable regions for process 1394

__TEXT 000000010390b000-000000010390c000 [ 4K] r-x/rwx SM=COW /Users/DumpAnalysis/Documents/AMCDA-
Dumps/Apps/App1/Build/Products/Release/App1

[...]

==== Writable regions for process 1394

__DATA 000000010390c000-000000010390d000 [ 4K] rw-/rwx SM=PRV /Users/DumpAnalysis/Documents/AMCDA-
Dumps/Apps/App1/Build/Products/Release/App1

[...]

12. Compare with the section information in the core dump:

(gdb) maintenance info sections

Exec file:
`/Users/DumpAnalysis/Documents/AMCDA-Dumps/Apps/App1/Build/Products/Release/App1', file type mach-o-le.
0x0000000000000000->0x0000000000000000 at 0x00000000: LC_SEGMENT.__PAGEZERO ALLOC LOAD CODE HAS_CONTENTS
0x0000000100000000->0x0000000100001000 at 0x00000000: LC_SEGMENT.__TEXT ALLOC LOAD CODE HAS_CONTENTS
0x00000001000009e0->0x0000000100000cd3 at 0x000009e0: LC_SEGMENT.__TEXT.__text ALLOC LOAD READONLY CODE
HAS_CONTENTS
0x0000000100000cd4->0x0000000100000ce6 at 0x00000cd4: LC_SEGMENT.__TEXT.__stubs ALLOC LOAD CODE HAS_CONTENTS
0x0000000100000ce8->0x0000000100000d16 at 0x00000ce8: LC_SEGMENT.__TEXT.__stub_helper ALLOC LOAD CODE HAS_CONTENTS
0x0000000100000d16->0x0000000100000d66 at 0x00000d16: LC_SEGMENT.__TEXT.__unwind_info ALLOC LOAD CODE HAS_CONTENTS
0x0000000100000d68->0x0000000100001000 at 0x00000d68: LC_SEGMENT.__TEXT.__eh_frame ALLOC LOAD CODE HAS_CONTENTS
0x0000000100001000->0x0000000100002000 at 0x00001000: LC_SEGMENT.__DATA ALLOC LOAD CODE HAS_CONTENTS
0x0000000100001000->0x0000000100001028 at 0x00001000: LC_SEGMENT.__DATA.__program_vars ALLOC LOAD CODE
HAS_CONTENTS
0x0000000100001028->0x0000000100001038 at 0x00001028: LC_SEGMENT.__DATA.__nl_symbol_ptr ALLOC LOAD CODE
HAS_CONTENTS
0x0000000100001038->0x0000000100001050 at 0x00001038: LC_SEGMENT.__DATA.__la_symbol_ptr ALLOC LOAD CODE
HAS_CONTENTS
0x0000000100001050->0x0000000100001070 at 0x00000000: LC_SEGMENT.__DATA.__common ALLOC
0x0000000100002000->0x00000001000023b0 at 0x00002000: LC_SEGMENT.__LINKEDIT ALLOC LOAD CODE HAS_CONTENTS
0x0000000000000000->0x00000000000001a0 at 0x000020d0: LC_SYMTAB.stabs HAS_CONTENTS
0x0000000000000000->0x0000000000000120 at 0x00002290: LC_SYMTAB.stabstr HAS_CONTENTS
0x0000000000000000->0x0000000000000100 at 0x000020d0: LC_DYSYMTAB.localstabs HAS_CONTENTS
0x0000000000000000->0x00000000000000a0 at 0x000021d0: LC_DYSYMTAB.nonlocalstabs HAS_CONTENTS
0x0000000000000000->0x0000000000000018 at 0x000004b0: LC_LOAD_DYLINKER HAS_CONTENTS
0x0000000000000000->0x00000000000000a8 at 0x00000500: LC_THREAD.x86_THREAD_STATE64.0 HAS_CONTENTS
0x0000000000000000->0x0000000000000030 at 0x000005b0: LC_LOAD_DYLIB HAS_CONTENTS
Core file:
`/Users/DumpAnalysis/Documents/AMCDA-Dumps/core.1394', file type mach-o-le.
0x000000010390b000->0x000000010390c000 at 0x00002000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x000000010390c000->0x000000010390d000 at 0x00003000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x000000010390d000->0x000000010390e000 at 0x00004000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x000000010390e000->0x000000010390f000 at 0x00005000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x000000010390f000->0x0000000103910000 at 0x00006000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x0000000103910000->0x0000000103911000 at 0x00007000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x0000000103911000->0x0000000103926000 at 0x00008000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x0000000103926000->0x0000000103927000 at 0x0001d000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x0000000103927000->0x0000000103928000 at 0x0001e000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x0000000103928000->0x000000010393d000 at 0x0001f000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x000000010393d000->0x000000010393e000 at 0x00034000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x000000010393e000->0x000000010393f000 at 0x00035000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x000000010393f000->0x0000000103940000 at 0x00036000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x0000000103940000->0x00000001039c2000 at 0x00037000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x0000000103a00000->0x0000000103b00000 at 0x000b9000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x0000000103b00000->0x0000000103b01000 at 0x001b9000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x0000000103b01000->0x0000000103b83000 at 0x001ba000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x0000000103b83000->0x0000000103b84000 at 0x0023c000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x0000000103b84000->0x0000000103c06000 at 0x0023d000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x0000000103c06000->0x0000000103c07000 at 0x002bf000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x0000000103c07000->0x0000000103c89000 at 0x002c0000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x0000000103c89000->0x0000000103c8a000 at 0x00342000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x0000000103c8a000->0x0000000103d0c000 at 0x00343000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS

49
 0x00007fff5f50b000->0x00007fff62d0b000 at 0x003c5000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x00007fff62d0b000->0x00007fff6350a000 at 0x03bc5000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x00007fff6350a000->0x00007fff6350b000 at 0x043c4000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x00007fff6350b000->0x00007fff63540000 at 0x043c5000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x00007fff63540000->0x00007fff63542000 at 0x043fa000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x00007fff63542000->0x00007fff6357c000 at 0x043fc000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x00007fff6357c000->0x00007fff6358f000 at 0x04436000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x00007fff749b8000->0x00007fff74a00000 at 0x04449000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x00007fff74a00000->0x00007fff74c00000 at 0x04491000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x00007fff74c00000->0x00007fff74e00000 at 0x04691000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x00007fff74e00000->0x00007fff75000000 at 0x04891000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x00007fff75000000->0x00007fff75200000 at 0x04a91000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x00007fff75200000->0x00007fff75400000 at 0x04c91000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x00007fff75400000->0x00007fff75600000 at 0x04e91000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x00007fff75600000->0x00007fff75800000 at 0x05091000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x00007fff75800000->0x00007fff75a00000 at 0x05291000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x00007fff75a00000->0x00007fff75c00000 at 0x05491000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x00007fff75c00000->0x00007fff75e00000 at 0x05691000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x00007fff75e00000->0x00007fff76200000 at 0x05891000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x00007fff76200000->0x00007fff76400000 at 0x05c91000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x00007fff76400000->0x00007fff764ac000 at 0x05e91000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x00007fff849b8000->0x00007fff91a28000 at 0x05f3d000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x00007fff91a28000->0x00007fff94b30000 at 0x12fad000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x00007fffffe00000->0x00007fffffe02000 at 0x160b5000: LC_SEGMENT. ALLOC LOAD CODE HAS_CONTENTS
0x0000000000000000->0x00000000000000b0 at 0x00000d68: LC_THREAD.x86_THREAD_STATE.0 HAS_CONTENTS
0x0000000000000000->0x0000000000000214 at 0x00000e20: LC_THREAD.x86_FLOAT_STATE.0 HAS_CONTENTS
0x0000000000000000->0x0000000000000018 at 0x0000103c: LC_THREAD.x86_EXCEPTION_STATE.0 HAS_CONTENTS
0x0000000000000000->0x00000000000000b0 at 0x00001064: LC_THREAD.x86_THREAD_STATE.1 HAS_CONTENTS
0x0000000000000000->0x0000000000000214 at 0x0000111c: LC_THREAD.x86_FLOAT_STATE.1 HAS_CONTENTS
0x0000000000000000->0x0000000000000018 at 0x00001338: LC_THREAD.x86_EXCEPTION_STATE.1 HAS_CONTENTS
0x0000000000000000->0x00000000000000b0 at 0x00001360: LC_THREAD.x86_THREAD_STATE.2 HAS_CONTENTS
0x0000000000000000->0x0000000000000214 at 0x00001418: LC_THREAD.x86_FLOAT_STATE.2 HAS_CONTENTS
0x0000000000000000->0x0000000000000018 at 0x00001634: LC_THREAD.x86_EXCEPTION_STATE.2 HAS_CONTENTS
0x0000000000000000->0x00000000000000b0 at 0x0000165c: LC_THREAD.x86_THREAD_STATE.3 HAS_CONTENTS
0x0000000000000000->0x0000000000000214 at 0x00001714: LC_THREAD.x86_FLOAT_STATE.3 HAS_CONTENTS
0x0000000000000000->0x0000000000000018 at 0x00001930: LC_THREAD.x86_EXCEPTION_STATE.3 HAS_CONTENTS
0x0000000000000000->0x00000000000000b0 at 0x00001958: LC_THREAD.x86_THREAD_STATE.4 HAS_CONTENTS
0x0000000000000000->0x0000000000000214 at 0x00001a10: LC_THREAD.x86_FLOAT_STATE.4 HAS_CONTENTS
0x0000000000000000->0x0000000000000018 at 0x00001c2c: LC_THREAD.x86_EXCEPTION_STATE.4 HAS_CONTENTS
0x0000000000000000->0x00000000000000b0 at 0x00001c54: LC_THREAD.x86_THREAD_STATE.5 HAS_CONTENTS
0x0000000000000000->0x0000000000000214 at 0x00001d0c: LC_THREAD.x86_FLOAT_STATE.5 HAS_CONTENTS
0x0000000000000000->0x0000000000000018 at 0x00001f28: LC_THREAD.x86_EXCEPTION_STATE.5 HAS_CONTENTS

13. Dump data with possible symbolic information:

(gdb) x/512a 0x000000010390c000

0x10390c000: 0x10390b000 0x10390c050 <NXArgc>
0x10390c010: 0x10390c058 <NXArgv> 0x10390c060 <environ>
0x10390c020: 0x10390c068 <__progname> 0x7fff8911a6a0 <dyld_stub_binder>
0x10390c030: 0x7fff63546d80 0x10390bcf8
0x10390c040: 0x7fff84dbab01 <pthread_create> 0x7fff84d6ebef <sleep>
0x10390c050 <NXArgc>: 0x1 0x7fff6350aaf0
0x10390c060 <environ>: 0x7fff6350ab00 0x7fff6350ac73
0x10390c070: 0x0 0x0
0x10390c080: 0x0 0x0
0x10390c090: 0x0 0x0
0x10390c0a0: 0x0 0x0
0x10390c0b0: 0x0 0x0
0x10390c0c0: 0x0 0x0
0x10390c0d0: 0x0 0x0
0x10390c0e0: 0x0 0x0
0x10390c0f0: 0x0 0x0
0x10390c100: 0x0 0x0
0x10390c110: 0x0 0x0
0x10390c120: 0x0 0x0
0x10390c130: 0x0 0x0
0x10390c140: 0x0 0x0

50
0x10390c150: 0x0 0x0
0x10390c160: 0x0 0x0
0x10390c170: 0x0 0x0
0x10390c180: 0x0 0x0
0x10390c190: 0x0 0x0
0x10390c1a0: 0x0 0x0
0x10390c1b0: 0x0 0x0
0x10390c1c0: 0x0 0x0
0x10390c1d0: 0x0 0x0
0x10390c1e0: 0x0 0x0
0x10390c1f0: 0x0 0x0
0x10390c200: 0x0 0x0
0x10390c210: 0x0 0x0
0x10390c220: 0x0 0x0
0x10390c230: 0x0 0x0
0x10390c240: 0x0 0x0
0x10390c250: 0x0 0x0
0x10390c260: 0x0 0x0
0x10390c270: 0x0 0x0
0x10390c280: 0x0 0x0
0x10390c290: 0x0 0x0
---Type <return> to continue, or q <return> to quit---q
Quit

14. Dump the contents of memory pointed to by environ variable in null-terminated string format:

(gdb) x/100s 0x7fff6350ab00

[...]
0x7fff6350abd5: ""
0x7fff6350abd6: ""
0x7fff6350abd7: ""
0x7fff6350abd8: "/Users/DumpAnalysis/Documents/AMCDA-
Dumps/Apps/App1/Build/Products/Release/App1"
0x7fff6350ac28: "/Users/DumpAnalysis/Documents/AMCDA-
Dumps/Apps/App1/Build/Products/Release/App1"
0x7fff6350ac78: "TERM_PROGRAM=Apple_Terminal"
0x7fff6350ac94: "TERM=xterm-256color"
0x7fff6350aca8: "SHELL=/bin/bash"
0x7fff6350acb8: "TMPDIR=/var/folders/ww/rmtqfhl93yj4213dnl2rqy6w0000gn/T/"
0x7fff6350acf1: "Apple_PubSub_Socket_Render=/tmp/launch-mYEvtN/Render"
0x7fff6350ad26: "TERM_PROGRAM_VERSION=303.2"
0x7fff6350ad41: "TERM_SESSION_ID=2B039506-8384-4620-B354-120BE31AEA84"
0x7fff6350ad76: "USER=DumpAnalysis"
0x7fff6350ad88: "COMMAND_MODE=unix2003"
0x7fff6350ad9e: "SSH_AUTH_SOCK=/tmp/launch-9sm7dH/Listeners"
0x7fff6350adc9: "__CF_USER_TEXT_ENCODING=0x1F5:0:0"
0x7fff6350adeb: "Apple_Ubiquity_Message=/tmp/launch-tWsFs8/Apple_Ubiquity_Message"
0x7fff6350ae2c:
"PATH=/Applications/Xcode.app/Contents/Developer/usr/bin/:/usr/bin:/bin:/usr/sbin:/sbin:/usr/lo
cal/bin:/usr/X11/bin"
0x7fff6350ae9f: "PWD=/Users/DumpAnalysis"
0x7fff6350aeb7: "LANG=en_IE.UTF-8"
---Type <return> to continue, or q <return> to quit---
0x7fff6350aec8: "SHLVL=1"
0x7fff6350aed0: "HOME=/Users/DumpAnalysis"
0x7fff6350aee9: "LOGNAME=DumpAnalysis"
0x7fff6350aefe: "DISPLAY=/tmp/launch-M8cgb1/org.x:0"
0x7fff6350af21: "SECURITYSESSIONID=186af"

51
0x7fff6350af39: "_=/Users/DumpAnalysis/Documents/AMCDA-
Dumps/Apps/App1/Build/Products/Release/App1"
0x7fff6350af8b: "OLDPWD=/usr/share/man/man1"
0x7fff6350afa6: ""
0x7fff6350afa7: ""
0x7fff6350afa8: "stack_guard=0x74843dc6068699c3"
0x7fff6350afc7: "malloc_entropy=0x7406669509034332,0x71e4e2253a6d22b0"
0x7fff6350affc: ""
0x7fff6350affd: ""

15. Get the list of loaded modules:

(gdb) info sharedlibrary
The DYLD shared library state has been initialized from the executable's shared library information. All symbols should be present, but the addresses of some
symbols may move when the program is executed, as DYLD may relocate library load addresses if necessary.
Requested State Current State
Num Basename Type Address Reason | | Source
| | | | | | | |
1 App1 - 0x10390b000 exec Y Y /Users/DumpAnalysis/Documents/AMCDA-Dumps/Apps/App1/Build/Products/Release/App1 at 0x10390b000
(offset 0x390b000)
(objfile is) [memory object "/Users/DumpAnalysis/Documents/AMCDA-Dumps/Apps/App1/Build/Products/Release/App1" at
0x10390b000]
2 dyld - 0x7fff6350b000 dyld Y Y /usr/lib/dyld at 0x7fff6350b000 (offset 0x7fff6350b001) with prefix "__dyld_"
(objfile is) [memory object "/usr/lib/dyld" at 0x7fff6350b000]
3 libSystem.B.dylib - 0x7fff91376000 dyld Y Y /usr/lib/libSystem.B.dylib at 0x7fff91376000 (offset 0x49b8000)
(objfile is) [memory object "/usr/lib/libSystem.B.dylib" at 0x7fff91376000]
4 libcache.dylib - 0x7fff91a22000 dyld Y Y /usr/lib/system/libcache.dylib at 0x7fff91a22000 (offset 0x49b8000)
(objfile is) [memory object "/usr/lib/system/libcache.dylib" at 0x7fff91a22000]
5 libcommonCrypto.dylib - 0x7fff8fe4b000 dyld Y Y /usr/lib/system/libcommonCrypto.dylib at 0x7fff8fe4b000 (offset 0x49b8000)
(objfile is) [memory object "/usr/lib/system/libcommonCrypto.dylib" at 0x7fff8fe4b000]
6 libcompiler_rt.dylib - 0x7fff8b3dd000 dyld Y Y /usr/lib/system/libcompiler_rt.dylib at 0x7fff8b3dd000 (offset 0x49b8000)
(objfile is) [memory object "/usr/lib/system/libcompiler_rt.dylib" at 0x7fff8b3dd000]
7 libcopyfile.dylib - 0x7fff8fe02000 dyld Y Y /usr/lib/system/libcopyfile.dylib at 0x7fff8fe02000 (offset 0x49b8000)
(objfile is) [memory object "/usr/lib/system/libcopyfile.dylib" at 0x7fff8fe02000]
8 libdispatch.dylib - 0x7fff85022000 dyld Y Y /usr/lib/system/libdispatch.dylib at 0x7fff85022000 (offset 0x49b8000)
(objfile is) [memory object "/usr/lib/system/libdispatch.dylib" at 0x7fff85022000]
9 libdnsinfo.dylib - 0x7fff8bd1a000 dyld Y Y /usr/lib/system/libdnsinfo.dylib at 0x7fff8bd1a000 (offset 0x49b8000)
(objfile is) [memory object "/usr/lib/system/libdnsinfo.dylib" at 0x7fff8bd1a000]
10 libdyld.dylib - 0x7fff89119000 dyld Y Y /usr/lib/system/libdyld.dylib at 0x7fff89119000 (offset 0x49b8000)
(objfile is) [memory object "/usr/lib/system/libdyld.dylib" at 0x7fff89119000]
11 libkeymgr.dylib - 0x7fff8b3b4000 dyld Y Y /usr/lib/system/libkeymgr.dylib at 0x7fff8b3b4000 (offset 0x49b8000)
(objfile is) [memory object "/usr/lib/system/libkeymgr.dylib" at 0x7fff8b3b4000]
12 liblaunch.dylib - 0x7fff8cfa3000 dyld Y Y /usr/lib/system/liblaunch.dylib at 0x7fff8cfa3000 (offset 0x49b800 ---Type <return> to continue,
or q <return> to quit---
0)
(objfile is) [memory object "/usr/lib/system/liblaunch.dylib" at 0x7fff8cfa3000]
13 libmacho.dylib - 0x7fff8a0ef000 dyld Y Y /usr/lib/system/libmacho.dylib at 0x7fff8a0ef000 (offset 0x49b8000)
(objfile is) [memory object "/usr/lib/system/libmacho.dylib" at 0x7fff8a0ef000]
14 libmathCommon.A.dylib - 0x7fff89114000 dyld Y Y /usr/lib/system/libmathCommon.A.dylib at 0x7fff89114000 (offset 0x49b8000)
(objfile is) [memory object "/usr/lib/system/libmathCommon.A.dylib" at 0x7fff89114000]
15 libquarantine.dylib - 0x7fff8b248000 dyld Y Y /usr/lib/system/libquarantine.dylib at 0x7fff8b248000 (offset 0x49b8000)
(objfile is) [memory object "/usr/lib/system/libquarantine.dylib" at 0x7fff8b248000]
16 libremovefile.dylib - 0x7fff85ae3000 dyld Y Y /usr/lib/system/libremovefile.dylib at 0x7fff85ae3000 (offset 0x49b8000)
(objfile is) [memory object "/usr/lib/system/libremovefile.dylib" at 0x7fff85ae3000]
17 libsystem_blocks.dylib - 0x7fff84d68000 dyld Y Y /usr/lib/system/libsystem_blocks.dylib at 0x7fff84d68000 (offset 0x49b8000)
(objfile is) [memory object "/usr/lib/system/libsystem_blocks.dylib" at 0x7fff84d68000]
18 libsystem_c.dylib - 0x7fff84d6a000 dyld Y Y /usr/lib/system/libsystem_c.dylib at 0x7fff84d6a000 (offset 0x49b8000)
(objfile is) [memory object "/usr/lib/system/libsystem_c.dylib" at 0x7fff84d6a000]
19 libsystem_dnssd.dylib - 0x7fff8a2ac000 dyld Y Y /usr/lib/system/libsystem_dnssd.dylib at 0x7fff8a2ac000 (offset 0x49b8000)
(objfile is) [memory object "/usr/lib/system/libsystem_dnssd.dylib" at 0x7fff8a2ac000]
20 libsystem_info.dylib - 0x7fff8ae26000 dyld Y Y /usr/lib/system/libsystem_info.dylib at 0x7fff8ae26000 (offset 0x49b8000)
(objfile is) [memory object "/usr/lib/system/libsystem_info.dylib" at 0x7fff8ae26000]
21 libsystem_kernel.dylib - 0x7fff8a0f6000 dyld Y Y /usr/lib/system/libsystem_kernel.dylib at 0x7fff8a0f6000 (offset 0x49b8000)
(objfile is) [memory object "/usr/lib/system/libsystem_kernel.dylib" at 0x7fff8a0f6000]
22 libsystem_network.dylib - 0x7fff8c528000 dyld Y Y /usr/lib/system/libsystem_network.dylib at 0x7fff8c528000 (offset 0x49b8000)
(objfile is) [memory object "/usr/lib/system/libsystem_network.dylib" at 0x7fff8c528000]
23 libsystem_notify.dylib - 0x7fff90c0f000 dyld Y Y /usr/lib/system/libsystem_notify.dylib at 0x7fff90c0f000 (offset 0x49b8000)
---Type <return> to continue, or q <return> to quit---
(objfile is) [memory object "/usr/lib/system/libsystem_notify.dylib" at 0x7fff90c0f000]
24 libsystem_sandbox.dylib - 0x7fff89740000 dyld Y Y /usr/lib/system/libsystem_sandbox.dylib at 0x7fff89740000 (offset 0x49b8000)
(objfile is) [memory object "/usr/lib/system/libsystem_sandbox.dylib" at 0x7fff89740000]
25 libunc.dylib - 0x7fff855f0000 dyld Y Y /usr/lib/system/libunc.dylib at 0x7fff855f0000 (offset 0x49b8000)
(objfile is) [memory object "/usr/lib/system/libunc.dylib" at 0x7fff855f0000]
26 libunwind.dylib - 0x7fff91489000 dyld Y Y /usr/lib/system/libunwind.dylib at 0x7fff91489000 (offset 0x49b8000)
(objfile is) [memory object "/usr/lib/system/libunwind.dylib" at 0x7fff91489000]
27 libxpc.dylib - 0x7fff849f2000 dyld Y Y /usr/lib/system/libxpc.dylib at 0x7fff849f2000 (offset 0x49b8000)
(objfile is) [memory object "/usr/lib/system/libxpc.dylib" at 0x7fff849f2000]

52
 Exercise A1 (LLDB)

Goal: Learn how to list stack traces, disassemble functions, check their correctness, dump data, compare core
dumps with diagnostic reports, get environment

Patterns: Manual Dump, Stack Trace, Stack Trace Collection, Annotated Disassembly, Paratext, Not My Version,
Environment Hint

1. Load a core dump core.1394 and App1 executable:

$ lldb -c ~/Documents/AMCDA-Dumps/core.1394 -f ~/Documents/AMCDA-

Dumps/Apps/App1/Build/Products/Release/App1
error: core.1394 is a corrupt mach-o file: load command 46 LC_SEGMENT_64 has a fileoff +
filesize (0x160b7000) that extends beyond the end of the file (0x160b5000), the segment will
be truncated
Core file '/Users/DumpAnalysis/Documents/AMCDA-Dumps/core.1394' (x86_64) was loaded.
Process 0 stopped
* thread #1: tid = 0x0000, 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10,
stop reason = signal SIGSTOP
frame #0: 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10
libsystem_kernel.dylib`__semwait_signal + 10:
-> 0x7fff8a10ce42: jae 0x7fff8a10ce49 ; __semwait_signal + 17
0x7fff8a10ce44: jmpq 0x7fff8a10dffc ; cerror
0x7fff8a10ce49: ret
0x7fff8a10ce4a: nop
thread #2: tid = 0x0001, 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10,
stop reason = signal SIGSTOP
frame #0: 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10
libsystem_kernel.dylib`__semwait_signal + 10:
-> 0x7fff8a10ce42: jae 0x7fff8a10ce49 ; __semwait_signal + 17
0x7fff8a10ce44: jmpq 0x7fff8a10dffc ; cerror
0x7fff8a10ce49: ret
0x7fff8a10ce4a: nop
thread #3: tid = 0x0002, 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10,
stop reason = signal SIGSTOP
frame #0: 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10
libsystem_kernel.dylib`__semwait_signal + 10:
-> 0x7fff8a10ce42: jae 0x7fff8a10ce49 ; __semwait_signal + 17
0x7fff8a10ce44: jmpq 0x7fff8a10dffc ; cerror
0x7fff8a10ce49: ret
0x7fff8a10ce4a: nop
thread #4: tid = 0x0003, 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10,
stop reason = signal SIGSTOP
frame #0: 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10
libsystem_kernel.dylib`__semwait_signal + 10:
-> 0x7fff8a10ce42: jae 0x7fff8a10ce49 ; __semwait_signal + 17
0x7fff8a10ce44: jmpq 0x7fff8a10dffc ; cerror
0x7fff8a10ce49: ret
0x7fff8a10ce4a: nop

54
 thread #5: tid = 0x0004, 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10,
stop reason = signal SIGSTOP
frame #0: 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10
libsystem_kernel.dylib`__semwait_signal + 10:
-> 0x7fff8a10ce42: jae 0x7fff8a10ce49 ; __semwait_signal + 17
0x7fff8a10ce44: jmpq 0x7fff8a10dffc ; cerror
0x7fff8a10ce49: ret
0x7fff8a10ce4a: nop
thread #6: tid = 0x0005, 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10,
stop reason = signal SIGSTOP
frame #0: 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10
libsystem_kernel.dylib`__semwait_signal + 10:
-> 0x7fff8a10ce42: jae 0x7fff8a10ce49 ; __semwait_signal + 17
0x7fff8a10ce44: jmpq 0x7fff8a10dffc ; cerror
0x7fff8a10ce49: ret
0x7fff8a10ce4a: nop
(lldb)

Note: We see LLDB listed 6 threads with their TIDs numbered from 0. Also we have code disassembly starting from
the next instruction that was to be executed if dump wasn’t saved. The nice feature is annotated disassembly that
shows symbolic names for jump and call destinations.

2. List all threads:

(lldb) thread list

Process 0 stopped
* thread #1: tid = 0x0000, 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10,
stop reason = signal SIGSTOP
thread #2: tid = 0x0001, 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10,
stop reason = signal SIGSTOP
thread #3: tid = 0x0002, 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10,
stop reason = signal SIGSTOP
thread #4: tid = 0x0003, 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10,
stop reason = signal SIGSTOP
thread #5: tid = 0x0004, 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10,
stop reason = signal SIGSTOP
thread #6: tid = 0x0005, 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10,
stop reason = signal SIGSTOP

Note: Compared to GDB here threads are listed according to increasing thread number order.

3. Get all thread stack traces:

(lldb) thread backtrace all

* thread #1: tid = 0x0000, 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10,

stop reason = signal SIGSTOP
frame #0: 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10
frame #1: 0x00007fff84d6edea libsystem_c.dylib`nanosleep + 164
frame #2: 0x00007fff84d6ec2c libsystem_c.dylib`sleep + 61
frame #3: 0x00007fff84d6ec08 libsystem_c.dylib`sleep + 25
frame #4: 0x000000010390bcc3 App1`main + 195
frame #5: 0x000000010390ba14 App1`start + 52

55
 thread #2: tid = 0x0001, 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10,
stop reason = signal SIGSTOP
frame #0: 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10
frame #1: 0x00007fff84d6edea libsystem_c.dylib`nanosleep + 164
frame #2: 0x00007fff84d6ec2c libsystem_c.dylib`sleep + 61
frame #3: 0x00007fff84d6ec08 libsystem_c.dylib`sleep + 25
frame #4: 0x000000010390ba32 App1`bar_one + 18
frame #5: 0x000000010390ba49 App1`foo_one + 9
frame #6: 0x000000010390ba61 App1`thread_one + 17
frame #7: 0x00007fff84db88bf libsystem_c.dylib`_pthread_start + 335
frame #8: 0x00007fff84dbbb75 libsystem_c.dylib`thread_start + 13

thread #3: tid = 0x0002, 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10,

stop reason = signal SIGSTOP
frame #0: 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10
frame #1: 0x00007fff84d6edea libsystem_c.dylib`nanosleep + 164
frame #2: 0x00007fff84d6ec2c libsystem_c.dylib`sleep + 61
frame #3: 0x00007fff84d6ec08 libsystem_c.dylib`sleep + 25
frame #4: 0x000000010390ba92 App1`bar_two + 18
frame #5: 0x000000010390baa9 App1`foo_two + 9
frame #6: 0x000000010390bac1 App1`thread_two + 17
frame #7: 0x00007fff84db88bf libsystem_c.dylib`_pthread_start + 335
frame #8: 0x00007fff84dbbb75 libsystem_c.dylib`thread_start + 13

thread #4: tid = 0x0003, 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10,

stop reason = signal SIGSTOP
frame #0: 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10
frame #1: 0x00007fff84d6edea libsystem_c.dylib`nanosleep + 164
frame #2: 0x00007fff84d6ec2c libsystem_c.dylib`sleep + 61
frame #3: 0x00007fff84d6ec08 libsystem_c.dylib`sleep + 25
frame #4: 0x000000010390baf2 App1`bar_three + 18
frame #5: 0x000000010390bb09 App1`foo_three + 9
frame #6: 0x000000010390bb21 App1`thread_three + 17
frame #7: 0x00007fff84db88bf libsystem_c.dylib`_pthread_start + 335
frame #8: 0x00007fff84dbbb75 libsystem_c.dylib`thread_start + 13

thread #5: tid = 0x0004, 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10,

stop reason = signal SIGSTOP
frame #0: 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10
frame #1: 0x00007fff84d6edea libsystem_c.dylib`nanosleep + 164
frame #2: 0x00007fff84d6ec2c libsystem_c.dylib`sleep + 61
frame #3: 0x00007fff84d6ec08 libsystem_c.dylib`sleep + 25
frame #4: 0x000000010390bb52 App1`bar_four + 18
frame #5: 0x000000010390bb69 App1`foo_four + 9
frame #6: 0x000000010390bb81 App1`thread_four + 17
frame #7: 0x00007fff84db88bf libsystem_c.dylib`_pthread_start + 335
frame #8: 0x00007fff84dbbb75 libsystem_c.dylib`thread_start + 13

thread #6: tid = 0x0005, 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10,

stop reason = signal SIGSTOP
frame #0: 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10
frame #1: 0x00007fff84d6edea libsystem_c.dylib`nanosleep + 164
frame #2: 0x00007fff84d6ec2c libsystem_c.dylib`sleep + 61
frame #3: 0x00007fff84d6ec08 libsystem_c.dylib`sleep + 25
frame #4: 0x000000010390bbb2 App1`bar_five + 18
frame #5: 0x000000010390bbc9 App1`foo_five + 9
frame #6: 0x000000010390bbe1 App1`thread_five + 17
frame #7: 0x00007fff84db88bf libsystem_c.dylib`_pthread_start + 335
frame #8: 0x00007fff84dbbb75 libsystem_c.dylib`thread_start + 13

56
4. Switch to the thread #3 and get its stack trace:

(lldb) thread select 3

* thread #3: tid = 0x0002, 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10,
stop reason = signal SIGSTOP
frame #0: 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10
libsystem_kernel.dylib`__semwait_signal + 10:
-> 0x7fff8a10ce42: jae 0x7fff8a10ce49 ; __semwait_signal + 17
0x7fff8a10ce44: jmpq 0x7fff8a10dffc ; cerror
0x7fff8a10ce49: ret
0x7fff8a10ce4a: nop

(lldb) bt
* thread #3: tid = 0x0002, 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10,
stop reason = signal SIGSTOP
frame #0: 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10
frame #1: 0x00007fff84d6edea libsystem_c.dylib`nanosleep + 164
frame #2: 0x00007fff84d6ec2c libsystem_c.dylib`sleep + 61
frame #3: 0x00007fff84d6ec08 libsystem_c.dylib`sleep + 25
frame #4: 0x000000010390ba92 App1`bar_two + 18
frame #5: 0x000000010390baa9 App1`foo_two + 9
frame #6: 0x000000010390bac1 App1`thread_two + 17
frame #7: 0x00007fff84db88bf libsystem_c.dylib`_pthread_start + 335
frame #8: 0x00007fff84dbbb75 libsystem_c.dylib`thread_start + 13

Note: We can also list any thread stack trace without switching to it:

(lldb) thread backtrace 4

thread #4: tid = 0x0003, 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10,
stop reason = signal SIGSTOP
frame #0: 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10
frame #1: 0x00007fff84d6edea libsystem_c.dylib`nanosleep + 164
frame #2: 0x00007fff84d6ec2c libsystem_c.dylib`sleep + 61
frame #3: 0x00007fff84d6ec08 libsystem_c.dylib`sleep + 25
frame #4: 0x000000010390baf2 App1`bar_three + 18
frame #5: 0x000000010390bb09 App1`foo_three + 9
frame #6: 0x000000010390bb21 App1`thread_three + 17
frame #7: 0x00007fff84db88bf libsystem_c.dylib`_pthread_start + 335
frame #8: 0x00007fff84dbbb75 libsystem_c.dylib`thread_start + 13

5. Check that bar_two called sleep function:

(lldb) di -n bar_two
App1`bar_two:
0x10390ba80: pushq %rbp
0x10390ba81: movq %rsp, %rbp
0x10390ba84: subq $16, %rsp
0x10390ba88: movl $4294967295, %edi
0x10390ba8d: callq 0x10390bce0 ; symbol stub for: sleep
0x10390ba92: movl %eax, -4(%rbp)
0x10390ba95: addq $16, %rsp
0x10390ba99: popq %rbp
0x10390ba9a: ret
0x10390ba9b: nopl (%rax,%rax)

57
(lldb) bt
* thread #3: tid = 0x0002, 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10,
stop reason = signal SIGSTOP
frame #0: 0x00007fff8a10ce42 libsystem_kernel.dylib`__semwait_signal + 10
frame #1: 0x00007fff84d6edea libsystem_c.dylib`nanosleep + 164
frame #2: 0x00007fff84d6ec2c libsystem_c.dylib`sleep + 61
frame #3: 0x00007fff84d6ec08 libsystem_c.dylib`sleep + 25
frame #4: 0x000000010390ba92 App1`bar_two + 18
frame #5: 0x000000010390baa9 App1`foo_two + 9
frame #6: 0x000000010390bac1 App1`thread_two + 17
frame #7: 0x00007fff84db88bf libsystem_c.dylib`_pthread_start + 335
frame #8: 0x00007fff84dbbb75 libsystem_c.dylib`thread_start + 13

6. Compare with Intel disassembly flavor:

(lldb) settings set target.x86-disassembly-flavor intel

(lldb) di -n bar_two
App1`bar_two:
0x10390ba80: push RBP
0x10390ba81: mov RBP, RSP
0x10390ba84: sub RSP, 16
0x10390ba88: mov EDI, 4294967295
0x10390ba8d: call 0x10390bce0 ; symbol stub for: sleep
0x10390ba92: mov DWORD PTR [RBP - 4], EAX
0x10390ba95: add RSP, 16
0x10390ba99: pop RBP
0x10390ba9a: ret
0x10390ba9b: nop DWORD PTR [RAX + RAX]

(lldb) set disassembly-flavor att

(lldb)

7. Follow bar_two function to sleep function code:

(lldb) di -n bar_two
App1`bar_two:
0x10390ba80: pushq %rbp
0x10390ba81: movq %rsp, %rbp
0x10390ba84: subq $16, %rsp
0x10390ba88: movl $4294967295, %edi
0x10390ba8d: callq 0x10390bce0 ; symbol stub for: sleep
0x10390ba92: movl %eax, -4(%rbp)
0x10390ba95: addq $16, %rsp
0x10390ba99: popq %rbp
0x10390ba9a: ret
0x10390ba9b: nopl (%rax,%rax)

(lldb) di -a 0x10390bce0
App1`symbol stub for: sleep:
0x10390bce0: jmpq *866(%rip) ; (void *)0x00007fff84d6ebef: sleep

58
8. Disassemble the annotated value:

(lldb) di -a 0x00007fff84d6ebef
libsystem_c.dylib`sleep:
0x7fff84d6ebef: pushq %rbp
0x7fff84d6ebf0: movq %rsp, %rbp
0x7fff84d6ebf3: pushq %rbx
0x7fff84d6ebf4: subq $40, %rsp
0x7fff84d6ebf8: testl %edi, %edi
0x7fff84d6ebfa: movl %edi, %ebx
0x7fff84d6ebfc: jns 0x7fff84d6ec11 ; sleep + 34
0x7fff84d6ebfe: movl $2147483647, %edi
0x7fff84d6ec03: callq 0x7fff84d6ebef ; sleep
0x7fff84d6ec08: leal -2147483647(%rbx,%rax), %eax
0x7fff84d6ec0f: jmp 0x7fff84d6ec4f ; sleep + 96
0x7fff84d6ec11: movl %ebx, %eax
0x7fff84d6ec13: movq %rax, -24(%rbp)
0x7fff84d6ec17: movq $0, -16(%rbp)
0x7fff84d6ec1f: leaq -24(%rbp), %rdi
0x7fff84d6ec23: leaq -40(%rbp), %rsi
0x7fff84d6ec27: callq 0x7fff84d6ed46 ; nanosleep
0x7fff84d6ec2c: cmpl $-1, %eax
0x7fff84d6ec2f: je 0x7fff84d6ec37 ; sleep + 72
0x7fff84d6ec31: xorl %ebx, %ebx
0x7fff84d6ec33: movl %ebx, %eax
0x7fff84d6ec35: jmp 0x7fff84d6ec4f ; sleep + 96
0x7fff84d6ec37: callq 0x7fff84e0cc88 ; __error
0x7fff84d6ec3c: cmpl $4, (%rax)
0x7fff84d6ec3f: jne 0x7fff84d6ec33 ; sleep + 68
0x7fff84d6ec41: cmpq $0, -32(%rbp)
0x7fff84d6ec46: setne %al
0x7fff84d6ec49: movzbl %al, %eax
0x7fff84d6ec4c: addl -40(%rbp), %eax
0x7fff84d6ec4f: addq $40, %rsp
0x7fff84d6ec53: popq %rbx
0x7fff84d6ec54: popq %rbp

9. Compare stack trace for thread #3 (core thread 2) and its module info with the diagnostic report
App1_1394.crash:

Process: App1 [1394]

Path: /Users/USER/Documents/*/App1
Identifier: App1
Version: ??? (???)
Code Type: X86-64 (Native)
Parent Process: bash [661]

Date/Time: 2012-07-24 00:20:26.078 +0100

OS Version: Mac OS X 10.7.4 (11E53)
Report Version: 9

Crashed Thread: 0 Dispatch queue: com.apple.main-thread

Exception Type: EXC_CRASH (SIGABRT)

Exception Codes: 0x0000000000000000, 0x0000000000000000

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread

0 libsystem_kernel.dylib 0x00007fff8a10ce42 __semwait_signal + 10
1 libsystem_c.dylib 0x00007fff84d6edea nanosleep + 164
2 libsystem_c.dylib 0x00007fff84d6ec2c sleep + 61
3 libsystem_c.dylib 0x00007fff84d6ec08 sleep + 25
4 App1 0x000000010390bcc3 main + 195
5 App1 0x000000010390ba14 start + 52

59
Thread 1:
0 libsystem_kernel.dylib 0x00007fff8a10ce42 __semwait_signal + 10
1 libsystem_c.dylib 0x00007fff84d6edea nanosleep + 164
2 libsystem_c.dylib 0x00007fff84d6ec2c sleep + 61
3 libsystem_c.dylib 0x00007fff84d6ec08 sleep + 25
4 App1 0x000000010390ba32 bar_one + 18
5 App1 0x000000010390ba49 foo_one + 9
6 App1 0x000000010390ba61 thread_one + 17
7 libsystem_c.dylib 0x00007fff84db88bf _pthread_start + 335
8 libsystem_c.dylib 0x00007fff84dbbb75 thread_start + 13

Thread 2:
0 libsystem_kernel.dylib 0x00007fff8a10ce42 __semwait_signal + 10
1 libsystem_c.dylib 0x00007fff84d6edea nanosleep + 164
2 libsystem_c.dylib 0x00007fff84d6ec2c sleep + 61
3 libsystem_c.dylib 0x00007fff84d6ec08 sleep + 25
4 App1 0x000000010390ba92 bar_two + 18
5 App1 0x000000010390baa9 foo_two + 9
6 App1 0x000000010390bac1 thread_two + 17
7 libsystem_c.dylib 0x00007fff84db88bf _pthread_start + 335
8 libsystem_c.dylib 0x00007fff84dbbb75 thread_start + 13

Thread 3:
0 libsystem_kernel.dylib 0x00007fff8a10ce42 __semwait_signal + 10
1 libsystem_c.dylib 0x00007fff84d6edea nanosleep + 164
2 libsystem_c.dylib 0x00007fff84d6ec2c sleep + 61
3 libsystem_c.dylib 0x00007fff84d6ec08 sleep + 25
4 App1 0x000000010390baf2 bar_three + 18
5 App1 0x000000010390bb09 foo_three + 9
6 App1 0x000000010390bb21 thread_three + 17
7 libsystem_c.dylib 0x00007fff84db88bf _pthread_start + 335
8 libsystem_c.dylib 0x00007fff84dbbb75 thread_start + 13

Thread 4:
0 libsystem_kernel.dylib 0x00007fff8a10ce42 __semwait_signal + 10
1 libsystem_c.dylib 0x00007fff84d6edea nanosleep + 164
2 libsystem_c.dylib 0x00007fff84d6ec2c sleep + 61
3 libsystem_c.dylib 0x00007fff84d6ec08 sleep + 25
4 App1 0x000000010390bb52 bar_four + 18
5 App1 0x000000010390bb69 foo_four + 9
6 App1 0x000000010390bb81 thread_four + 17
7 libsystem_c.dylib 0x00007fff84db88bf _pthread_start + 335
8 libsystem_c.dylib 0x00007fff84dbbb75 thread_start + 13

Thread 5:
0 libsystem_kernel.dylib 0x00007fff8a10ce42 __semwait_signal + 10
1 libsystem_c.dylib 0x00007fff84d6edea nanosleep + 164
2 libsystem_c.dylib 0x00007fff84d6ec2c sleep + 61
3 libsystem_c.dylib 0x00007fff84d6ec08 sleep + 25
4 App1 0x000000010390bbb2 bar_five + 18
5 App1 0x000000010390bbc9 foo_five + 9
6 App1 0x000000010390bbe1 thread_five + 17
7 libsystem_c.dylib 0x00007fff84db88bf _pthread_start + 335
8 libsystem_c.dylib 0x00007fff84dbbb75 thread_start + 13

Thread 0 crashed with X86 Thread State (64-bit):

rax: 0x0000000000000004 rbx: 0x00007fff6350aa08 rcx: 0x00007fff6350a9c8 rdx: 0x0000000000000001
rdi: 0x0000000000000c03 rsi: 0x0000000000000000 rbp: 0x00007fff6350a9f0 rsp: 0x00007fff6350a9c8
r8: 0x000000007fffffff r9: 0x0000000000000000 r10: 0x0000000000000001 r11: 0xffffff80002da8d0
r12: 0x0000000000000000 r13: 0x0000000000000000 r14: 0x00007fff6350aa18 r15: 0x0000000000000000
rip: 0x00007fff8a10ce42 rfl: 0x0000000000000247 cr2: 0x0000000103d0b880
Logical CPU: 0

60
Binary Images:
0x10390b000 - 0x10390bfff +App1 (??? - ???) <5BC0342F-7E97-3A7D-8EA6-75A0468021EA>
/Users/USER/Documents/*/App1
0x7fff6350b000 - 0x7fff6353fbaf dyld (195.6 - ???) <0CD1B35B-A28F-32DA-B72E-452EAD609613> /usr/lib/dyld
0x7fff849f2000 - 0x7fff84a0ffff libxpc.dylib (77.19.0 - compatibility 1.0.0) <9F57891B-D7EF-3050-BEDD-
21E7C6668248> /usr/lib/system/libxpc.dylib
0x7fff84d68000 - 0x7fff84d69ff7 libsystem_blocks.dylib (53.0.0 - compatibility 1.0.0) <8BCA214A-8992-34B2-
A8B9-B74DEACA1869> /usr/lib/system/libsystem_blocks.dylib
0x7fff84d6a000 - 0x7fff84e47fef libsystem_c.dylib (763.13.0 - compatibility 1.0.0) <41B43515-2806-3FBC-ACF1-
A16F35B7E290> /usr/lib/system/libsystem_c.dylib
0x7fff85022000 - 0x7fff85030fff libdispatch.dylib (187.9.0 - compatibility 1.0.0) <1D5BE322-A9B9-3BCE-8FAC-
076FB07CF54A> /usr/lib/system/libdispatch.dylib
0x7fff855f0000 - 0x7fff855f1fff libunc.dylib (24.0.0 - compatibility 1.0.0) <337960EE-0A85-3DD0-A760-
7134CF4C0AFF> /usr/lib/system/libunc.dylib
0x7fff85ae3000 - 0x7fff85ae4ff7 libremovefile.dylib (21.1.0 - compatibility 1.0.0) <739E6C83-AA52-3C6C-A680-
B37FE2888A04> /usr/lib/system/libremovefile.dylib
0x7fff89114000 - 0x7fff89118fff libmathCommon.A.dylib (2026.0.0 - compatibility 1.0.0) <FF83AFF7-42B2-306E-
90AF-D539C51A4542> /usr/lib/system/libmathCommon.A.dylib
0x7fff89119000 - 0x7fff8911dfff libdyld.dylib (195.5.0 - compatibility 1.0.0) <380C3F44-0CA7-3514-8080-
46D1C9DF4FCD> /usr/lib/system/libdyld.dylib
0x7fff89740000 - 0x7fff89741ff7 libsystem_sandbox.dylib (??? - ???) <96D38E74-F18F-3CCB-A20B-E8E3ADC4E166>
/usr/lib/system/libsystem_sandbox.dylib
0x7fff8a0ef000 - 0x7fff8a0f5fff libmacho.dylib (800.0.0 - compatibility 1.0.0) <165514D7-1BFA-38EF-A151-
676DCD21FB64> /usr/lib/system/libmacho.dylib
0x7fff8a0f6000 - 0x7fff8a116fff libsystem_kernel.dylib (1699.26.8 - compatibility 1.0.0) <1DDC0B0F-DB2A-34D6-
895D-E5B2B5618946> /usr/lib/system/libsystem_kernel.dylib
0x7fff8a2ac000 - 0x7fff8a2b4fff libsystem_dnssd.dylib (??? - ???) <D9BB1F87-A42B-3CBC-9DC2-FC07FCEF0016>
/usr/lib/system/libsystem_dnssd.dylib
0x7fff8ae26000 - 0x7fff8ae61fff libsystem_info.dylib (??? - ???) <35F90252-2AE1-32C5-8D34-782C614D9639>
/usr/lib/system/libsystem_info.dylib
0x7fff8b248000 - 0x7fff8b24afff libquarantine.dylib (36.6.0 - compatibility 1.0.0) <0EBF714B-4B69-3E1F-9A7D-
6BBC2AACB310> /usr/lib/system/libquarantine.dylib
0x7fff8b3b4000 - 0x7fff8b3b4fff libkeymgr.dylib (23.0.0 - compatibility 1.0.0) <61EFED6A-A407-301E-B454-
CD18314F0075> /usr/lib/system/libkeymgr.dylib
0x7fff8b3dd000 - 0x7fff8b3e2fff libcompiler_rt.dylib (6.0.0 - compatibility 1.0.0) <98ECD5F6-E85C-32A5-98CD-
8911230CB66A> /usr/lib/system/libcompiler_rt.dylib
0x7fff8bd1a000 - 0x7fff8bd1bfff libdnsinfo.dylib (395.11.0 - compatibility 1.0.0) <853BAAA5-270F-3FDC-B025-
D448DB72E1C3> /usr/lib/system/libdnsinfo.dylib
0x7fff8c528000 - 0x7fff8c52dff7 libsystem_network.dylib (??? - ???) <5DE7024E-1D2D-34A2-80F4-08326331A75B>
/usr/lib/system/libsystem_network.dylib
0x7fff8cfa3000 - 0x7fff8cfadff7 liblaunch.dylib (392.38.0 - compatibility 1.0.0) <6ECB7F19-B384-32C1-8652-
2463C1CF4815> /usr/lib/system/liblaunch.dylib
0x7fff8fe02000 - 0x7fff8fe09fff libcopyfile.dylib (85.1.0 - compatibility 1.0.0) <0AB51EE2-E914-358C-AC19-
47BC024BDAE7> /usr/lib/system/libcopyfile.dylib
0x7fff8fe4b000 - 0x7fff8fe8dff7 libcommonCrypto.dylib (55010.0.0 - compatibility 1.0.0) <BB770C22-8C57-365A-
8716-4A3C36AE7BFB> /usr/lib/system/libcommonCrypto.dylib
0x7fff90c0f000 - 0x7fff90c18ff7 libsystem_notify.dylib (80.1.0 - compatibility 1.0.0) <A4D651E3-D1C6-3934-
AD49-7A104FD14596> /usr/lib/system/libsystem_notify.dylib
0x7fff91376000 - 0x7fff913a3fe7 libSystem.B.dylib (159.1.0 - compatibility 1.0.0) <7BEBB139-50BB-3112-947A-
F4AA168F991C> /usr/lib/libSystem.B.dylib
0x7fff91489000 - 0x7fff9148fff7 libunwind.dylib (30.0.0 - compatibility 1.0.0) <1E9C6C8C-CBE8-3F4B-A5B5-
E03E3AB53231> /usr/lib/system/libunwind.dylib
0x7fff91a22000 - 0x7fff91a27fff libcache.dylib (47.0.0 - compatibility 1.0.0) <1571C3AB-BCB2-38CD-B3B2-
C5FC3F927C6A> /usr/lib/system/libcache.dylib

External Modification Summary:

Calls made by other processes targeting this process:
task_for_pid: 2
thread_create: 0
thread_set_state: 0
Calls made by this process:
task_for_pid: 0
thread_create: 0
thread_set_state: 0
Calls made by all processes on this machine:
task_for_pid: 2696
thread_create: 0
thread_set_state: 0

61
VM Region Summary:
ReadOnly portion of Libraries: Total=50.2M resident=50.2M(100%) swapped_out_or_unallocated=0K(0%)
Writable regions: Total=38.9M written=10.8M(28%) resident=42.6M(110%) swapped_out=0K(0%)
unallocated=16777216.0T(45221404475392%)

REGION TYPE VIRTUAL

=========== =======
MALLOC 1220K
Stack 66.6M
__DATA 464K
__LINKEDIT 47.7M
__TEXT 2484K
shared memory 12K
=========== =======
TOTAL 118.4M

10. Get App1 data section from the output of vmmap_1394.log:

Virtual Memory Map of process 1394 (App1)

Output report format: 2.2 -- 64-bit process

==== Non-writable regions for process 1394

__TEXT 000000010390b000-000000010390c000 [ 4K] r-x/rwx SM=COW /Users/DumpAnalysis/Documents/AMCDA-
Dumps/Apps/App1/Build/Products/Release/App1

[...]

==== Writable regions for process 1394

__DATA 000000010390c000-000000010390d000 [ 4K] rw-/rwx SM=PRV /Users/DumpAnalysis/Documents/AMCDA-
Dumps/Apps/App1/Build/Products/Release/App1

[...]

11. Compare with the section information in the core dump:

(lldb) image dump sections App1

Sections for '/Users/DumpAnalysis/Documents/AMCDA-Dumps/Apps/App1/Build/Products/Release/App1' (x86_64):
SectID Type Load Address File Off. File Size Flags Section Name
---------- ---------------- --------------------------------------- ---------- ---------- ---------- ----------------------------
0x00000100 container [0x0000000000000000-0x0000000100000000)* 0x00000000 0x00000000 0x00000000 App1.__PAGEZERO
0x00000200 container [0x000000010390b000-0x000000010390c000) 0x00000000 0x00001000 0x00000000 App1.__TEXT
0x00000001 code [0x000000010390b9e0-0x000000010390bcd3) 0x000009e0 0x000002f3 0x80000400 App1.__TEXT.__text
0x00000002 code [0x000000010390bcd4-0x000000010390bce6) 0x00000cd4 0x00000012 0x80000408 App1.__TEXT.__stubs
0x00000003 code [0x000000010390bce8-0x000000010390bd16) 0x00000ce8 0x0000002e 0x80000400 App1.__TEXT.__stub_helper
0x00000004 code [0x000000010390bd16-0x000000010390bd66) 0x00000d16 0x00000050 0x00000000 App1.__TEXT.__unwind_info
0x00000005 eh-frame [0x000000010390bd68-0x000000010390c000) 0x00000d68 0x00000298 0x00000000 App1.__TEXT.__eh_frame
0x00000300 container [0x000000010390c000-0x000000010390d000) 0x00001000 0x00001000 0x00000000 App1.__DATA
0x00000006 data [0x000000010390c000-0x000000010390c028) 0x00001000 0x00000028 0x00000000 App1.__DATA.__program_vars
0x00000007 data-ptrs [0x000000010390c028-0x000000010390c038) 0x00001028 0x00000010 0x00000006 App1.__DATA.__nl_symbol_ptr
0x00000008 data-ptrs [0x000000010390c038-0x000000010390c050) 0x00001038 0x00000018 0x00000007 App1.__DATA.__la_symbol_ptr
0x00000009 zero-fill [0x000000010390c050-0x000000010390c070) 0x00000000 0x00000000 0x00000001 App1.__DATA.__common
0x00000400 container [0x000000010390d000-0x000000010390d3b0) 0x00002000 0x000003b0 0x00000000 App1.__LINKEDIT

12. Dump data with possible symbolic information:

(lldb) x/512a 0x000000010390c000

error: Normally, 'memory read' will not read over 1024 bytes of data.
error: Please use --force to override this restriction just once.
error: or set target.max-memory-read-size if you will often need a larger limit.

62
(lldb) x/512a 0x000000010390c000 --force
0x10390c000: 0x000000010390b000
0x10390c008: 0x000000010390c050 App1`NXArgc
0x10390c010: 0x000000010390c058 App1`NXArgv
0x10390c018: 0x000000010390c060 App1`environ
0x10390c020: 0x000000010390c068
0x10390c028: 0x00007fff8911a6a0 libdyld.dylib`dyld_stub_binder
0x10390c030: 0x00007fff63546d80 dyld`initialPoolContent + 2128
0x10390c038: 0x000000010390bcf8
0x10390c040: 0x00007fff84dbab01 libsystem_c.dylib`pthread_create
0x10390c048: 0x00007fff84d6ebef libsystem_c.dylib`sleep
0x10390c050: 0x0000000000000001
0x10390c058: 0x00007fff6350aaf0
0x10390c060: 0x00007fff6350ab00
0x10390c068: 0x00007fff6350ac73
0x10390c070: 0x0000000000000000
0x10390c078: 0x0000000000000000
0x10390c080: 0x0000000000000000
0x10390c088: 0x0000000000000000
0x10390c090: 0x0000000000000000
[...]

13. Dump the contents of memory pointed to by environ variable in null-terminated string format:

(lldb) x/100s 0x00007fff6350ab00

[...]
0x7fff6350abd5: ""
0x7fff6350abd6: ""
0x7fff6350abd7: ""
0x7fff6350abd8: "/Users/DumpAnalysis/Documents/AMCDA-Dumps/Apps/App1/Build/Products/Release/App1"
0x7fff6350ac28: "/Users/DumpAnalysis/Documents/AMCDA-Dumps/Apps/App1/Build/Products/Release/App1"
0x7fff6350ac78: "TERM_PROGRAM=Apple_Terminal"
0x7fff6350ac94: "TERM=xterm-256color"
0x7fff6350aca8: "SHELL=/bin/bash"
0x7fff6350acb8: "TMPDIR=/var/folders/ww/rmtqfhl93yj4213dnl2rqy6w0000gn/T/"
0x7fff6350acf1: "Apple_PubSub_Socket_Render=/tmp/launch-mYEvtN/Render"
0x7fff6350ad26: "TERM_PROGRAM_VERSION=303.2"
0x7fff6350ad41: "TERM_SESSION_ID=2B039506-8384-4620-B354-120BE31AEA84"
0x7fff6350ad76: "USER=DumpAnalysis"
0x7fff6350ad88: "COMMAND_MODE=unix2003"
0x7fff6350ad9e: "SSH_AUTH_SOCK=/tmp/launch-9sm7dH/Listeners"
0x7fff6350adc9: "__CF_USER_TEXT_ENCODING=0x1F5:0:0"
0x7fff6350adeb: "Apple_Ubiquity_Message=/tmp/launch-tWsFs8/Apple_Ubiquity_Message"
0x7fff6350ae2c:
"PATH=/Applications/Xcode.app/Contents/Developer/usr/bin/:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin"
0x7fff6350ae9f: "PWD=/Users/DumpAnalysis"
0x7fff6350aeb7: "LANG=en_IE.UTF-8"
---Type <return> to continue, or q <return> to quit---
0x7fff6350aec8: "SHLVL=1"
0x7fff6350aed0: "HOME=/Users/DumpAnalysis"
0x7fff6350aee9: "LOGNAME=DumpAnalysis"
0x7fff6350aefe: "DISPLAY=/tmp/launch-M8cgb1/org.x:0"
0x7fff6350af21: "SECURITYSESSIONID=186af"
0x7fff6350af39: "_=/Users/DumpAnalysis/Documents/AMCDA-Dumps/Apps/App1/Build/Products/Release/App1"
0x7fff6350af8b: "OLDPWD=/usr/share/man/man1"
0x7fff6350afa6: ""
0x7fff6350afa7: ""
0x7fff6350afa8: "stack_guard=0x74843dc6068699c3"
0x7fff6350afc7: "malloc_entropy=0x7406669509034332,0x71e4e2253a6d22b0"
0x7fff6350affc: ""
0x7fff6350affd: ""

63
14. Get the list of loaded modules:
(lldb) image list
[ 0] 5BC0342F-7E97-3A7D-8EA6-75A0468021EA 0x000000010390b000 /Users/DumpAnalysis/Documents/AMCDA-
Dumps/Apps/App1/Build/Products/Release/App1
[ 1] 7BEBB139-50BB-3112-947A-F4AA168F991C 0x00007fff91376000 /usr/lib/libSystem.B.dylib (0x00007fff91376000)
[ 2] 1571C3AB-BCB2-38CD-B3B2-C5FC3F927C6A 0x00007fff91a22000 /usr/lib/system/libcache.dylib (0x00007fff91a22000)
[ 3] BB770C22-8C57-365A-8716-4A3C36AE7BFB 0x00007fff8fe4b000 /usr/lib/system/libcommonCrypto.dylib (0x00007fff8fe4b000)
[ 4] 98ECD5F6-E85C-32A5-98CD-8911230CB66A 0x00007fff8b3dd000 /usr/lib/system/libcompiler_rt.dylib (0x00007fff8b3dd000)
[ 5] 0AB51EE2-E914-358C-AC19-47BC024BDAE7 0x00007fff8fe02000 /usr/lib/system/libcopyfile.dylib (0x00007fff8fe02000)
[ 6] 1D5BE322-A9B9-3BCE-8FAC-076FB07CF54A 0x00007fff85022000 /usr/lib/system/libdispatch.dylib (0x00007fff85022000)
[ 7] 853BAAA5-270F-3FDC-B025-D448DB72E1C3 0x00007fff8bd1a000 /usr/lib/system/libdnsinfo.dylib (0x00007fff8bd1a000)
[ 8] 380C3F44-0CA7-3514-8080-46D1C9DF4FCD 0x00007fff89119000 /usr/lib/system/libdyld.dylib (0x00007fff89119000)
[ 9] 61EFED6A-A407-301E-B454-CD18314F0075 0x00007fff8b3b4000 /usr/lib/system/libkeymgr.dylib (0x00007fff8b3b4000)
[ 10] 6ECB7F19-B384-32C1-8652-2463C1CF4815 0x00007fff8cfa3000 /usr/lib/system/liblaunch.dylib (0x00007fff8cfa3000)
[ 11] 165514D7-1BFA-38EF-A151-676DCD21FB64 0x00007fff8a0ef000 /usr/lib/system/libmacho.dylib (0x00007fff8a0ef000)
[ 12] FF83AFF7-42B2-306E-90AF-D539C51A4542 0x00007fff89114000 /usr/lib/system/libmathCommon.A.dylib (0x00007fff89114000)
[ 13] 0EBF714B-4B69-3E1F-9A7D-6BBC2AACB310 0x00007fff8b248000 /usr/lib/system/libquarantine.dylib (0x00007fff8b248000)
[ 14] 739E6C83-AA52-3C6C-A680-B37FE2888A04 0x00007fff85ae3000 /usr/lib/system/libremovefile.dylib (0x00007fff85ae3000)
[ 15] 8BCA214A-8992-34B2-A8B9-B74DEACA1869 0x00007fff84d68000 /usr/lib/system/libsystem_blocks.dylib (0x00007fff84d68000)
[ 16] 41B43515-2806-3FBC-ACF1-A16F35B7E290 0x00007fff84d6a000 /usr/lib/system/libsystem_c.dylib (0x00007fff84d6a000)
[ 17] D9BB1F87-A42B-3CBC-9DC2-FC07FCEF0016 0x00007fff8a2ac000 /usr/lib/system/libsystem_dnssd.dylib (0x00007fff8a2ac000)
[ 18] 35F90252-2AE1-32C5-8D34-782C614D9639 0x00007fff8ae26000 /usr/lib/system/libsystem_info.dylib (0x00007fff8ae26000)
[ 19] 1DDC0B0F-DB2A-34D6-895D-E5B2B5618946 0x00007fff8a0f6000 /usr/lib/system/libsystem_kernel.dylib (0x00007fff8a0f6000)
[ 20] 5DE7024E-1D2D-34A2-80F4-08326331A75B 0x00007fff8c528000 /usr/lib/system/libsystem_network.dylib (0x00007fff8c528000)
[ 21] A4D651E3-D1C6-3934-AD49-7A104FD14596 0x00007fff90c0f000 /usr/lib/system/libsystem_notify.dylib (0x00007fff90c0f000)
[ 22] 96D38E74-F18F-3CCB-A20B-E8E3ADC4E166 0x00007fff89740000 /usr/lib/system/libsystem_sandbox.dylib (0x00007fff89740000)
[ 23] 337960EE-0A85-3DD0-A760-7134CF4C0AFF 0x00007fff855f0000 /usr/lib/system/libunc.dylib (0x00007fff855f0000)
[ 24] 1E9C6C8C-CBE8-3F4B-A5B5-E03E3AB53231 0x00007fff91489000 /usr/lib/system/libunwind.dylib (0x00007fff91489000)
[ 25] 9F57891B-D7EF-3050-BEDD-21E7C6668248 0x00007fff849f2000 /usr/lib/system/libxpc.dylib (0x00007fff849f2000)
[ 26] 0CD1B35B-A28F-32DA-B72E-452EAD609613 0x00007fff6350b000 /usr/lib/dyld (0x00007fff6350b000)
(lldb)

64
Published by OpenTask, Republic of Ireland

Copyright © 2015 by OpenTask

Copyright © 2015 by Software Diagnostics Services

Copyright © 2015 by Dmitry Vostokov

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or
transmitted, in any form or by any means, without the prior written permission of the
publisher.

You must not circulate this book in any other binding or cover, and you must impose the same
condition on any acquirer.

Product and company names mentioned in this book may be trademarks of their owners.

OpenTask books and magazines are available through booksellers and distributors worldwide.
For further information or comments send requests to [email protected].

A CIP catalog record for this book is available from the British Library.

ISBN-l3: 978-1-908043-97-9 (Paperback)

1st printing, 2015

2
Contents

Presentation Slides and Transcript .................................................................................................................................5

Core Dump Collection...................................................................................................................................................25
Practice Exercises .........................................................................................................................................................31
Exercise 0..................................................................................................................................................................36
Exercise A1 ...............................................................................................................................................................40
Exercise A2D .............................................................................................................................................................53
Exercise A2C .............................................................................................................................................................58
Exercise A3 ...............................................................................................................................................................62
Exercise A4 ...............................................................................................................................................................66
Exercise A5 ...............................................................................................................................................................72
Exercise A6 ...............................................................................................................................................................76
Exercise A7 ...............................................................................................................................................................93
Exercise A8 .............................................................................................................................................................102
Exercise A9 .............................................................................................................................................................117
Exercise A10 ...........................................................................................................................................................132
Exercise A11 ...........................................................................................................................................................149
Exercise A12 ...........................................................................................................................................................157
App Source Code ........................................................................................................................................................171
App0 .......................................................................................................................................................................173
App1 .......................................................................................................................................................................174
App2D .....................................................................................................................................................................175
App2C .....................................................................................................................................................................177
App3 .......................................................................................................................................................................179
App4 .......................................................................................................................................................................181
App5 .......................................................................................................................................................................183
App6 .......................................................................................................................................................................185
App7 .......................................................................................................................................................................187
App8 .......................................................................................................................................................................189
App9 .......................................................................................................................................................................191
App10 .....................................................................................................................................................................193
App11 / App12 .......................................................................................................................................................195
Selected Patterns .......................................................................................................................................................197
NULL Pointer (data) ................................................................................................................................................199
3
Incomplete Stack Trace ..........................................................................................................................................200
Stack Trace .............................................................................................................................................................201
NULL Pointer (code)................................................................................................................................................202
Spiking Thread ........................................................................................................................................................203
Dynamic Memory Corruption (process heap) .........................................................................................................204
Execution Residue ..................................................................................................................................................205
Coincidental Symbolic Information.........................................................................................................................207
Stack Overflow (user mode) ...................................................................................................................................208
Divide by Zero (user mode) ....................................................................................................................................209
Local Buffer Overflow .............................................................................................................................................210
C++ Exception .........................................................................................................................................................211
Paratext ..................................................................................................................................................................212
Active Thread .........................................................................................................................................................213
Lateral Damage.......................................................................................................................................................214
Critical Region .........................................................................................................................................................215

4
 Exercise A1

Goal: Learn how to list stack traces, disassemble functions, check their correctness, dump data, get environment.

Patterns: Manual Dump, Stack Trace, Stack Trace Collection, Annotated Disassembly, Paratext, Not My Version,
Environment Hint.

1. Load a core dump core.3308 and App1 executable:

training@debian64:~/ALCDA$ gdb -c ./App1/core.3308 -se ./App1/App1

GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/training/ALCDA/App1/App1...done.
[New LWP 3309]
[New LWP 3310]
[New LWP 3311]
[New LWP 3312]
[New LWP 3313]
[New LWP 3308]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/home/training/ALCDA/App1/App1'.
#0 0x000000000042fdf1 in nanosleep ()

2. List all threads:

(gdb) info threads

Id Target Id Frame
6 LWP 3308 0x000000000042fdf1 in nanosleep ()
5 LWP 3313 0x000000000042fdf1 in nanosleep ()
4 LWP 3312 0x000000000042fdf1 in nanosleep ()
3 LWP 3311 0x000000000042fdf1 in nanosleep ()
2 LWP 3310 0x000000000042fdf1 in nanosleep ()
* 1 LWP 3309 0x000000000042fdf1 in nanosleep ()

3. Get all thread stack traces:

(gdb) thread apply all bt

Thread 6 (LWP 3308):

#0 0x000000000042fdf1 in nanosleep ()
#1 0x000000000042fcc0 in sleep ()
#2 0x00000000004006c1 in main ()

40
Thread 5 (LWP 3313):
#0 0x000000000042fdf1 in nanosleep ()
#1 0x000000000042fcc0 in sleep ()
#2 0x00000000004005f2 in bar_five ()
#3 0x0000000000400602 in foo_five ()
#4 0x000000000040061a in thread_five ()
#5 0x00000000004015f0 in start_thread (arg=<optimized out>)
at pthread_create.c:304
#6 0x00000000004324a9 in clone ()
#7 0x0000000000000000 in ?? ()

Thread 4 (LWP 3312):

#0 0x000000000042fdf1 in nanosleep ()
#1 0x000000000042fcc0 in sleep ()
#2 0x00000000004005b5 in bar_four ()
#3 0x00000000004005c5 in foo_four ()
#4 0x00000000004005dd in thread_four ()
#5 0x00000000004015f0 in start_thread (arg=<optimized out>)
---Type <return> to continue, or q <return> to quit---
at pthread_create.c:304
#6 0x00000000004324a9 in clone ()
#7 0x0000000000000000 in ?? ()

Thread 3 (LWP 3311):

#0 0x000000000042fdf1 in nanosleep ()
#1 0x000000000042fcc0 in sleep ()
#2 0x0000000000400578 in bar_three ()
#3 0x0000000000400588 in foo_three ()
#4 0x00000000004005a0 in thread_three ()
#5 0x00000000004015f0 in start_thread (arg=<optimized out>)
at pthread_create.c:304
#6 0x00000000004324a9 in clone ()
#7 0x0000000000000000 in ?? ()

Thread 2 (LWP 3310):

#0 0x000000000042fdf1 in nanosleep ()
#1 0x000000000042fcc0 in sleep ()
#2 0x000000000040053b in bar_two ()
#3 0x000000000040054b in foo_two ()
#4 0x0000000000400563 in thread_two ()
#5 0x00000000004015f0 in start_thread (arg=<optimized out>)
at pthread_create.c:304
#6 0x00000000004324a9 in clone ()
---Type <return> to continue, or q <return> to quit---
#7 0x0000000000000000 in ?? ()

Thread 1 (LWP 3309):

#0 0x000000000042fdf1 in nanosleep ()
#1 0x000000000042fcc0 in sleep ()
#2 0x00000000004004fe in bar_one ()
#3 0x000000000040050e in foo_one ()
#4 0x0000000000400526 in thread_one ()
#5 0x00000000004015f0 in start_thread (arg=<optimized out>)
at pthread_create.c:304
#6 0x00000000004324a9 in clone ()
#7 0x0000000000000000 in ?? ()

41
4. Switch to the thread #2 and get its stack trace:

(gdb) thread 2
[Switching to thread 2 (LWP 3310)]
#0 0x000000000042fdf1 in nanosleep ()

(gdb) bt
#0 0x000000000042fdf1 in nanosleep ()
#1 0x000000000042fcc0 in sleep ()
#2 0x000000000040053b in bar_two ()
#3 0x000000000040054b in foo_two ()
#4 0x0000000000400563 in thread_two ()
#5 0x00000000004015f0 in start_thread (arg=<optimized out>)
at pthread_create.c:304
#6 0x00000000004324a9 in clone ()
#7 0x0000000000000000 in ?? ()

5. Check that bar_two called sleep function:

(gdb) disassemble bar_two

Dump of assembler code for function bar_two:
0x000000000040052d <+0>: push %rbp
0x000000000040052e <+1>: mov %rsp,%rbp
0x0000000000400531 <+4>: mov $0xffffffff,%edi
0x0000000000400536 <+9>: callq 0x42fbe0 <sleep>
0x000000000040053b <+14>: pop %rbp
0x000000000040053c <+15>: retq
End of assembler dump.

We see that the address in the stack trace for bar_two function is the address to return to after calling sleep
function.

6. Compare with Intel disassembly flavor:

(gdb) set disassembly-flavor intel

(gdb) disassemble bar_two

Dump of assembler code for function bar_two:
0x000000000040052d <+0>: push rbp
0x000000000040052e <+1>: mov rbp,rsp
0x0000000000400531 <+4>: mov edi,0xffffffff
0x0000000000400536 <+9>: call 0x42fbe0 <sleep>
0x000000000040053b <+14>: pop rbp
0x000000000040053c <+15>: ret
End of assembler dump.

(gdb) set disassembly-flavor att

42
7. Get App1 data section from the output of pmap (pmap.3308):

3308: ./App1
0000000000400000 732K r-x-- /home/training/ALCDA/App1/App1
00000000006b6000 8K rw--- /home/training/ALCDA/App1/App1
00000000006b8000 28K rw--- [ anon ]
000000000227c000 140K rw--- [ anon ]
00007f2257e66000 4K ----- [ anon ]
00007f2257e67000 8192K rw--- [ anon ]
00007f2258667000 4K ----- [ anon ]
00007f2258668000 8192K rw--- [ anon ]
00007f2258e68000 4K ----- [ anon ]
00007f2258e69000 8192K rw--- [ anon ]
00007f2259669000 4K ----- [ anon ]
00007f225966a000 8192K rw--- [ anon ]
00007f2259e6a000 4K ----- [ anon ]
00007f2259e6b000 8192K rw--- [ anon ]
00007ffc7d24d000 132K rw--- [ stack ]
00007ffc7d299000 4K r-x-- [ anon ]
ffffffffff600000 4K r-x-- [ anon ]
total 42028K

8. Compare with the section information in the core dump:

(gdb) maintenance info sections

Exec file:
`/home/training/ALCDA/App1/App1', file type elf64-x86-64.
0x00400158->0x00400178 at 0x00000158: .note.ABI-tag ALLOC LOAD READONLY DATA HAS_CONTENTS
0x00400178->0x0040019c at 0x00000178: .note.gnu.build-id ALLOC LOAD READONLY DATA HAS_CONTENTS
0x004001a0->0x004002d8 at 0x000001a0: .rela.plt ALLOC LOAD READONLY DATA HAS_CONTENTS
0x004002d8->0x004002e6 at 0x000002d8: .init ALLOC LOAD READONLY CODE HAS_CONTENTS
0x004002f0->0x004003c0 at 0x000002f0: .plt ALLOC LOAD READONLY CODE HAS_CONTENTS
0x004003c0->0x0048b1b8 at 0x000003c0: .text ALLOC LOAD READONLY CODE HAS_CONTENTS
0x0048b1c0->0x0048bd3e at 0x0008b1c0: __libc_freeres_fn ALLOC LOAD READONLY CODE HAS_CONTENTS
0x0048bd40->0x0048bda1 at 0x0008bd40: __libc_thread_freeres_fn ALLOC LOAD READONLY CODE HAS_CONTENTS
0x0048bda4->0x0048bdad at 0x0008bda4: .fini ALLOC LOAD READONLY CODE HAS_CONTENTS
0x0048bdc0->0x004a9d24 at 0x0008bdc0: .rodata ALLOC LOAD READONLY DATA HAS_CONTENTS
0x004a9d28->0x004a9d88 at 0x000a9d28: __libc_subfreeres ALLOC LOAD READONLY DATA HAS_CONTENTS
---Type <return> to continue, or q <return> to quit---
0x004a9d88->0x004a9d90 at 0x000a9d88: __libc_atexit ALLOC LOAD READONLY DATA HAS_CONTENTS
0x004a9d90->0x004a9d98 at 0x000a9d90: __libc_thread_subfreeres ALLOC LOAD READONLY DATA HAS_CONTENTS
0x004a9d98->0x004b686c at 0x000a9d98: .eh_frame ALLOC LOAD READONLY DATA HAS_CONTENTS
0x004b686c->0x004b6986 at 0x000b686c: .gcc_except_table ALLOC LOAD READONLY DATA HAS_CONTENTS
0x006b6988->0x006b69b0 at 0x000b6988: .tdata ALLOC LOAD DATA HAS_CONTENTS
0x006b69b0->0x006b69e0 at 0x000b69b0: .tbss ALLOC
0x006b69b0->0x006b69c0 at 0x000b69b0: .init_array ALLOC LOAD DATA HAS_CONTENTS
0x006b69c0->0x006b69d0 at 0x000b69c0: .fini_array ALLOC LOAD DATA HAS_CONTENTS
0x006b69d0->0x006b69d8 at 0x000b69d0: .jcr ALLOC LOAD DATA HAS_CONTENTS
0x006b69e0->0x006b6a50 at 0x000b69e0: .data.rel.ro ALLOC LOAD DATA HAS_CONTENTS
0x006b6a50->0x006b6a60 at 0x000b6a50: .got ALLOC LOAD DATA HAS_CONTENTS
0x006b6a60->0x006b6ae0 at 0x000b6a60: .got.plt ALLOC LOAD DATA HAS_CONTENTS
0x006b6ae0->0x006b77f0 at 0x000b6ae0: .data ALLOC LOAD DATA HAS_CONTENTS
0x006b7800->0x006beb68 at 0x000b77f0: .bss ALLOC
0x006beb68->0x006beb98 at 0x000b77f0: __libc_freeres_ptrs ALLOC
0x00000000->0x00000038 at 0x000b77f0: .comment READONLY HAS_CONTENTS
0x00000000->0x00000390 at 0x000b7830: .debug_aranges READONLY HAS_CONTENTS
---Type <return> to continue, or q <return> to quit---
0x00000000->0x00000ac3 at 0x000b7bc0: .debug_pubnames READONLY HAS_CONTENTS
0x00000000->0x00011440 at 0x000b8683: .debug_info READONLY HAS_CONTENTS
0x00000000->0x000021b1 at 0x000c9ac3: .debug_abbrev READONLY HAS_CONTENTS
0x00000000->0x00002ebc at 0x000cbc74: .debug_line READONLY HAS_CONTENTS
0x00000000->0x000038da at 0x000ceb30: .debug_str READONLY HAS_CONTENTS
0x00000000->0x0000878e at 0x000d240a: .debug_loc READONLY HAS_CONTENTS
0x00000000->0x00001280 at 0x000dab98: .debug_ranges READONLY HAS_CONTENTS

43
Core file:
`/home/training/ALCDA/./App1/core.3308', file type elf64-x86-64.
0x00000000->0x00002aa8 at 0x00000318: note0 READONLY HAS_CONTENTS
0x00000000->0x000000d8 at 0x00000438: .reg/3309 HAS_CONTENTS
0x00000000->0x000000d8 at 0x00000438: .reg HAS_CONTENTS
0x00000000->0x00000200 at 0x0000052c: .reg2/3309 HAS_CONTENTS
0x00000000->0x00000200 at 0x0000052c: .reg2 HAS_CONTENTS
0x00000000->0x00000340 at 0x00000740: .reg-xstate/3309 HAS_CONTENTS
0x00000000->0x00000340 at 0x00000740: .reg-xstate HAS_CONTENTS
0x00000000->0x000000d8 at 0x00000b04: .reg/3310 HAS_CONTENTS
0x00000000->0x00000200 at 0x00000bf8: .reg2/3310 HAS_CONTENTS
0x00000000->0x00000340 at 0x00000e0c: .reg-xstate/3310 HAS_CONTENTS
0x00000000->0x000000d8 at 0x000011d0: .reg/3311 HAS_CONTENTS
0x00000000->0x00000200 at 0x000012c4: .reg2/3311 HAS_CONTENTS
0x00000000->0x00000340 at 0x000014d8: .reg-xstate/3311 HAS_CONTENTS
0x00000000->0x000000d8 at 0x0000189c: .reg/3312 HAS_CONTENTS
0x00000000->0x00000200 at 0x00001990: .reg2/3312 HAS_CONTENTS
---Type <return> to continue, or q <return> to quit---
0x00000000->0x00000340 at 0x00001ba4: .reg-xstate/3312 HAS_CONTENTS
0x00000000->0x000000d8 at 0x00001f68: .reg/3313 HAS_CONTENTS
0x00000000->0x00000200 at 0x0000205c: .reg2/3313 HAS_CONTENTS
0x00000000->0x00000340 at 0x00002270: .reg-xstate/3313 HAS_CONTENTS
0x00000000->0x000000d8 at 0x00002634: .reg/3308 HAS_CONTENTS
0x00000000->0x00000200 at 0x00002728: .reg2/3308 HAS_CONTENTS
0x00000000->0x00000340 at 0x0000293c: .reg-xstate/3308 HAS_CONTENTS
0x00000000->0x00000130 at 0x00002c90: .auxv HAS_CONTENTS
0x00400000->0x00400000 at 0x00002dc0: load1 ALLOC READONLY CODE
0x006b6000->0x006b8000 at 0x00002dc0: load2 ALLOC LOAD HAS_CONTENTS
0x006b8000->0x006bf000 at 0x00004dc0: load3 ALLOC LOAD HAS_CONTENTS
0x0227c000->0x0229f000 at 0x0000bdc0: load4 ALLOC LOAD HAS_CONTENTS
0x7f2257e67000->0x7f2258667000 at 0x0002edc0: load5 ALLOC LOAD HAS_CONTENTS
0x7f2258668000->0x7f2258e68000 at 0x0082edc0: load6 ALLOC LOAD HAS_CONTENTS
0x7f2258e69000->0x7f2259669000 at 0x0102edc0: load7 ALLOC LOAD HAS_CONTENTS
0x7f225966a000->0x7f2259e6a000 at 0x0182edc0: load8 ALLOC LOAD HAS_CONTENTS
0x7f2259e6b000->0x7f225a66b000 at 0x0202edc0: load9 ALLOC LOAD HAS_CONTENTS
0x7ffc7d24d000->0x7ffc7d26e000 at 0x0282edc0: load10 ALLOC LOAD HAS_CONTENTS
0x7ffc7d299000->0x7ffc7d29a000 at 0x0284fdc0: load11 ALLOC LOAD READONLY CODE HAS_CONTENTS
0xffffffffff600000->0xffffffffff601000 at 0x02850dc0: load12 ALLOC LOAD READONLY CODE HAS_CONTENTS

9. Dump data with possible symbolic information:

(gdb) x/512a 0x006b6000

0x6b6000: 0x0 0xc2740000001c
0x6b6010: 0x50fffd2880 0x80e0a7e100e4400
0x6b6020: 0x80e470b46 0xc29400000014
0x6b6030: 0x8fffd28b0 0x0
0x6b6040: 0xc2ac00000014 0x15fffd28a8
0x6b6050: 0x0 0xc2c400000014
0x6b6060: 0x8fffd28b0 0x0
0x6b6070: 0xc2dc00000014 0x8fffd28a8
0x6b6080: 0x0 0xc2f400000014
0x6b6090: 0x8fffd28a0 0x0
0x6b60a0: 0xc30c0000001c 0x24fffd2898
0x6b60b0: 0x80e0a5a300e4400 0xb42
0x6b60c0: 0xc32c00000014 0x8fffd28a8
0x6b60d0: 0x0 0xc34400000014
0x6b60e0: 0x8fffd28a0 0x0
0x6b60f0: 0xc35c0000002c 0x110fffd2898
0x6b6100: 0xe580283100e4100 0x44100e0ae4020580
0x6b6110: 0x44100e490b41080e 0x80e
0x6b6120: 0xc38c00000014 0x1fffd2978
0x6b6130: 0x0 0xc3a40000003c
0x6b6140: 0x166fffd2970 0xd430286100e4100
0x6b6150: 0x58d048e038f4a06 0x8150078347068c49
44
0x6b6160: 0x70c0a8702098008 0x20cc6a2020b4b08
0x6b6170: 0x8 0xc3e400000034
---Type <return> to continue, or q <return> to quit---
0x6b6180: 0xe6fffd2aa0 0xd430286100e4100
0x6b6190: 0x783088109805006 0x4e048e058d4f068c
0x6b61a0: 0x8070c0a5b02038f 0x8020cc655020b41
0x6b61b0: 0xc41c00000034 0xc1fffd2b58
0x6b61c0: 0xd430286100e4100 0x58d048e038f4a06
0x6b61d0: 0x8153078348068c45 0x20cc68f02098008
0x6b61e0: 0x8 0xc45400000034
0x6b61f0: 0xf1fffd2bf0 0xd430286100e4100
0x6b6200: 0x815e098007834806 0x8f048e058d068c08
0x6b6210: 0xb4508070c0a6103 0x8020cc69d02
0x6b6220: 0xc48c00000014 0x1afffd2cb8
0x6b6230: 0x0 0xc4a40000002c
0x6b6240: 0x99fffd2cc0 0xd430286100e4100
0x6b6250: 0x58d048e038f4606 0x730207834f068c4c
0x6b6260: 0x8070c 0xc4d400000014
0x6b6270: 0x46fffd2d30 0x0
0x6b6280: 0xc4ec00000014 0x1bfffd2d68
0x6b6290: 0x0 0xc5040000004c
0x6b62a0: 0xa3fffd2d70 0xe42028f100e4200
0x6b62b0: 0x48d200e45038e18 0x300e44058c280e45
0x6b62c0: 0x480783380e410686 0x41380e0a5202500e
0x6b62d0: 0x200e42280e41300e 0xe42100e42180e42
0x6b62e0: 0xb4908 0xc55400000044
0x6b62f0: 0xc8fffd2dd0 0xe46028f100e4200
---Type <return> to continue, or q <return> to quit---
0x6b6300: 0x48d200e42038e18 0x300e44058c280e45
0x6b6310: 0x470783380e410686 0xe41380ea202500e
0x6b6320: 0x42200e42280e4130 0x80e42100e42180e
0x6b6330: 0xc59c0000002c 0x67fffd2e58
0x6b6340: 0x80e0a7a100e4400 0xb47080e0a490b42
0x6b6350: 0xe460b47080e0a49 0x8
0x6b6360: 0xc5cc00000024 0x13cfffd2e98
0x6b6370: 0xe4b028c04834a00 0x80e0a7a02038640
0x6b6380: 0xb41 0xc5f400000034
0x6b6390: 0x109fffd2fb0 0xe480286100e4100
0x6b63a0: 0xa68300e44038318 0x80e41100e41180e
0x6b63b0: 0x41180e0a97020b49 0xb47080e41100e
0x6b63c0: 0xc62c00000024 0x6bfffd3088
0x6b63d0: 0x80e0a77100e4400 0xb49080e0a470b45
0x6b63e0: 0xb49080e0a47 0xc6540000004c
0x6b63f0: 0x178fffd30d0 0xe45028f100e4200
0x6b6400: 0x48d200e42038e18 0x300e41058c280e42
0x6b6410: 0x440783380e410686 0x380e0a015103700e
0x6b6420: 0xe42280e41300e41 0x42100e42180e4220
0x6b6430: 0xb41080e 0xc6a40000004c
0x6b6440: 0x157fffd3200 0xe49028f100e4200
0x6b6450: 0x48d200e42038e18 0x300e45058c280e48
0x6b6460: 0x4a0783380e410686 0x41380e012703700e
0x6b6470: 0x200e42280e41300e 0xe42100e42180e42
---Type <return> to continue, or q <return> to quit---
0x6b6480: 0x8 0xc6f400000024
0x6b6490: 0xb0fffd3310 0x8d4d058606834a00
0x6b64a0: 0x48c400e4c028e03 0x80e8c02
0x6b64b0: 0xc71c0000004c 0x194fffd3398
0x6b64c0: 0xe4a028f100e4200 0x48d200e45038e18
0x6b64d0: 0x300e41058c280e45 0x4a0783380e470686
0x6b64e0: 0x380e0a015403700e 0xe42280e41300e44

45
0x6b64f0: 0x42100e42180e4220 0xb47080e
0x6b6500: 0xc76c00000024 0x6bfffd34e8
0x6b6510: 0x80e0a77100e4400 0xb49080e0a470b45
0x6b6520: 0xb49080e0a47 0xc7940000004c
0x6b6530: 0x673fffd3530 0xe42028f100e4200
0x6b6540: 0x48d200e42038e18 0x300e41058c280e42
0x6b6550: 0x470783380e410686 0x380e0a7d0201900e
0x6b6560: 0xe42280e41300e44 0x42100e42180e4220
0x6b6570: 0xb45080e 0xc7e400000024
0x6b6580: 0xcffffd3b60 0x8c4d058606834a00
0x6b6590: 0x28e400e4c038d04 0x80eab02
0x6b65a0: 0xc80c0000004c 0x4b3fffd3c08
0x6b65b0: 0xe42028f100e4200 0x48d200e42038e18
0x6b65c0: 0x300e41058c280e42 0x470783380e410686
0x6b65d0: 0x380e0af20201a00e 0xe42280e41300e43
0x6b65e0: 0x42100e42180e4220 0xb41080e
0x6b65f0: 0xc85c00000014 0x8afffd4078
---Type <return> to continue, or q <return> to quit---
0x6b6600: 0x80e6c200e460200 0xc87400000014
0x6b6610: 0x9fffd40f0 0x0
0x6b6620: 0xc88c0000001c 0x26fffd40e8
0x6b6630: 0xa4a0283100e4100 0x80e510b45080e
0x6b6640: 0xc8ac0000001c 0x72fffd40f8
0x6b6650: 0xa7e0283100e5b00 0x80e4f0b45080e
0x6b6660: 0xc8cc00000014 0x9fffd4158
0x6b6670: 0x0 0xc8e40000001c
0x6b6680: 0x1afffd4150 0xe540283100e4100
0x6b6690: 0x8 0xc9040000003c
0x6b66a0: 0x113fffd4150 0xe44028c100e4200
0x6b66b0: 0x483200e44038618 0x100e41180e0ab902
0x6b66c0: 0xe0a560b4a080e42 0x47080e42100e4118
0x6b66d0: 0xb 0xc94400000014
0x6b66e0: 0x5fffd4230 0x0
0x6b66f0: 0xc95c00000014 0x25fffd4228
0x6b6700: 0x80e49100e5400 0xc97400000044
0x6b6710: 0x1f8fffd4240 0xe42028e100e4200
0x6b6720: 0x48c200e45038d18 0x300e440586280e41
0x6b6730: 0xacb02700e440683 0x200e41280e44300e
0x6b6740: 0xe42100e42180e42 0xb4108
0x6b6750: 0xc9bc0000002c 0x7cfffd43f8
0x6b6760: 0x80e0a76100e4400 0xb49080e0a570b46
0x6b6770: 0xe470b49080e0a47 0x8
---Type <return> to continue, or q <return> to quit---
0x6b6780: 0xc9ec00000024 0x13cfffd4448
0x6b6790: 0x5a020283100e4500 0xedb020b41080e0a
0x6b67a0: 0x8 0xca140000004c
0x6b67b0: 0x242fffd4560 0xe45028e100e6200
0x6b67c0: 0x48c200e45038d18 0x300e410586280e44
0x6b67d0: 0x7e0301800e440683 0x280ec341300e0a01
0x6b67e0: 0x180ecc42200ec641 0x80ece42100ecd42
0x6b67f0: 0xb45 0xca6400000034
0x6b6800: 0x1aafffd4760 0x43180e47100e4200
0x6b6810: 0x43200e42028f038e 0x300e41280e42048d
0x6b6820: 0x4501900e44380e41 0x58c06860783
0x6b6830: 0xca9c0000001c 0x87fffd48d8
0x6b6840: 0x8302864a600e4e00 0x3
0x6b6850: 0xcabc00000014 0x15fffd4948
0x6b6860: 0x0 0x901ffff00000000
0x6b6870: 0x601910070044c 0x5c01a41001ffff00
0x6b6880: 0x3c10502f30000 0x1ffff0000050481

46
0x6b6890: 0x1b10001b603670a 0x961201ffff000046
0x6b68a0: 0x309b6000004eb02 0x1b60a96000b82
0x6b68b0: 0x301b90c01ffff00 0x2ac02830003e5
0x6b68c0: 0x501c61101ffff00 0x8ae068b01fd0000
0x6b68d0: 0xffff00000508b400 0x9500018105660a01
0x6b68e0: 0x801ffff00000501 0x561004d053d
0x6b68f0: 0x1d301c11e01ffff 0xba20503f90000
---Type <return> to continue, or q <return> to quit---
0x6b6900: 0xa406cb0000050684 0x2a50990000b8a02
0x6b6910: 0x5720a01ffff0000 0x502950001d5
0x6b6920: 0x920301990b01ffff 0xff00000502ce0002
0x6b6930: 0x1f705600a01ff 0x1ffff00000502b3
0x6b6940: 0x850002c903028a0b 0xc01ffff00000503
0x6b6950: 0x970004db029601eb 0xa01ffff00000505
0x6b6960: 0x501ef0001b3056b 0x5650a01ffff0000
0x6b6970: 0x501e90001ad 0x1f705600a01ffff
0x6b6980: 0x502b300 0x6bdec0 <_res>
0x6b6990: 0x6b7640 <_nl_global_locale> 0x6b7640 <_nl_global_locale>
0x6b69a0: 0x6b7660 <_nl_global_locale+32> 0x6b7648 <_nl_global_locale+8>
0x6b69b0 <__init_array_start>: 0x4004b0 <frame_dummy> 0x42f4c0 <init_cacheinfo>
0x6b69c0 <__fini_array_start>: 0x400480 <__do_global_dtors_aux> 0x46fcc0 <fini>
0x6b69d0 <__JCR_LIST__>: 0x0 0x0
0x6b69e0 <_dl_argv>: 0x6b72c0 <program_invocation_short_name> 0x7ffc7d26c7e8
0x6b69f0 <_dl_random>: 0x7ffc7d26c9b9 0x0
0x6b6a00 <__stack_prot>: 0x1000000 0x0
0x6b6a10 <env_path_list>: 0xffffffffffffffff 0x0
0x6b6a20 <capstr>: 0x6be130 <result.11783> 0x1
0x6b6a30 <max_capstrlen>: 0x0 0x0
0x6b6a40 <rtld_search_dirs>: 0x227d190 0x0
---Type <return> to continue, or q <return> to quit---
0x6b6a50: 0x403c00 <pthread_cancel> 0x0
0x6b6a60 <_GLOBAL_OFFSET_TABLE_>: 0x0 0x0
0x6b6a70 <_GLOBAL_OFFSET_TABLE_+16>: 0x0 0x41ea40 <__stpcpy_ssse3>
0x6b6a80 <_GLOBAL_OFFSET_TABLE_+32>: 0x41b040 <__strcpy_ssse3> 0x426950 <__memmove_ssse3>
0x6b6a90 <_GLOBAL_OFFSET_TABLE_+48>: 0x423f00 <__rawmemchr_sse42> 0x453760
<__strstr_sse42>
0x6b6aa0 <_GLOBAL_OFFSET_TABLE_+64>: 0x470340 <__strncpy_ssse3> 0x425300 <__memcmp_sse4_1>
0x6b6ab0 <_GLOBAL_OFFSET_TABLE_+80>: 0x421820 <__strcasecmp_l_sse42> 0x41da30
<__memset_sse2>
0x6b6ac0 <_GLOBAL_OFFSET_TABLE_+96>: 0x41a080 <__strcmp_sse42> 0x47f710
<__strncasecmp_l_sse42>
0x6b6ad0 <_GLOBAL_OFFSET_TABLE_+112>: 0x421810 <__strcasecmp_sse42> 0x418b50
<__strchr_sse42>
0x6b6ae0 <data_start>: 0x0 0x0
0x6b6af0 <__nptl_nthreads>: 0x6 0x0
0x6b6b00 <stack_used>: 0x7f22586669c0 0x7f225a66a9c0
0x6b6b10 <stack_cache>: 0x6b6b10 <stack_cache> 0x6b6b10 <stack_cache>
0x6b6b20 <__sched_fifo_min_prio>: 0xffffffffffffffff 0x800000
0x6b6b30 <_dl_tls_static_size>: 0x1160 0x48c997 <_nl_default_default_domain>
0x6b6b40 <locale_alias_path.12333>: 0x48c9c9 0x6bc6e0 <initial>
0x6b6b50: 0x0 0x0
0x6b6b60 <_IO_2_1_stdin_>: 0xfbad2088 0x0
---Type <return> to continue, or q <return> to quit---
0x6b6b70 <_IO_2_1_stdin_+16>: 0x0 0x0
0x6b6b80 <_IO_2_1_stdin_+32>: 0x0 0x0
0x6b6b90 <_IO_2_1_stdin_+48>: 0x0 0x0
0x6b6ba0 <_IO_2_1_stdin_+64>: 0x0 0x0
0x6b6bb0 <_IO_2_1_stdin_+80>: 0x0 0x0
0x6b6bc0 <_IO_2_1_stdin_+96>: 0x0 0x0
0x6b6bd0 <_IO_2_1_stdin_+112>: 0x0 0xffffffffffffffff

47
0x6b6be0 <_IO_2_1_stdin_+128>: 0x0 0x6bcb20 <_IO_stdfile_0_lock>
0x6b6bf0 <_IO_2_1_stdin_+144>: 0xffffffffffffffff 0x0
0x6b6c00 <_IO_2_1_stdin_+160>: 0x6b6e20 <_IO_wide_data_0> 0x0
0x6b6c10 <_IO_2_1_stdin_+176>: 0x0 0x0
0x6b6c20 <_IO_2_1_stdin_+192>: 0x0 0x0
0x6b6c30 <_IO_2_1_stdin_+208>: 0x0 0x48d440 <_IO_file_jumps>
0x6b6c40 <_IO_2_1_stdout_>: 0xfbad2084 0x0
0x6b6c50 <_IO_2_1_stdout_+16>: 0x0 0x0
0x6b6c60 <_IO_2_1_stdout_+32>: 0x0 0x0
0x6b6c70 <_IO_2_1_stdout_+48>: 0x0 0x0
0x6b6c80 <_IO_2_1_stdout_+64>: 0x0 0x0
0x6b6c90 <_IO_2_1_stdout_+80>: 0x0 0x0
0x6b6ca0 <_IO_2_1_stdout_+96>: 0x0 0x6b6b60 <_IO_2_1_stdin_>
0x6b6cb0 <_IO_2_1_stdout_+112>: 0x1 0xffffffffffffffff
0x6b6cc0 <_IO_2_1_stdout_+128>: 0x0 0x6bcb30 <_IO_stdfile_1_lock>
0x6b6cd0 <_IO_2_1_stdout_+144>: 0xffffffffffffffff 0x0
0x6b6ce0 <_IO_2_1_stdout_+160>: 0x6b6f80 <_IO_wide_data_1> 0x0
---Type <return> to continue, or q <return> to quit---
0x6b6cf0 <_IO_2_1_stdout_+176>: 0x0 0x0
0x6b6d00 <_IO_2_1_stdout_+192>: 0x0 0x0
0x6b6d10 <_IO_2_1_stdout_+208>: 0x0 0x48d440 <_IO_file_jumps>
0x6b6d20 <_IO_2_1_stderr_>: 0xfbad2086 0x0
0x6b6d30 <_IO_2_1_stderr_+16>: 0x0 0x0
0x6b6d40 <_IO_2_1_stderr_+32>: 0x0 0x0
0x6b6d50 <_IO_2_1_stderr_+48>: 0x0 0x0
0x6b6d60 <_IO_2_1_stderr_+64>: 0x0 0x0
0x6b6d70 <_IO_2_1_stderr_+80>: 0x0 0x0
0x6b6d80 <_IO_2_1_stderr_+96>: 0x0 0x6b6c40 <_IO_2_1_stdout_>
0x6b6d90 <_IO_2_1_stderr_+112>: 0x2 0xffffffffffffffff
0x6b6da0 <_IO_2_1_stderr_+128>: 0x0 0x6bcb40 <_IO_stdfile_2_lock>
0x6b6db0 <_IO_2_1_stderr_+144>: 0xffffffffffffffff 0x0
0x6b6dc0 <_IO_2_1_stderr_+160>: 0x6b70e0 <_IO_wide_data_2> 0x0
0x6b6dd0 <_IO_2_1_stderr_+176>: 0x0 0x0
0x6b6de0 <_IO_2_1_stderr_+192>: 0x0 0x0
0x6b6df0 <_IO_2_1_stderr_+208>: 0x0 0x48d440 <_IO_file_jumps>
0x6b6e00 <_IO_list_all>: 0x6b6d20 <_IO_2_1_stderr_> 0x0
0x6b6e10: 0x0 0x0
0x6b6e20 <_IO_wide_data_0>: 0x0 0x0
0x6b6e30 <_IO_wide_data_0+16>: 0x0 0x0
0x6b6e40 <_IO_wide_data_0+32>: 0x0 0x0
0x6b6e50 <_IO_wide_data_0+48>: 0x0 0x0
0x6b6e60 <_IO_wide_data_0+64>: 0x0 0x0
---Type <return> to continue, or q <return> to quit---
0x6b6e70 <_IO_wide_data_0+80>: 0x0 0x0
0x6b6e80 <_IO_wide_data_0+96>: 0x0 0x0
0x6b6e90 <_IO_wide_data_0+112>: 0x0 0x0
0x6b6ea0 <_IO_wide_data_0+128>: 0x0 0x0
0x6b6eb0 <_IO_wide_data_0+144>: 0x0 0x0
0x6b6ec0 <_IO_wide_data_0+160>: 0x0 0x0
0x6b6ed0 <_IO_wide_data_0+176>: 0x0 0x0
0x6b6ee0 <_IO_wide_data_0+192>: 0x0 0x0
0x6b6ef0 <_IO_wide_data_0+208>: 0x0 0x0
0x6b6f00 <_IO_wide_data_0+224>: 0x0 0x0
0x6b6f10 <_IO_wide_data_0+240>: 0x0 0x0
0x6b6f20 <_IO_wide_data_0+256>: 0x0 0x0
0x6b6f30 <_IO_wide_data_0+272>: 0x0 0x0
0x6b6f40 <_IO_wide_data_0+288>: 0x0 0x0
0x6b6f50 <_IO_wide_data_0+304>: 0x0 0x0
0x6b6f60 <_IO_wide_data_0+320>: 0x48d1c0 <_IO_wfile_jumps> 0x0
0x6b6f70: 0x0 0x0

48
0x6b6f80 <_IO_wide_data_1>: 0x0 0x0
0x6b6f90 <_IO_wide_data_1+16>: 0x0 0x0
0x6b6fa0 <_IO_wide_data_1+32>: 0x0 0x0
0x6b6fb0 <_IO_wide_data_1+48>: 0x0 0x0
0x6b6fc0 <_IO_wide_data_1+64>: 0x0 0x0
0x6b6fd0 <_IO_wide_data_1+80>: 0x0 0x0
0x6b6fe0 <_IO_wide_data_1+96>: 0x0 0x0
---Type <return> to continue, or q <return> to quit---
0x6b6ff0 <_IO_wide_data_1+112>: 0x0 0x0

The output is in the following format:

address: value1 value2

Because the size of each value is 8 bytes the next address is +16 bytes or +10hex. The addresses can have associated
symbolic names:

address <name>: value1 value2

For example, from the output above:

0x6b6af0 <__nptl_nthreads>: 0x6 0x0

Each value may also have an associated symbolic value:

address <name>: value1 <name1> value2

For example, from the output above:

0x6b69e0 <_dl_argv>: 0x6b72c0 <program_invocation_short_name> 0x7ffc7d26c7e8

10. Explore the contents of memory pointed to by __nptl_nthreads, _dl_argv, program_invocation_short_name

and 0x7ffc7d26c7e8 addresses:

(gdb) x/u 0x6b6af0

0x6b6af0 <__nptl_nthreads>: 6

(gdb) x/u &__nptl_nthreads

0x6b6af0 <__nptl_nthreads>: 6

(gdb) x/2a 0x6b69e0

0x6b69e0 <_dl_argv>: 0x6b72c0 <program_invocation_short_name> 0x7ffc7d26c7e8

(gdb) x/2a &_dl_argv

0x6b69e0 <_dl_argv>: 0x6b72c0 <program_invocation_short_name> 0x7ffc7d26c7e8

(gdb) x/a 0x6b72c0

0x6b72c0 <program_invocation_short_name>: 0x7ffc7d26d9a9

(gdb) x/a &program_invocation_short_name

0x6b72c0 <program_invocation_short_name>: 0x7ffc7d26d9a9

(gdb) x/s 0x7ffc7d26d9a9

0x7ffc7d26d9a9: "App1"

49
(gdb) x/10a 0x7ffc7d26c7e8
0x7ffc7d26c7e8: 0x0 0x1
0x7ffc7d26c7f8: 0x7ffc7d26d9a7 0x0
0x7ffc7d26c808: 0x7ffc7d26d9ae 0x7ffc7d26d9be
0x7ffc7d26c818: 0x7ffc7d26d9c9 0x7ffc7d26d9d9
0x7ffc7d26c828: 0x7ffc7d26d9e7 0x7ffc7d26df08

(gdb) x/10c 0x7ffc7d26d9a7

0x7ffc7d26d9a7: 46 '.' 47 '/' 65 'A' 112 'p' 112 'p' 49 '1' 0 '\000' 83 'S'
0x7ffc7d26d9af: 72 'H' 69 'E'

(gdb) x/s 0x7ffc7d26d9a7

0x7ffc7d26d9a7: "./App1"

(gdb) x/5s 0x7ffc7d26d9a7

0x7ffc7d26d9a7: "./App1"
0x7ffc7d26d9ae: "SHELL=/bin/bash"
0x7ffc7d26d9be: "TERM=linux"
0x7ffc7d26d9c9: "HUSHLOGIN=FALSE"
0x7ffc7d26d9d9: "USER=training"

11. Explore the contents of memory pointed to by environ variable address:

(gdb) x/a &environ

0x6bd4c8 <environ>: 0x7ffc7d26c808

(gdb) x/10a 0x7ffc7d26c808

0x7ffc7d26c808: 0x7ffc7d26d9ae 0x7ffc7d26d9be
0x7ffc7d26c818: 0x7ffc7d26d9c9 0x7ffc7d26d9d9
0x7ffc7d26c828: 0x7ffc7d26d9e7 0x7ffc7d26df08
0x7ffc7d26c838: 0x7ffc7d26df20 0x7ffc7d26df5e
0x7ffc7d26c848: 0x7ffc7d26df7c 0x7ffc7d26df8d

(gdb) x/4s 0x7ffc7d26d9ae

0x7ffc7d26d9ae: "SHELL=/bin/bash"
0x7ffc7d26d9be: "TERM=linux"
0x7ffc7d26d9c9: "HUSHLOGIN=FALSE"
0x7ffc7d26d9d9: "USER=training"

12. Get the list of loaded modules:

(gdb) info sharedlibrary

No shared libraries loaded at this time.

We don’t see any shared libraries because they were statically linked. We also created the version of a dynamically
linked App1.shared executable. If we load its core dump we see the list of shared libraries:

training@debian64:~/ALCDA$ gdb -c ./App1/core.5476 -se ./App1/App1.shared

GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/training/ALCDA/App1/App1.shared...(no debugging symbols
found)...done.
[New LWP 5477]
50
[New LWP 5478]
[New LWP 5479]
[New LWP 5480]
[New LWP 5481]
[New LWP 5476]

warning: Can't read pathname for load map: Input/output error.

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/home/training/ALCDA/App1/App1.shared'.
#0 0x00007f25a013e48d in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6

(gdb) info sharedlibrary

From To Syms Read Shared Object Library
0x00007f25a0423690 0x00007f25a042ece8 Yes (*) /lib/x86_64-linux-gnu/libpthread.so.0
0x00007f25a00b1b80 0x00007f25a01c9c2c Yes (*) /lib/x86_64-linux-gnu/libc.so.6
0x00007f25a063aaf0 0x00007f25a0652c83 Yes (*) /lib64/ld-linux-x86-64.so.2
(*): Shared library is missing debugging information.

13. Disassemble bar_two function and follow the indirect sleep function call:

(gdb) disassemble bar_two

Dump of assembler code for function bar_two:
0x00000000004005f9 <+0>: push %rbp
0x00000000004005fa <+1>: mov %rsp,%rbp
0x00000000004005fd <+4>: mov $0xffffffff,%edi
0x0000000000400602 <+9>: callq 0x4004a0 <sleep@plt>
0x0000000000400607 <+14>: pop %rbp
0x0000000000400608 <+15>: retq
End of assembler dump.

(gdb) disassemble 0x4004a0

Dump of assembler code for function sleep@plt:
0x00000000004004a0 <+0>: jmpq *0x20090a(%rip) # 0x600db0 <[email protected]>
0x00000000004004a6 <+6>: pushq $0x2
0x00000000004004ab <+11>: jmpq 0x400470
End of assembler dump.

14. Dump the annotated value as a memory address interpreting its contents as a symbol:

(gdb) x/a 0x600db0

0x600db0 <[email protected]>: 0x7f25a013e220 <sleep>

51
Published by OpenTask, Republic of Ireland

Copyright © 2018 by OpenTask

Copyright © 2018 by Software Diagnostics Services

Copyright © 2018 by Dmitry Vostokov

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or
transmitted, in any form or by any means, without the prior written permission of the
publisher.

Product and company names mentioned in this book may be trademarks of their owners.

OpenTask books and magazines are available through booksellers and distributors worldwide.
For further information or comments send requests to [email protected].

A CIP catalog record for this book is available from the British Library.

ISBN-l3: 978-1-908043-89-4 (Paperback)

Revision 2.00 (September 2018)

2
Contents

About the Author.............................................................................................................................................................. 5

Presentation Slides and Transcript ................................................................................................................................... 7
Practice Exercises ........................................................................................................................................................... 35
Exercise 0 .................................................................................................................................................................... 41
Exercise D1 ................................................................................................................................................................. 50
Exercise D2 ................................................................................................................................................................. 70
Exercise D3 ................................................................................................................................................................. 83
Exercise D4 ............................................................................................................................................................... 108
Exercise D5 ............................................................................................................................................................... 115
Exercise D6 ............................................................................................................................................................... 134
Exercise D7 ............................................................................................................................................................... 143
Exercise D8 ............................................................................................................................................................... 151
Exercise K0 ................................................................................................................................................................ 163
Exercise KD6 ............................................................................................................................................................. 180
Exercise KD9 ............................................................................................................................................................. 213
Exercise KD10 ........................................................................................................................................................... 232
Exercise MD11 .......................................................................................................................................................... 255
Appendix ....................................................................................................................................................................... 299
Complete Stack Traces from x64 System ................................................................................................................. 301

3
 Exercise D1

Goal: Learn how code generation parameters can influence process execution behavior.

Elementary Diagnostics Patterns: Crash.

Memory Analysis Patterns: Exception Stack Trace.

Debugging Implementation Patterns: Scope, Variable Value, Type Structure, Code Breakpoint.

1. Launch WinDbg from Windows Kits \ WinDbg (X64).

2. Open \AWD3\AppD1A\x64\Release\AppD1A.exe executable:

50
3. You get the executable file loaded and ready for a debugging session:

4. Open a log file:

0:000> .logopen C:\AWD3\D1A.log

Opened log file 'C:\AWD3\D1A.log'

5. Set up a link to Microsoft symbol server and reload symbols:

0:000> .symfix c:\mss

0:000> .reload
Reloading current modules
...........

51
6. lm command lists module information:

0:000> lm
start end module name
00007ff6`01800000 00007ff6`0181b000 AppD1A (deferred)
00007ffd`0a1f0000 00007ffd`0a27b000 apphelp (deferred)
00007ffd`0c770000 00007ffd`0c790000 win32u (deferred)
00007ffd`0c7e0000 00007ffd`0c8da000 ucrtbase (deferred)
00007ffd`0cbb0000 00007ffd`0cc4f000 msvcp_win (deferred)
00007ffd`0cc50000 00007ffd`0cec3000 KERNELBASE (deferred)
00007ffd`0d690000 00007ffd`0d822000 gdi32full (deferred)
00007ffd`0d900000 00007ffd`0d9b2000 KERNEL32 (deferred)
00007ffd`0d9c0000 00007ffd`0db50000 USER32 (deferred)
00007ffd`0e9e0000 00007ffd`0ea08000 GDI32 (deferred)
00007ffd`103a0000 00007ffd`10581000 ntdll (pdb symbols)
c:\mss\ntdll.pdb\EA3C05F9EA540B02C1971816AF7CC8D21\ntdll.pdb

7. We continue process execution using g command and ignore any first chance exceptions until we come to a
second chance exception:

0:000> g
ModLoad: 00007ffd`101c0000 00007ffd`101ed000 C:\WINDOWS\System32\IMM32.DLL
(4f80.707c): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
USER32!StringDuplicateW+0x20:
00007ffd`0d9c5cbc 66392c41 cmp word ptr [rcx+rax*2],bp ds:01816bb0`00000000=????

0:000> g
(4f80.707c): Access violation - code c0000005 (!!! second chance !!!)
USER32!StringDuplicateW+0x20:
00007ffd`0d9c5cbc 66392c41 cmp word ptr [rcx+rax*2],bp ds:01816bb0`00000000=????

8. We see that a crash happened in USER32 module with the following CPU state:

0:000> r
rax=0000000000000000 rbx=0000005d794ff9d0 rcx=01816bb000000000
rdx=01816bb000000000 rsi=0000005d794ff960 rdi=01816bb000000000
rip=00007ffd0d9c5cbc rsp=0000005d794ff860 rbp=0000000000000000
r8=0000005d794ff9d0 r9=0000000000000000 r10=0000019011140000
r11=0000000000000000 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl zr ac po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010254
USER32!StringDuplicateW+0x20:
00007ffd`0d9c5cbc 66392c41 cmp word ptr [rcx+rax*2],bp ds:01816bb0`00000000=????

52
9. The default analysis command also gives us a source code:

0:000> !analyze -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************

*** WARNING: Unable to verify checksum for AppD1A.exe

KEY_VALUES_STRING: 1

TIMELINE_ANALYSIS: 1

Timeline: !analyze.Start
Name: <blank>
Time: 2018-09-12T11:47:03.53Z
Diff: 946 mSec

Timeline: Dump.Current
Name: <blank>
Time: 2018-09-12T11:47:04.0Z
Diff: 0 mSec

Timeline: Process.Start
Name: <blank>
Time: 2018-09-12T11:27:00.0Z
Diff: 1204000 mSec

Timeline: OS.Boot
Name: <blank>
Time: 2018-09-06T17:44:34.0Z
Diff: 496950000 mSec

DUMP_CLASS: 2

DUMP_QUALIFIER: 0

FAULTING_IP:
USER32!StringDuplicateW+20
00007ffd`0d9c5cbc 66392c41 cmp word ptr [rcx+rax*2],bp

EXCEPTION_RECORD: (.exr -1)

ExceptionAddress: 00007ffd0d9c5cbc (USER32!StringDuplicateW+0x0000000000000020)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000000
Parameter[1]: ffffffffffffffff
Attempt to read from address ffffffffffffffff

FAULTING_THREAD: 0000707c

DEFAULT_BUCKET_ID: INVALID_POINTER_READ

PROCESS_NAME: AppD1A.exe

53
FOLLOWUP_IP:
AppD1A!MyRegisterClass+8d [c:\awd3\appd1a\appd1a\appd1a.cpp @ 84]
00007ff6`0180116d 4883c478 add rsp,78h

READ_ADDRESS: ffffffffffffffff

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The
memory could not be %s.

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The
memory could not be %s.

EXCEPTION_CODE_STR: c0000005

EXCEPTION_PARAMETER1: 0000000000000000

EXCEPTION_PARAMETER2: ffffffffffffffff

WATSON_BKT_PROCSTAMP: 5b94d979

WATSON_BKT_MODULE: USER32.dll

WATSON_BKT_MODSTAMP: fd9a9c22

WATSON_BKT_MODOFFSET: 5cbc

WATSON_BKT_MODVER: 10.0.17134.1

MODULE_VER_PRODUCT: Microsoft® Windows® Operating System

BUILD_VERSION_STRING: 17134.1.amd64fre.rs4_release.180410-1804

MODLIST_WITH_TSCHKSUM_HASH: c517e1747eba893f351ec565e72502936e283027

MODLIST_SHA1_HASH: f6d6417e5a956d590c2325ca86fc187e87a812ad

NTGLOBALFLAG: 70

PROCESS_BAM_CURRENT_THROTTLED: 0

PROCESS_BAM_PREVIOUS_THROTTLED: 0

APPLICATION_VERIFIER_FLAGS: 0

PRODUCT_TYPE: 1

SUITE_MASK: 272

DUMP_TYPE: fe

ANALYSIS_SESSION_HOST: DESKTOP-IS6V2L0

ANALYSIS_SESSION_TIME: 09-12-2018 12:47:03.0053

ANALYSIS_VERSION: 10.0.17134.12 amd64fre

THREAD_ATTRIBUTES:
OS_LOCALE: ENG

PROBLEM_CLASSES:

54
 ID: [0n309]
Type: [@ACCESS_VIOLATION]
Class: Addendum
Scope: BUCKET_ID
Name: Omit
Data: Omit
PID: [Unspecified]
TID: [0x707c]
Frame: [0] : USER32!StringDuplicateW

ID: [0n281]
Type: [INVALID_POINTER_READ]
Class: Primary
Scope: DEFAULT_BUCKET_ID (Failure Bucket ID prefix)
BUCKET_ID
Name: Add
Data: Omit
PID: [Unspecified]
TID: [0x707c]
Frame: [0] : USER32!StringDuplicateW

BUGCHECK_STR: APPLICATION_FAULT_INVALID_POINTER_READ

PRIMARY_PROBLEM_CLASS: APPLICATION_FAULT

LAST_CONTROL_TRANSFER: from 00007ffd0d9c5475 to 00007ffd0d9c5cbc

STACK_TEXT:
0000005d`794ff860 00007ffd`0d9c5475 : 0000005d`794ff9d0 01816bb0`00000000 0000005d`794ff960
00007ff6`01800000 : USER32!StringDuplicateW+0x20
0000005d`794ff890 00007ffd`0d9c4c52 : 0000005d`794ffc70 0000005d`794ff9e0 00000000`00000000
00007ffd`0d9c4e40 : USER32!InitClsMenuNameW+0x75
0000005d`794ff8e0 00007ffd`0d9c46ff : 00000000`00000006 00000000`00000000 00000000`00000000
00000000`00000000 : USER32!RegisterClassExWOWW+0x116
0000005d`794ffc40 00007ff6`0180116d : 00000000`00000000 00000000`00000000 00000000`00000000
00000000`00000000 : USER32!RegisterClassW+0x6f
0000005d`794ffcd0 00007ff6`0180105c : 00007ff6`01800000 00000000`00000000 00000000`00000000
00000000`00000000 : AppD1A!MyRegisterClass+0x8d
0000005d`794ffd50 00007ff6`0180166e : 00007ff6`01800000 00000000`00000000 00000190`10d72aee
00000000`0000000a : AppD1A!wWinMain+0x5c
0000005d`794ffdb0 00007ffd`0d913034 : 00000000`00000000 00000000`00000000 00000000`00000000
00000000`00000000 : AppD1A!__scrt_common_main_seh+0x106
0000005d`794ffdf0 00007ffd`10411431 : 00000000`00000000 00000000`00000000 00000000`00000000
00000000`00000000 : KERNEL32!BaseThreadInitThunk+0x14
0000005d`794ffe20 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000
00000000`00000000 : ntdll!RtlUserThreadStart+0x21

STACK_COMMAND: ~0s ; .cxr ; kb

THREAD_SHA1_HASH_MOD_FUNC: a981f01cd8fc185e8c4ffb6f2411e0ae6f8e3a0e

THREAD_SHA1_HASH_MOD_FUNC_OFFSET: ee1d72c7551cebfa6cb33a5bba1435f33ab539d3

THREAD_SHA1_HASH_MOD: 363898a2e705fbd38e6a7fe68b9fe8bfa6feab5a

FAULT_INSTR_CODE: 78c48348

FAULTING_SOURCE_LINE: c:\awd3\appd1a\appd1a\appd1a.cpp

55
FAULTING_SOURCE_FILE: c:\awd3\appd1a\appd1a\appd1a.cpp

FAULTING_SOURCE_LINE_NUMBER: 84

FAULTING_SOURCE_CODE:
80: wc.lpszMenuName = MAKEINTRESOURCE(IDC_APPD1A);
81: wc.lpszClassName = szWindowClass;
82:
83: return RegisterClass(&wc);
> 84: }
85:
86: //
87: // FUNCTION: InitInstance(HINSTANCE, int)
88: //
89: // PURPOSE: Saves instance handle and creates main window

SYMBOL_STACK_INDEX: 4

SYMBOL_NAME: appd1a!MyRegisterClass+8d

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: AppD1A

IMAGE_NAME: AppD1A.exe

DEBUG_FLR_IMAGE_TIMESTAMP: 5b94d979

FAILURE_BUCKET_ID: INVALID_POINTER_READ_c0000005_AppD1A.exe!MyRegisterClass

BUCKET_ID: APPLICATION_FAULT_INVALID_POINTER_READ_appd1a!MyRegisterClass+8d

FAILURE_EXCEPTION_CODE: c0000005

FAILURE_IMAGE_NAME: AppD1A.exe

BUCKET_ID_IMAGE_STR: AppD1A.exe

FAILURE_MODULE_NAME: AppD1A

BUCKET_ID_MODULE_STR: AppD1A

FAILURE_FUNCTION_NAME: MyRegisterClass

BUCKET_ID_FUNCTION_STR: MyRegisterClass

BUCKET_ID_OFFSET: 8d

BUCKET_ID_MODTIMEDATESTAMP: 5b94d979

BUCKET_ID_MODCHECKSUM: 0

BUCKET_ID_MODVER_STR: 0.0.0.0

BUCKET_ID_PREFIX_STR: APPLICATION_FAULT_INVALID_POINTER_READ_

FAILURE_PROBLEM_CLASS: APPLICATION_FAULT

56
FAILURE_SYMBOL_NAME: AppD1A.exe!MyRegisterClass

TARGET_TIME: 2018-09-12T11:47:13.000Z

OSBUILD: 17134

OSSERVICEPACK: 1

SERVICEPACK_NUMBER: 0

OS_REVISION: 0

OSPLATFORM_TYPE: x64

OSNAME: Windows 10

OSEDITION: Windows 10 WinNt SingleUserTS

USER_LCID: 0

OSBUILD_TIMESTAMP: 2020-08-28 05:38:41

BUILDDATESTAMP_STR: 180410-1804

BUILDLAB_STR: rs4_release

BUILDOSVER_STR: 10.0.17134.1.amd64fre.rs4_release.180410-1804

ANALYSIS_SESSION_ELAPSED_TIME: 29fd

ANALYSIS_SOURCE: UM

FAILURE_ID_HASH_STRING: um:invalid_pointer_read_c0000005_appd1a.exe!myregisterclass

FAILURE_ID_HASH: {0e59b433-475d-53b5-9229-de642189649b}

Followup: MachineOwner
---------

10. We get a stack trace with frame numbers using kn command (k command also shows them by default):

0:000> kn
# Child-SP RetAddr Call Site
00 0000005d`794ff860 00007ffd`0d9c5475 USER32!StringDuplicateW+0x20
01 0000005d`794ff890 00007ffd`0d9c4c52 USER32!InitClsMenuNameW+0x75
02 0000005d`794ff8e0 00007ffd`0d9c46ff USER32!RegisterClassExWOWW+0x116
03 0000005d`794ffc40 00007ff6`0180116d USER32!RegisterClassW+0x6f
04 0000005d`794ffcd0 00007ff6`0180105c AppD1A!MyRegisterClass+0x8d
[c:\awd3\appd1a\appd1a\appd1a.cpp @ 84]
05 0000005d`794ffd50 00007ff6`0180166e AppD1A!wWinMain+0x5c [c:\awd3\appd1a\appd1a\appd1a.cpp @
41]
06 (Inline Function) --------`-------- AppD1A!invoke_main+0x21
[f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 118]
07 0000005d`794ffdb0 00007ffd`0d913034 AppD1A!__scrt_common_main_seh+0x106
[f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 288]
08 0000005d`794ffdf0 00007ffd`10411431 KERNEL32!BaseThreadInitThunk+0x14
09 0000005d`794ffe20 00000000`00000000 ntdll!RtlUserThreadStart+0x21

57
11. Now we can set the frame we want to investigate (from where RegisterClassW was called):

0:000> .frame 4
04 0000005d`794ffcd0 00007ff6`0180105c AppD1A!MyRegisterClass+0x8d
[c:\awd3\appd1a\appd1a\appd1a.cpp @ 84]

Note: You see a source code window immediately to the left of the command window:

58
12. Go to View \ Options menu and check that “Evaluate on hover” is checked:

59
13. If we select the source code window and hover a mouse pointer over wc variable we get structure variables:

We can also dump this variable using type information:

0:000> dt wc
Local var @ 0x5d794ffcf0 Type tagWNDCLASSW
+0x000 style : 3
+0x004 lpfnWndProc : 0x00007ff6`01801240 int64 AppD1A!WndProc+0
+0x00c cbClsExtra : 0n0
+0x010 cbWndExtra : 0n0
+0x014 hInstance : 0x00007ff6`01800000 HINSTANCE__
+0x01c hIcon : 0x00000000`01730ecf HICON__
+0x024 hCursor : 0x00000000`00010003 HICON__
+0x02c hbrBackground : 0x00000000`00000006 HBRUSH__
+0x034 lpszMenuName : 0x00000000`0000006d "--- memory read error at address
0x00000000`0000006d ---"
+0x03c lpszClassName : 0x00007ff6`01816bb0 "APPD1A"

14. We can also list all other local variables and parameters for the current frame:

0:000> dv /i /V
prv param 0000005d`794ffd50 @rsp+0x0080 hInstance = 0x00007ff6`01800000
prv local 0000005d`794ffcf0 @rsp+0x0020 wc = struct tagWNDCLASSW

Note: Since all structure members seem to be valid let’s compare it with another application that doesn’t crash.

60
15. Launch another instance of WinDbg from Windows Kits \ WinDbg (X64) and open
\AWD3\AppD1B\x64\Release\AppD1B.exe executable. We get the following output:

Microsoft (R) Windows Debugger Version 10.0.17134.12 AMD64

Copyright (c) Microsoft Corporation. All rights reserved.

CommandLine: C:\AWD3\AppD1B\x64\Release\AppD1B.exe
Symbol search path is: srv*
Executable search path is:
ModLoad: 00007ff6`9d280000 00007ff6`9d29b000 AppD1B.exe
ModLoad: 00007ffd`103a0000 00007ffd`10581000 ntdll.dll
ModLoad: 00007ffd`0d900000 00007ffd`0d9b2000 C:\WINDOWS\System32\KERNEL32.DLL
ModLoad: 00007ffd`0cc50000 00007ffd`0cec3000 C:\WINDOWS\System32\KERNELBASE.dll
ModLoad: 00007ffd`0a1f0000 00007ffd`0a27b000 C:\WINDOWS\SYSTEM32\apphelp.dll
ModLoad: 00007ffd`0d9c0000 00007ffd`0db50000 C:\WINDOWS\System32\USER32.dll
ModLoad: 00007ffd`0c770000 00007ffd`0c790000 C:\WINDOWS\System32\win32u.dll
ModLoad: 00007ffd`0e9e0000 00007ffd`0ea08000 C:\WINDOWS\System32\GDI32.dll
ModLoad: 00007ffd`0d690000 00007ffd`0d822000 C:\WINDOWS\System32\gdi32full.dll
ModLoad: 00007ffd`0cbb0000 00007ffd`0cc4f000 C:\WINDOWS\System32\msvcp_win.dll
ModLoad: 00007ffd`0c7e0000 00007ffd`0c8da000 C:\WINDOWS\System32\ucrtbase.dll
(8c34.8834): Break instruction exception - code 80000003 (first chance)
ntdll!LdrpDoDebuggerBreak+0x30:
00007ffd`1046cd9c cc int 3

16. We open a new log file, fix and reload symbols:

0:000> .logopen C:\AWD3\D1B.log

Opened log file 'C:\AWD3\D1B.log'

0:000> .symfix c:\mss

0:000> .reload
Reloading current modules
...........

61
17. If we run it via g command, we don’t get any exceptions:

62
18. So we choose Debug \ Break menu option and then Debug \ Restart. We get the following output:

0:000> g
ModLoad: 00007ffd`101c0000 00007ffd`101ed000 C:\WINDOWS\System32\IMM32.DLL
ModLoad: 00007ffd`0a390000 00007ffd`0a428000 C:\WINDOWS\system32\uxtheme.dll
ModLoad: 00007ffd`0ea30000 00007ffd`0eace000 C:\WINDOWS\System32\msvcrt.dll
ModLoad: 00007ffd`0dd60000 00007ffd`0e083000 C:\WINDOWS\System32\combase.dll
ModLoad: 00007ffd`0e680000 00007ffd`0e7a4000 C:\WINDOWS\System32\RPCRT4.dll
ModLoad: 00007ffd`0cad0000 00007ffd`0cb4a000 C:\WINDOWS\System32\bcryptPrimitives.dll
ModLoad: 00007ffd`101f0000 00007ffd`10365000 C:\WINDOWS\System32\MSCTF.dll
ModLoad: 00007ffd`0dd00000 00007ffd`0dd5b000 C:\WINDOWS\System32\sechost.dll
ModLoad: 00007ffd`0d830000 00007ffd`0d8f2000 C:\WINDOWS\System32\OLEAUT32.dll
ModLoad: 00007ffd`0aa90000 00007ffd`0aab9000 C:\WINDOWS\system32\dwmapi.dll
ModLoad: 00007ffd`0c6d0000 00007ffd`0c6e1000 C:\WINDOWS\System32\kernel.appcore.dll
ModLoad: 00007ffd`00880000 00007ffd`008eb000 C:\WINDOWS\system32\Oleacc.dll
(8c34.6b98): Break instruction exception - code 80000003 (first chance)
ntdll!DbgBreakPoint:
00007ffd`1043d880 cc int 3

0:001> .restart /f
NatVis script unloaded from 'C:\Program Files (x86)\Windows
Kits\10\Debuggers\x64\Visualizers\atlmfc.natvis'
NatVis script unloaded from 'C:\Program Files (x86)\Windows
Kits\10\Debuggers\x64\Visualizers\concurrency.natvis'
NatVis script unloaded from 'C:\Program Files (x86)\Windows
Kits\10\Debuggers\x64\Visualizers\cpp_rest.natvis'
NatVis script unloaded from 'C:\Program Files (x86)\Windows
Kits\10\Debuggers\x64\Visualizers\stl.natvis'
NatVis script unloaded from 'C:\Program Files (x86)\Windows
Kits\10\Debuggers\x64\Visualizers\Windows.Data.Json.natvis'
NatVis script unloaded from 'C:\Program Files (x86)\Windows
Kits\10\Debuggers\x64\Visualizers\Windows.Devices.Geolocation.natvis'
NatVis script unloaded from 'C:\Program Files (x86)\Windows
Kits\10\Debuggers\x64\Visualizers\Windows.Devices.Sensors.natvis'
NatVis script unloaded from 'C:\Program Files (x86)\Windows
Kits\10\Debuggers\x64\Visualizers\Windows.Media.natvis'
NatVis script unloaded from 'C:\Program Files (x86)\Windows
Kits\10\Debuggers\x64\Visualizers\windows.natvis'
NatVis script unloaded from 'C:\Program Files (x86)\Windows
Kits\10\Debuggers\x64\Visualizers\winrt.natvis'
CommandLine: C:\AWD3\AppD1B\x64\Release\AppD1B.exe

************* Path validation summary **************

Response Time (ms) Location
Deferred srv*

************* Path validation summary **************

Response Time (ms) Location
Deferred srv*
Symbol search path is: srv*
Executable search path is: srv*
ModLoad: 00007ff6`9d280000 00007ff6`9d29b000 AppD1B.exe
ModLoad: 00007ffd`103a0000 00007ffd`10581000 ntdll.dll
ModLoad: 00007ffd`0d900000 00007ffd`0d9b2000 C:\WINDOWS\System32\KERNEL32.DLL
ModLoad: 00007ffd`0cc50000 00007ffd`0cec3000 C:\WINDOWS\System32\KERNELBASE.dll
ModLoad: 00007ffd`0d9c0000 00007ffd`0db50000 C:\WINDOWS\System32\USER32.dll
ModLoad: 00007ffd`0c770000 00007ffd`0c790000 C:\WINDOWS\System32\win32u.dll
ModLoad: 00007ffd`0e9e0000 00007ffd`0ea08000 C:\WINDOWS\System32\GDI32.dll
ModLoad: 00007ffd`0d690000 00007ffd`0d822000 C:\WINDOWS\System32\gdi32full.dll
ModLoad: 00007ffd`0cbb0000 00007ffd`0cc4f000 C:\WINDOWS\System32\msvcp_win.dll

63
ModLoad: 00007ffd`0c7e0000 00007ffd`0c8da000 C:\WINDOWS\System32\ucrtbase.dll
(7628.9044): Break instruction exception - code 80000003 (first chance)
ntdll!LdrpDoDebuggerBreak+0x30:
00007ffd`1046cd9c cc int 3

19. Since we want to compare the same behavior of RegisterClassW function we need to put a breakpoint to
break in when this function is about to be executed. Then we would see WNDCLASS structure passed to it. We set a
pattern matching breakpoint using bm command:

0:000> bm *!RegisterClassW
*** WARNING: Unable to verify checksum for AppD1B.exe
1: 00007ffd`0cd40330 @!"KERNELBASE!RegisterClassW"
2: 00007ffd`0d9c4690 @!"USER32!RegisterClassW"

20. Indeed we a hit immediately:

0:000> g
ModLoad: 00007ffd`101c0000 00007ffd`101ed000 C:\WINDOWS\System32\IMM32.DLL
Breakpoint 2 hit
USER32!RegisterClassW:
00007ffd`0d9c4690 4053 push rbx

We get an identical stack trace prior to RegisterClassW when we compare with the previously running instance of
AppD1A.exe:

0:000> k ; AppD1B
# Child-SP RetAddr Call Site
00 00000075`4e9bf808 00007ff6`9d28116d USER32!RegisterClassW
01 00000075`4e9bf810 00007ff6`9d28105c AppD1B!MyRegisterClass+0x8d
[c:\awd3\appd1b\appd1b\appd1b.cpp @ 84]
02 00000075`4e9bf890 00007ff6`9d28166e AppD1B!wWinMain+0x5c [c:\awd3\appd1b\appd1b\appd1b.cpp @
41]
03 (Inline Function) --------`-------- AppD1B!invoke_main+0x21
[f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 118]
04 00000075`4e9bf900 00007ffd`0d913034 AppD1B!__scrt_common_main_seh+0x106
[f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 288]
05 00000075`4e9bf940 00007ffd`10411431 KERNEL32!BaseThreadInitThunk+0x14
06 00000075`4e9bf970 00000000`00000000 ntdll!RtlUserThreadStart+0x21

0:000> k ; AppD1A
# Child-SP RetAddr Call Site
00 0000005d`794ff860 00007ffd`0d9c5475 USER32!StringDuplicateW+0x20
01 0000005d`794ff890 00007ffd`0d9c4c52 USER32!InitClsMenuNameW+0x75
02 0000005d`794ff8e0 00007ffd`0d9c46ff USER32!RegisterClassExWOWW+0x116
03 0000005d`794ffc40 00007ff6`0180116d USER32!RegisterClassW+0x6f
04 0000005d`794ffcd0 00007ff6`0180105c AppD1A!MyRegisterClass+0x8d
[c:\awd3\appd1a\appd1a\appd1a.cpp @ 84]
05 0000005d`794ffd50 00007ff6`0180166e AppD1A!wWinMain+0x5c [c:\awd3\appd1a\appd1a\appd1a.cpp @
41]
06 (Inline Function) --------`-------- AppD1A!invoke_main+0x21
[f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 118]
07 0000005d`794ffdb0 00007ffd`0d913034 AppD1A!__scrt_common_main_seh+0x106
[f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 288]
08 0000005d`794ffdf0 00007ffd`10411431 KERNEL32!BaseThreadInitThunk+0x14
09 0000005d`794ffe20 00000000`00000000 ntdll!RtlUserThreadStart+0x21

64
21. We choose frame 1 which called RegisterClassW and immediately get access to wc variable (we also note
that function MyRegisterClass source code is identical to AppD1A):

0:000> kn
# Child-SP RetAddr Call Site
00 00000075`4e9bf808 00007ff6`9d28116d USER32!RegisterClassW
01 00000075`4e9bf810 00007ff6`9d28105c AppD1B!MyRegisterClass+0x8d
[c:\awd3\appd1b\appd1b\appd1b.cpp @ 84]
02 00000075`4e9bf890 00007ff6`9d28166e AppD1B!wWinMain+0x5c [c:\awd3\appd1b\appd1b\appd1b.cpp @
41]
03 (Inline Function) --------`-------- AppD1B!invoke_main+0x21
[f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 118]
04 00000075`4e9bf900 00007ffd`0d913034 AppD1B!__scrt_common_main_seh+0x106
[f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 288]
05 00000075`4e9bf940 00007ffd`10411431 KERNEL32!BaseThreadInitThunk+0x14
06 00000075`4e9bf970 00000000`00000000 ntdll!RtlUserThreadStart+0x21

0:000> .frame 1
01 00000075`4e9bf810 00007ff6`9d28105c AppD1B!MyRegisterClass+0x8d
[c:\awd3\appd1b\appd1b\appd1b.cpp @ 84]

0:000> dt wc ; AppD1B
Local var @ 0x754e9bf830 Type tagWNDCLASSW
+0x000 style : 3
+0x008 lpfnWndProc : 0x00007ff6`9d281240 int64 AppD1B!WndProc+0
+0x010 cbClsExtra : 0n0
+0x014 cbWndExtra : 0n0
+0x018 hInstance : 0x00007ff6`9d280000 HINSTANCE__
+0x020 hIcon : 0x00000000`04602229 HICON__
+0x028 hCursor : 0x00000000`00010003 HICON__
+0x030 hbrBackground : 0x00000000`00000006 HBRUSH__
+0x038 lpszMenuName : 0x00000000`0000006d "--- memory read error at address
0x00000000`0000006d ---"
+0x040 lpszClassName : 0x00007ff6`9d296bb0 "APPD1B"

22. But if we look at AppD1A structure variant we see its members have different offsets:

0:000> dt wc ; AppD1A
Local var @ 0x5d794ffcf0 Type tagWNDCLASSW
+0x000 style : 3
+0x004 lpfnWndProc : 0x00007ff6`01801240 int64 AppD1A!WndProc+0
+0x00c cbClsExtra : 0n0
+0x010 cbWndExtra : 0n0
+0x014 hInstance : 0x00007ff6`01800000 HINSTANCE__
+0x01c hIcon : 0x00000000`01730ecf HICON__
+0x024 hCursor : 0x00000000`00010003 HICON__
+0x02c hbrBackground : 0x00000000`00000006 HBRUSH__
+0x034 lpszMenuName : 0x00000000`0000006d "--- memory read error at address
0x00000000`0000006d ---"
+0x03c lpszClassName : 0x00007ff6`01816bb0 "APPD1A"

23. We close logs in both WinDbg instances:

0:000> .logclose ; AppD1A

Closing open log file C:\AWD3\D1A.log

0:000> .logclose ; AppD1B

Closing open log file C:\AWD3\D1B.log

65
Note: To avoid possible confusion and glitches, we recommend exiting WinDbg after each exercise.

24. The problem was partially fixed without changing alignment by using a different bigger structure
WNDCLASSEX and RegisterClassExW Win32 API function. We open \AWD3\AppD1C\x64\Release\AppD1C.exe in
another WinDbg instance:

Microsoft (R) Windows Debugger Version 10.0.17134.12 AMD64

Copyright (c) Microsoft Corporation. All rights reserved.

CommandLine: C:\AWD3\AppD1C\x64\Release\AppD1C.exe
Symbol search path is: srv*
Executable search path is:
ModLoad: 00007ff7`f84f0000 00007ff7`f850b000 AppD1C.exe
ModLoad: 00007ffd`103a0000 00007ffd`10581000 ntdll.dll
ModLoad: 00007ffd`0d900000 00007ffd`0d9b2000 C:\WINDOWS\System32\KERNEL32.DLL
ModLoad: 00007ffd`0cc50000 00007ffd`0cec3000 C:\WINDOWS\System32\KERNELBASE.dll
ModLoad: 00007ffd`0a1f0000 00007ffd`0a27b000 C:\WINDOWS\SYSTEM32\apphelp.dll
ModLoad: 00007ffd`0d9c0000 00007ffd`0db50000 C:\WINDOWS\System32\USER32.dll
ModLoad: 00007ffd`0c770000 00007ffd`0c790000 C:\WINDOWS\System32\win32u.dll
ModLoad: 00007ffd`0e9e0000 00007ffd`0ea08000 C:\WINDOWS\System32\GDI32.dll
ModLoad: 00007ffd`0d690000 00007ffd`0d822000 C:\WINDOWS\System32\gdi32full.dll
ModLoad: 00007ffd`0cbb0000 00007ffd`0cc4f000 C:\WINDOWS\System32\msvcp_win.dll
ModLoad: 00007ffd`0c7e0000 00007ffd`0c8da000 C:\WINDOWS\System32\ucrtbase.dll
(dec.331c): Break instruction exception - code 80000003 (first chance)
ntdll!LdrpDoDebuggerBreak+0x30:
00007ffd`1046cd9c cc int 3

0:000> .symfix c:\mss

0:000> .reload
Reloading current modules
..........

0:000> bm *!RegisterClassExW
*** WARNING: Unable to verify checksum for AppD1C.exe
1: 00007ffd`0cd40330 @!"KERNELBASE!RegisterClassExW"
2: 00007ffd`0d9c4660 @!"USER32!RegisterClassExW"

0:000> g
ModLoad: 00007ffd`101c0000 00007ffd`101ed000 C:\WINDOWS\System32\IMM32.DLL
Breakpoint 2 hit
USER32!RegisterClassExW:
00007ffd`0d9c4660 4883ec38 sub rsp,38h

0:000> kn
# Child-SP RetAddr Call Site
00 000000a4`e30ff858 00007ff7`f84f118a USER32!RegisterClassExW
01 000000a4`e30ff860 00007ff7`f84f105c AppD1C!MyRegisterClass+0xaa
[c:\awd3\appd1c\appd1c\appd1c.cpp @ 84]
02 000000a4`e30ff8e0 00007ff7`f84f167e AppD1C!wWinMain+0x5c [c:\awd3\appd1c\appd1c\appd1c.cpp @
38]
03 (Inline Function) --------`-------- AppD1C!invoke_main+0x21
[f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 118]
04 000000a4`e30ff940 00007ffd`0d913034 AppD1C!__scrt_common_main_seh+0x106
[f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 288]
05 000000a4`e30ff980 00007ffd`10411431 KERNEL32!BaseThreadInitThunk+0x14
06 000000a4`e30ff9b0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

66
0:000> .frame 1
01 000000a4`e30ff860 00007ff7`f84f105c AppD1C!MyRegisterClass+0xaa
[c:\awd3\appd1c\appd1c\appd1c.cpp @ 84]

0:000> dv /i /V
prv param 000000a4`e30ff8e0 @rsp+0x0080 hInstance = 0x00007ff7`f84f0000
prv local 000000a4`e30ff880 @rsp+0x0020 wcex = struct tagWNDCLASSEXW

Note: Adding a new extra member in the new structure shifts the remaining members and set the same layout as in
AppD1B:

0:000> dt wcex ; AppD1C

Local var @ 0xa4e30ff880 Type tagWNDCLASSEXW
+0x000 cbSize : 0x50
+0x004 style : 3
+0x008 lpfnWndProc : 0x00007ff7`f84f1250 int64 AppD1C!WndProc+0
+0x010 cbClsExtra : 0n0
+0x014 cbWndExtra : 0n0
+0x018 hInstance : 0x00007ff7`f84f0000 HINSTANCE__
+0x020 hIcon : 0x00000000`14a4261d HICON__
+0x028 hCursor : 0x00000000`00010003 HICON__
+0x030 hbrBackground : 0x00000000`00000006 HBRUSH__
+0x038 lpszMenuName : 0x00000000`0000006d "--- memory read error at address
0x00000000`0000006d ---"
+0x040 lpszClassName : 0x00007ff7`f8506bb0 "APPD1C"
+0x048 hIconSm : 0x00000000`00bf1e45 HICON__

0:000> dt wc ; AppD1B
Local var @ 0x754e9bf830 Type tagWNDCLASSW
+0x000 style : 3
+0x008 lpfnWndProc : 0x00007ff6`9d281240 int64 AppD1B!WndProc+0
+0x010 cbClsExtra : 0n0
+0x014 cbWndExtra : 0n0
+0x018 hInstance : 0x00007ff6`9d280000 HINSTANCE__
+0x020 hIcon : 0x00000000`04602229 HICON__
+0x028 hCursor : 0x00000000`00010003 HICON__
+0x030 hbrBackground : 0x00000000`00000006 HBRUSH__
+0x038 lpszMenuName : 0x00000000`0000006d "--- memory read error at address
0x00000000`0000006d ---"
+0x040 lpszClassName : 0x00007ff6`9d296bb0 "APPD1B"

Note: AppD1A wasn’t working because of structure member alignment. This models an old Windows 3.x project that
was ported to x64. It had the minimum alignment in the past to reduce memory consumption:

67
AppD1B was working because the alignment was changed to default. AppD1C still used the same 1-byte alignment
but because the bigger structure shifted members of the substructure it didn’t crash.

68
Published by OpenTask, Republic of Ireland

Copyright © 2017 by OpenTask

Copyright © 2017 by Software Diagnostics Services

Copyright © 2017 by Dmitry Vostokov

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or
transmitted, in any form or by any means, without the prior written permission of the
publisher.

You must not circulate this book in any other binding or cover, and you must impose the same
condition on any acquirer.

Product and company names mentioned in this book may be trademarks of their owners.

OpenTask books and magazines are available through booksellers and distributors worldwide.
For further information or comments send requests to [email protected].

A CIP catalog record for this book is available from the British Library.

ISBN-l3: 978-1-908043-86-3 (Paperback)

Revision 2.02 (October 2017)

2
Contents

About the Author.............................................................................................................................................................. 5

Introduction ...................................................................................................................................................................... 7
Practice Exercises ........................................................................................................................................................... 17
Exercise 0: Download, setup and verify your WinDbg installation ............................................................................ 22
Exercise M1A .............................................................................................................................................................. 35
Exercise M1B .............................................................................................................................................................. 48
Exercise M2................................................................................................................................................................. 60
Exercise M3................................................................................................................................................................. 77
Exercise M4............................................................................................................................................................... 130
Exercise M5............................................................................................................................................................... 186
Exercise M6............................................................................................................................................................... 210
Selected Q&A................................................................................................................................................................ 232
Appendix ....................................................................................................................................................................... 235
Malware Analysis Patterns ....................................................................................................................................... 237
Deviant Module .................................................................................................................................................... 237
Deviant Token....................................................................................................................................................... 244
Driver Device Collection ....................................................................................................................................... 245
Execution Residue ................................................................................................................................................ 246
Fake Module ......................................................................................................................................................... 270
Hidden Module ..................................................................................................................................................... 274
Hidden Process ..................................................................................................................................................... 276
Hooksware ............................................................................................................................................................ 278
Namespace ........................................................................................................................................................... 279
No Component Symbols ....................................................................................................................................... 280
Out-of-Module Pointer ......................................................................................................................................... 283
Packed Code ......................................................................................................................................................... 284
Patched Code........................................................................................................................................................ 287
Pre-Obfuscation Residue ...................................................................................................................................... 288
Raw Pointer .......................................................................................................................................................... 289
RIP Stack Trace ..................................................................................................................................................... 290
Self-Diagnosis (Kernel Mode) ............................................................................................................................... 292
Stack Trace Collection .......................................................................................................................................... 293
Stack Trace Collection (I/O Requests) .................................................................................................................. 301
3
 String Hint ............................................................................................................................................................. 305
Unknown Module ................................................................................................................................................. 307
Raw Stack Dump of All Threads (Kernel Space) ........................................................................................................ 310
Complete Stack Traces from x64 System ................................................................................................................. 311

4
 Exercise M1A

Goal: Look at module headers and version information before load.

Patterns: Unknown Module.

1. Launch WinDbg from Windows Kits \ WinDbg (X64) or Windows Kits \ WinDbg (X86).

2. Open \AWMA-Dumps\Executables\M1.exe

35
3. You get the EXE file loaded:

4. Symbols are not necessary for our exercise.

5. Open a log file:

0:000> .logopen C:\AWMA-Dumps\M1A.log

Opened log file 'C:\AWMA-Dumps\M1A.log'

36
6. lmv command lists module information:

0:000> lmv
start end module name
00000001`40000000 00000001`40018000 M1 C (no symbols)
Loaded symbol image file: M1.exe
Mapped memory image file: C:\AWMA-Dumps\Executables\M1.exe
Image path: C:\AWMA-Dumps\Executables\M1.exe
Image name: M1.exe
Timestamp: Mon Jan 28 15:24:45 2013 (5106983D)
CheckSum: 00000000
ImageSize: 00018000
Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4

Note module default load address.

7. !lmi command gives a bit more information:

0:000> !lmi 00000001`40000000

Loaded Module Info: [00000001`40000000]
Module: M1
Base Address: 0000000140000000
Image Name: M1.exe
Machine Type: 34404 (X64)
Time Stamp: 5106983d Mon Jan 28 15:24:45 2013
Size: 18000
CheckSum: 0
Characteristics: 22
Debug Data Dirs: Type Size VA Pointer
CODEVIEW 3b, e370, cb70 RSDS - GUID: {3F1487A5-A6DC-4351-AD23-76FC12BB9482}
Age: 1, Pdb: C:\Work\AWMA\M1\x64\Release\M1.pdb
?? 10, e3ac, cbac [Data not mapped]
Image Type: FILE - Image read successfully from debugger.
M1.exe
Symbol Type: NONE - PDB not found from image path.
Load Report: no symbols loaded

Note a reference to a PDB file. If left by a developer it might give some clues as we in other exercises.

8. We dump the first kilobyte:

0:000> dc 00000001`40000000 L100

00000001`40000000 00905a4d 00000003 00000004 0000ffff MZ..............
00000001`40000010 000000b8 00000000 00000040 00000000 ........@.......
00000001`40000020 00000000 00000000 00000000 00000000 ................
00000001`40000030 00000000 00000000 00000000 000000e8 ................
00000001`40000040 0eba1f0e cd09b400 4c01b821 685421cd ........!..L.!Th
00000001`40000050 70207369 72676f72 63206d61 6f6e6e61 is program canno
00000001`40000060 65622074 6e757220 206e6920 20534f44 t be run in DOS
00000001`40000070 65646f6d 0a0d0d2e 00000024 00000000 mode....$.......
00000001`40000080 cb8e1818 98e0795c 98e0795c 98e0795c ....\y..\y..\y..
00000001`40000090 982fbfad 98e0794e 982ebfad 98e07908 ../.Ny.......y..
00000001`400000a0 982dbfad 98e0795b 98e1795c 98e07903 ..-.[y..\y...y..
00000001`400000b0 98590ea0 98e07959 9833befe 98e0795e ..Y.Yy....3.^y..
00000001`400000c0 9829befe 98e0795d 9877795c 98e0795d ..).]y..\yw.]y..
00000001`400000d0 982cbefe 98e0795d 68636952 98e0795c ..,.]y..Rich\y..
00000001`400000e0 00000000 00000000 00004550 00068664 ........PE..d...
00000001`400000f0 5106983d 00000000 00000000 002200f0 =..Q..........".
37
00000001`40000100 000b020b 00007400 0000d200 00000000 .....t..........
00000001`40000110 000016a8 00001000 40000000 00000001 ...........@....
00000001`40000120 00001000 00000200 00000006 00000000 ................
00000001`40000130 00000006 00000000 00018000 00000400 ................
00000001`40000140 00000000 81600002 00100000 00000000 ......`.........
00000001`40000150 00001000 00000000 00100000 00000000 ................
00000001`40000160 00001000 00000000 00000000 00000010 ................
00000001`40000170 00000000 00000000 0000eaa4 0000003c ............<...
00000001`40000180 00015000 00001d68 00014000 0000078c .P..h....@......
00000001`40000190 00000000 00000000 00017000 00000530 .........p..0...
00000001`400001a0 00009320 00000038 00000000 00000000 ...8...........
00000001`400001b0 00000000 00000000 00000000 00000000 ................
00000001`400001c0 0000e300 00000070 00000000 00000000 ....p...........
00000001`400001d0 00009000 000002a0 00000000 00000000 ................
00000001`400001e0 00000000 00000000 00000000 00000000 ................
00000001`400001f0 7865742e 00000074 0000731b 00001000 .text....s......
00000001`40000200 00007400 00000400 00000000 00000000 .t..............
00000001`40000210 00000000 60000020 6164722e 00006174 .... ..`.rdata..
00000001`40000220 00006366 00009000 00006400 00007800 fc.......d...x..
00000001`40000230 00000000 00000000 00000000 40000040 ............@..@
00000001`40000240 7461642e 00000061 00003900 00010000 .data....9......
00000001`40000250 00001400 0000dc00 00000000 00000000 ................
00000001`40000260 00000000 c0000040 6164702e 00006174 [email protected]..
00000001`40000270 0000078c 00014000 00000800 0000f000 .....@..........
00000001`40000280 00000000 00000000 00000000 40000040 ............@..@
00000001`40000290 7273722e 00000063 00001d68 00015000 .rsrc...h....P..
00000001`400002a0 00001e00 0000f800 00000000 00000000 ................
00000001`400002b0 00000000 40000040 6c65722e 0000636f ....@[email protected]..
00000001`400002c0 00000c52 00017000 00000e00 00011600 R....p..........
00000001`400002d0 00000000 00000000 00000000 42000040 [email protected]
00000001`400002e0 00000000 00000000 00000000 00000000 ................
00000001`400002f0 00000000 00000000 00000000 00000000 ................
00000001`40000300 00000000 00000000 00000000 00000000 ................
00000001`40000310 00000000 00000000 00000000 00000000 ................
00000001`40000320 00000000 00000000 00000000 00000000 ................
00000001`40000330 00000000 00000000 00000000 00000000 ................
00000001`40000340 00000000 00000000 00000000 00000000 ................
00000001`40000350 00000000 00000000 00000000 00000000 ................
00000001`40000360 00000000 00000000 00000000 00000000 ................
00000001`40000370 00000000 00000000 00000000 00000000 ................
00000001`40000380 00000000 00000000 00000000 00000000 ................
00000001`40000390 00000000 00000000 00000000 00000000 ................
00000001`400003a0 00000000 00000000 00000000 00000000 ................
00000001`400003b0 00000000 00000000 00000000 00000000 ................
00000001`400003c0 00000000 00000000 00000000 00000000 ................
00000001`400003d0 00000000 00000000 00000000 00000000 ................
00000001`400003e0 00000000 00000000 00000000 00000000 ................
00000001`400003f0 00000000 00000000 00000000 00000000 ................

38
9. !dh command dumps PE header:

0:000> !dh 00000001`40000000

File Type: EXECUTABLE IMAGE

FILE HEADER VALUES
8664 machine (X64)
6 number of sections
5106983D time date stamp Mon Jan 28 15:24:45 2013

0 file pointer to symbol table

0 number of symbols
F0 size of optional header
22 characteristics
Executable
App can handle >2gb addresses

OPTIONAL HEADER VALUES

20B magic #
11.00 linker version
7400 size of code
D200 size of initialized data
0 size of uninitialized data
16A8 address of entry point
1000 base of code
----- new -----
0000000140000000 image base
1000 section alignment
200 file alignment
2 subsystem (Windows GUI)
6.00 operating system version
0.00 image version
6.00 subsystem version
18000 size of image
400 size of headers
0 checksum
0000000000100000 size of stack reserve
0000000000001000 size of stack commit
0000000000100000 size of heap reserve
0000000000001000 size of heap commit
8160 DLL characteristics
High entropy VA supported
Dynamic base
NX compatible
Terminal server aware
0 [ 0] address [size] of Export Directory
EAA4 [ 3C] address [size] of Import Directory
15000 [ 1D68] address [size] of Resource Directory
14000 [ 78C] address [size] of Exception Directory
0 [ 0] address [size] of Security Directory
17000 [ 530] address [size] of Base Relocation Directory
9320 [ 38] address [size] of Debug Directory
0 [ 0] address [size] of Description Directory
0 [ 0] address [size] of Special Directory
0 [ 0] address [size] of Thread Storage Directory
E300 [ 70] address [size] of Load Configuration Directory
0 [ 0] address [size] of Bound Import Directory
9000 [ 2A0] address [size] of Import Address Table Directory
0 [ 0] address [size] of Delay Import Directory
0 [ 0] address [size] of COR20 Header Directory

39
 0 [ 0] address [size] of Reserved Directory

SECTION HEADER #1
.text name
731B virtual size
1000 virtual address
7400 size of raw data
400 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
60000020 flags
Code
(no align specified)
Execute Read

SECTION HEADER #2
.rdata name
6366 virtual size
9000 virtual address
6400 size of raw data
7800 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
40000040 flags
Initialized Data
(no align specified)
Read Only

Debug Directories(2)
Type Size Address Pointer
cv 3b e370 cb70 Format: RSDS, guid, 1,
C:\Work\AWMA\M1\x64\Release\M1.pdb
( 12) 10 e3ac cbac

SECTION HEADER #3
.data name
3900 virtual size
10000 virtual address
1400 size of raw data
DC00 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
C0000040 flags
Initialized Data
(no align specified)
Read Write

40
SECTION HEADER #4
.pdata name
78C virtual size
14000 virtual address
800 size of raw data
F000 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
40000040 flags
Initialized Data
(no align specified)
Read Only

SECTION HEADER #5
.rsrc name
1D68 virtual size
15000 virtual address
1E00 size of raw data
F800 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
40000040 flags
Initialized Data
(no align specified)
Read Only

SECTION HEADER #6
.reloc name
C52 virtual size
17000 virtual address
E00 size of raw data
11600 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
42000040 flags
Initialized Data
Discardable
(no align specified)
Read Only

Note Import Directory, Import Address Table Directory, and code .text section.

10. Let’s look at Import Address Table Directory before dynamic linking takes place:

0:000> dps 00000001`40000000+9000

00000001`40009000 ????????`????????
00000001`40009008 ????????`????????
00000001`40009010 ????????`????????
00000001`40009018 ????????`????????
00000001`40009020 ????????`????????
00000001`40009028 ????????`????????
00000001`40009030 ????????`????????
00000001`40009038 ????????`????????
41
00000001`40009040 ????????`????????
00000001`40009048 ????????`????????
00000001`40009050 ????????`????????
00000001`40009058 ????????`????????
00000001`40009060 ????????`????????
00000001`40009068 ????????`????????
00000001`40009070 ????????`????????
00000001`40009078 ????????`????????

We see it is inaccessible or not present. However, Import Directory is available, and we can dump its contents using
the module image address, relative offset, and size (in bytes). It is an array of structures each of 5 double words (4
bytes per double word). This is why we use dd command and divide the size by 4:

0:000> dd 00000001`40000000+EAA4 L3C/4

00000001`4000eaa4 0000eae0 00000000 00000000 0000ed90
00000001`4000eab4 00009000 0000ece0 00000000 00000000
00000001`4000eac4 0000eed8 00009200 00000000 00000000
00000001`4000ead4 00000000 00000000 00000000

The first double word in each structure is a relative offset to a relative offset to an array of names such as function
names, and the fourth double word is a relative offset to an import DLL name:

0:000> da 00000001`40000000+0000ed90
00000001`4000ed90 "KERNEL32.dll"

0:000> da 00000001`40000000+0000eed8
00000001`4000eed8 "USER32.dll"

We now examine function names to be imported from KERNEL32.dll:

0:000> dc 00000001`40000000+0000eae0
00000001`4000eae0 00000000`0000ed80 00000000`0000f34a
00000001`4000eaf0 00000000`0000f33a 00000000`0000f326
00000001`4000eb00 00000000`0000f316 00000000`0000f304
00000001`4000eb10 00000000`0000f2f4 00000000`0000f2e0
00000001`4000eb20 00000000`0000f2d0 00000000`0000f2c4
00000001`4000eb30 00000000`0000f2b2 00000000`0000f29c
00000001`4000eb40 00000000`0000f28e 00000000`0000f282
00000001`4000eb50 00000000`0000eee4 00000000`0000eef6

0:000> dc 00000001`40000000+00000000`0000ed80 L100

00000001`4000ed80 6f4c03c6 694c6461 72617262 00005779 ..LoadLibraryW..
00000001`4000ed90 4e52454b 32334c45 6c6c642e 02330000 KERNEL32.dll..3.
00000001`4000eda0 64616f4c 69727453 0057676e 6f4c021e LoadStringW...Lo
00000001`4000edb0 63416461 656c6563 6f746172 00577372 adAcceleratorsW.
00000001`4000edc0 65470175 73654d74 65676173 03410057 u.GetMessageW.A.
00000001`4000edd0 6e617254 74616c73 63634165 72656c65 TranslateAcceler
00000001`4000ede0 726f7461 03430057 6e617254 74616c73 atorW.C.Translat
00000001`4000edf0 73654d65 65676173 00b60000 70736944 eMessage....Disp
00000001`4000ee00 68637461 7373654d 57656761 02260000 atchMessageW..&.
00000001`4000ee10 64616f4c 6e6f6349 02240057 64616f4c LoadIconW.$.Load
00000001`4000ee20 73727543 0057726f 6552028a 74736967 CursorW...Regist
00000001`4000ee30 6c437265 45737361 00005778 72430071 erClassExW..q.Cr
00000001`4000ee40 65746165 646e6957 7845776f 03240057 eateWindowExW.$.
00000001`4000ee50 776f6853 646e6957 0000776f 7055035b ShowWindow..[.Up
00000001`4000ee60 65746164 646e6957 0000776f 694400b3 dateWindow....Di
00000001`4000ee70 676f6c61 50786f42 6d617261 00ad0057 alogBoxParamW...
00000001`4000ee80 74736544 57796f72 6f646e69 00a10077 DestroyWindow...
00000001`4000ee90 57666544 6f646e69 6f725077 00005763 DefWindowProcW..
42
00000001`4000eea0 6542000e 506e6967 746e6961 00ea0000 ..BeginPaint....
00000001`4000eeb0 50646e45 746e6961 02720000 74736f50 EndPaint..r.Post
00000001`4000eec0 74697551 7373654d 00656761 6e4500e8 QuitMessage...En
00000001`4000eed0 61694464 00676f6c 52455355 642e3233 dDialog.USER32.d
00000001`4000eee0 00006c6c 654701e9 6d6f4374 646e616d ll....GetCommand
00000001`4000eef0 656e694c 03860057 65447349 67677562 LineW...IsDebugg
00000001`4000ef00 72507265 6e657365 038b0074 72507349 erPresent...IsPr
00000001`4000ef10 7365636f 46726f73 75746165 72506572 ocessorFeaturePr
00000001`4000ef20 6e657365 02700074 4c746547 45747361 esent.p.GetLastE
00000001`4000ef30 726f7272 05250000 4c746553 45747361 rror..%.SetLastE
00000001`4000ef40 726f7272 022e0000 43746547 65727275 rror....GetCurre
00000001`4000ef50 6854746e 64616572 00006449 6e450140 [email protected]
00000001`4000ef60 65646f63 6e696f50 00726574 65440118 codePointer...De
00000001`4000ef70 65646f63 6e696f50 00726574 78450173 codePointer.s.Ex
00000001`4000ef80 72507469 7365636f 02860073 4d746547 itProcess...GetM
00000001`4000ef90 6c75646f 6e614865 45656c64 00005778 oduleHandleExW..
00000001`4000efa0 654702bc 6f725074 64644163 73736572 ..GetProcAddress
00000001`4000efb0 03ef0000 746c754d 74794269 576f5465 ....MultiByteToW
00000001`4000efc0 43656469 00726168 654702e4 64745374 ideChar...GetStd
00000001`4000efd0 646e6148 0000656c 72570601 46657469 Handle....WriteF
00000001`4000efe0 00656c69 65470283 646f4d74 46656c75 ile...GetModuleF
00000001`4000eff0 4e656c69 57656d61 02c10000 50746547 ileNameW....GetP
00000001`4000f000 65636f72 65487373 00007061 6547025e rocessHeap..^.Ge
00000001`4000f010 6c694674 70795465 036f0065 74696e49 tFileType.o.Init
00000001`4000f020 696c6169 7243657a 63697469 65536c61 ializeCriticalSe
00000001`4000f030 6f697463 646e416e 6e697053 6e756f43 ctionAndSpinCoun
00000001`4000f040 011f0074 656c6544 72436574 63697469 t...DeleteCritic
00000001`4000f050 65536c61 6f697463 02de006e 53746547 alSection...GetS
00000001`4000f060 74726174 6e497075 00576f66 7551043f tartupInfoW.?.Qu
00000001`4000f070 50797265 6f667265 6e616d72 6f436563 eryPerformanceCo
00000001`4000f080 65746e75 022a0072 43746547 65727275 unter.*.GetCurre
00000001`4000f090 7250746e 7365636f 00644973 654702fb ntProcessId...Ge
00000001`4000f0a0 73795374 546d6574 41656d69 6c694673 tSystemTimeAsFil
00000001`4000f0b0 6d695465 02470065 45746547 7269766e eTime.G.GetEnvir
00000001`4000f0c0 656d6e6f 7453746e 676e6972 00005773 onmentStringsW..
00000001`4000f0d0 724601bd 6e456565 6f726976 6e656d6e ..FreeEnvironmen
00000001`4000f0e0 72745374 73676e69 04bb0057 436c7452 tStringsW...RtlC
00000001`4000f0f0 75747061 6f436572 7865746e 04c20074 aptureContext...
00000001`4000f100 4c6c7452 756b6f6f 6e754670 6f697463 RtlLookupFunctio
00000001`4000f110 746e456e 00007972 745204c9 7269566c nEntry....RtlVir
00000001`4000f120 6c617574 69776e55 0000646e 6e5505a0 tualUnwind....Un
00000001`4000f130 646e6168 4564656c 70656378 6e6f6974 handledException
00000001`4000f140 746c6946 00007265 6553055f 686e5574 Filter.._.SetUnh
00000001`4000f150 6c646e61 78456465 74706563 466e6f69 andledExceptionF
00000001`4000f160 65746c69 02290072 43746547 65727275 ilter.).GetCurre
00000001`4000f170 7250746e 7365636f 057e0073 6d726554 ntProcess.~.Term

43
We can also get offsets by using -i or -a options for !dh command:

0:000> !dh -i 00000001`40000000

_IMAGE_IMPORT_DESCRIPTOR 000000014000eaa4
KERNEL32.dll
0000000140009000 Import Address Table
000000014000EAE0 Import Name Table
0 time date stamp
0 Index of first forwarder reference

_IMAGE_IMPORT_DESCRIPTOR 000000014000eab8
USER32.dll
0000000140009200 Import Address Table
000000014000ECE0 Import Name Table
0 time date stamp
0 Index of first forwarder reference

11. Close the log file:

0:000> .logclose
Closing open log file C:\AWMA-Dumps\M1A.log

To avoid possible confusion and glitches, we recommend exiting WinDbg after each exercise.

44
 Windows Debugging,
Disassembling,
Reversing
Practical Foundations: Training Course

Dmitry Vostokov
Software Diagnostics Services
2

Published by OpenTask, Republic of Ireland

Copyright © 2009 by Dmitry Vostokov

Copyright © 2015 by Software Diagnostics Services

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or
transmitted, in any form or by any means, without the prior written permission of the publisher.

You must not circulate this book in any other binding or cover and you must impose the same
condition on any acquirer.

OpenTask books are available through booksellers and distributors worldwide. For further
information or comments send requests to:

[email protected]

Product and company names mentioned in this book may be trademarks of their owners.

A CIP catalog record for this book is available from the British Library.

ISBN-13: 978-1-908043-94-8

First printing, 2015

Revision 2.0
 Contents 3

Summary of Contents
Contents........................................................................................................................................................................................5

Preface to the New Edition ................................................................................................................................................ 15

Combined Preface from Previous Editions ................................................................................................................. 17

About the Author ................................................................................................................................................................... 19

Chapter x86.1: Memory, Registers, and Simple Arithmetic ................................................................................. 21

Chapter x86.2: Debug and Release Binaries ............................................................................................................... 35

Chapter x86.3: Number Representations .................................................................................................................... 50

Chapter x86.4: Pointers ...................................................................................................................................................... 57

Chapter x86.5: Bytes, Words, and Double Words .................................................................................................... 73

Chapter x86.6: Pointers to Memory ............................................................................................................................... 78

Chapter x86.7: Logical Instructions and EIP ........................................................................................................... 100

Chapter x86.8: Reconstructing a Program with Pointers .................................................................................. 108

Chapter x86.9: Memory and Stacks............................................................................................................................. 116

Chapter x86.10: Frame Pointer and Local Variables ........................................................................................... 136

Chapter x86.11: Function Parameters ....................................................................................................................... 151

Chapter x86.12: More Instructions ............................................................................................................................. 165

Chapter x86.13: Function Pointer Parameters....................................................................................................... 176

Chapter x86.14: Summary of Code Disassembly Patterns ................................................................................ 182

Chapter x64.1: Memory, Registers, and Simple Arithmetic .............................................................................. 187

Chapter x64.2: Debug and Release Binaries ............................................................................................................ 202

Chapter x64.3: Number Representations ................................................................................................................. 217

Chapter x64.4: Pointers ................................................................................................................................................... 224

Chapter x64.5: Bytes, Words, and Double Words ................................................................................................. 242

Chapter x64.6: Pointers to Memory ............................................................................................................................ 248

Chapter x64.7: Logical Instructions and EIP ........................................................................................................... 271

4

Chapter x64.8: Reconstructing a Program with Pointers .................................................................................. 279

Chapter x64.9: Memory and Stacks............................................................................................................................. 288

Chapter x64.10: Local Variables ................................................................................................................................... 308

Chapter x64.11: Function Parameters ....................................................................................................................... 320

Chapter x64.12: More Instructions ............................................................................................................................. 330

Chapter x64.13: Function Pointer Parameters....................................................................................................... 341

Chapter x64.14: Summary of Code Disassembly Patterns ................................................................................ 345

 Contents 5

Contents

Contents........................................................................................................................................................................................5

Preface to the New Edition ................................................................................................................................................ 15

Combined Preface from Previous Editions ................................................................................................................. 17

About the Author ................................................................................................................................................................... 19

Chapter x86.1: Memory, Registers, and Simple Arithmetic ................................................................................. 21

Memory and Registers inside an Idealized Computer ...................................................................................... 21

Memory and Registers inside Intel 32-bit PC ....................................................................................................... 22

“Arithmetic” Project: Memory Layout and Registers ........................................................................................ 23

“Arithmetic” Project: A Computer Program .......................................................................................................... 24

“Arithmetic” Project: Assigning Numbers to Memory Locations ................................................................. 25

Assigning Numbers to Registers ................................................................................................................................ 27

“Arithmetic” Project: Adding Numbers to Memory Cells................................................................................. 28

Incrementing/Decrementing Numbers in Memory and Registers .............................................................. 30

Multiplying Numbers ...................................................................................................................................................... 32

Multiplication and Registers ........................................................................................................................................ 34

Chapter x86.2: Debug and Release Binaries ............................................................................................................... 35

“Arithmetic” Project: C/C++ Program...................................................................................................................... 35

Downloading and Configuring WinDbg Debugger ............................................................................................. 36

WinDbg Disassembly Output – Debug Executable ............................................................................................. 38

WinDbg Disassembly Output – Release Executable........................................................................................... 49

Chapter x86.3: Number Representations .................................................................................................................... 50

Numbers and Their Representations ....................................................................................................................... 50

Decimal Representation (Base Ten) ......................................................................................................................... 51

Ternary Representation (Base Three)..................................................................................................................... 52

6

Binary Representation (Base Two) .......................................................................................................................... 53

Hexadecimal Representation (Base Sixteen) ........................................................................................................ 54

Why Hexadecimals are used? ...................................................................................................................................... 55

Chapter x86.4: Pointers ...................................................................................................................................................... 57

A Definition ......................................................................................................................................................................... 57

“Pointers” Project: Memory Layout and Registers ............................................................................................. 58

“Pointers” Project: Calculations.................................................................................................................................. 59

Using Pointers to Assign Numbers to Memory Cells ......................................................................................... 60

Adding Numbers Using Pointers ................................................................................................................................ 66

Multiplying Numbers Using Pointers ....................................................................................................................... 69

Chapter x86.5: Bytes, Words, and Double Words .................................................................................................... 73

Using Hexadecimal Numbers ...................................................................................................................................... 73

Byte Granularity................................................................................................................................................................ 74

Bit Granularity ................................................................................................................................................................... 75

Memory Layout ................................................................................................................................................................. 76

Chapter x86.6: Pointers to Memory ............................................................................................................................... 78

Pointers Revisited ............................................................................................................................................................ 78

Addressing Types ............................................................................................................................................................. 79

Registers Revisited .......................................................................................................................................................... 85

NULL Pointers.................................................................................................................................................................... 86

Invalid Pointers ................................................................................................................................................................. 87

Variables as Pointers ...................................................................................................................................................... 88

Pointer Initialization ....................................................................................................................................................... 89

Note: Initialized and Uninitialized Data .................................................................................................................. 90

More Pseudo Notation.................................................................................................................................................... 91

“MemoryPointers” Project: Memory Layout ......................................................................................................... 92

 Contents 7

Chapter x86.7: Logical Instructions and EIP ........................................................................................................... 100

Instruction Format........................................................................................................................................................ 100

Logical Shift Instructions ........................................................................................................................................... 101

Logical Operations ........................................................................................................................................................ 102

Zeroing Memory or Registers................................................................................................................................... 103

Instruction Pointer ....................................................................................................................................................... 104

Note: Code Section ........................................................................................................................................................ 105

Chapter x86.8: Reconstructing a Program with Pointers .................................................................................. 108

Example of Disassembly Output: No Optimization ......................................................................................... 108

Reconstructing C/C++ Code: Part 1 ....................................................................................................................... 111

Reconstructing C/C++ Code: Part 2 ....................................................................................................................... 112

Reconstructing C/C++ Code: Part 3 ....................................................................................................................... 113

Reconstructing C/C++ Code: C/C++ program ................................................................................................... 114

Example of Disassembly Output: Optimized Program................................................................................... 115

Chapter x86.9: Memory and Stacks............................................................................................................................. 116

Stack: A Definition......................................................................................................................................................... 116

Stack Implementation in Memory .......................................................................................................................... 117

Things to Remember .................................................................................................................................................... 119

PUSH Instruction ........................................................................................................................................................... 120

POP instruction .............................................................................................................................................................. 121

Register Review ............................................................................................................................................................. 122

Application Memory Simplified ............................................................................................................................... 123

Stack Overflow................................................................................................................................................................ 124

Jumps .................................................................................................................................................................................. 126

Calls ..................................................................................................................................................................................... 128

Call Stack ........................................................................................................................................................................... 130

8

Exploring Stack in WinDbg ........................................................................................................................................ 132

Chapter x86.10: Frame Pointer and Local Variables ........................................................................................... 136

Stack Usage ...................................................................................................................................................................... 136

Register Review ............................................................................................................................................................. 137

Addressing Array Elements ...................................................................................................................................... 138

Stack Structure (No Function Parameters) ........................................................................................................ 139

Raw Stack (No Local Variables and Function Parameters) ......................................................................... 140

Function Prolog .............................................................................................................................................................. 141

Function Epilog .............................................................................................................................................................. 142

“Local Variables” Project ............................................................................................................................................ 143

Disassembly of Optimized Executable (Release Configuration) ................................................................ 148

Advanced Topic: FPO ................................................................................................................................................... 149

Chapter x86.11: Function Parameters ....................................................................................................................... 151

“FunctionParameters” Project ................................................................................................................................. 151

Stack Structure ............................................................................................................................................................... 152

Stack Structure with FPO ........................................................................................................................................... 154

Function Prolog and Epilog ....................................................................................................................................... 156

Project Disassembled Code with Comments...................................................................................................... 157

Release Build with FPO Enabled ............................................................................................................................. 162

Cdecl Calling Convention............................................................................................................................................ 163

Parameter Mismatch Problem ................................................................................................................................. 164

Chapter x86.12: More Instructions ............................................................................................................................. 165

CPU Flags Register ........................................................................................................................................................ 165

The Fastest Way to Fill Memory.............................................................................................................................. 166

Testing for 0..................................................................................................................................................................... 168

TEST - Logical Compare .............................................................................................................................................. 169

 Contents 9

CMP – Compare Two Operands ............................................................................................................................... 170

TEST or CMP? .................................................................................................................................................................. 171

Conditional Jumps ......................................................................................................................................................... 172

The Structure of Registers ......................................................................................................................................... 173

Function Return Value ................................................................................................................................................ 174

Using Byte Registers .................................................................................................................................................... 175

Chapter x86.13: Function Pointer Parameters....................................................................................................... 176

“FunctionPointerParameters” Project .................................................................................................................. 176

Commented Disassembly ........................................................................................................................................... 177

Dynamic Addressing of Local Variables ............................................................................................................... 180

Chapter x86.14: Summary of Code Disassembly Patterns ................................................................................ 182

Function Prolog / Epilog ............................................................................................................................................ 182

Passing Parameters ...................................................................................................................................................... 183

LEA (Load Effective Address) .................................................................................................................................. 184

Accessing Parameters and Local Variables ........................................................................................................ 185

Chapter x64.1: Memory, Registers, and Simple Arithmetic .............................................................................. 187

Memory and Registers inside an Idealized Computer ................................................................................... 187

Memory and Registers inside Intel 64-bit PC .................................................................................................... 188

“Arithmetic” Project: Memory Layout and Registers ..................................................................................... 189

“Arithmetic” Project: A Computer Program ....................................................................................................... 190

“Arithmetic” Project: Assigning Numbers to Memory Locations .............................................................. 191

Assigning Numbers to Registers ............................................................................................................................. 193

“Arithmetic” Project: Adding Numbers to Memory Cells.............................................................................. 194

Incrementing/Decrementing Numbers in Memory and Registers ........................................................... 197

Multiplying Numbers ................................................................................................................................................... 200

Chapter x64.2: Debug and Release Binaries ............................................................................................................ 202

10

“Arithmetic” Project: C/C++ Program................................................................................................................... 202

Downloading and Configuring WinDbg Debugger .......................................................................................... 203

WinDbg Disassembly Output – Debug Executable .......................................................................................... 205

WinDbg Disassembly Output – Release Executable........................................................................................ 216

Chapter x64.3: Number Representations ................................................................................................................. 217

Numbers and Their Representations .................................................................................................................... 217

Decimal Representation (Base Ten) ...................................................................................................................... 218

Ternary Representation (Base Three).................................................................................................................. 219

Binary Representation (Base Two) ....................................................................................................................... 220

Hexadecimal Representation (Base Sixteen) ..................................................................................................... 221

Why Hexadecimals are used? ................................................................................................................................... 222

Chapter x64.4: Pointers ................................................................................................................................................... 224

A Definition ...................................................................................................................................................................... 224

“Pointers” Project: Memory Layout and Registers .......................................................................................... 225

“Pointers” Project: Calculations............................................................................................................................... 226

Using Pointers to Assign Numbers to Memory Cells ...................................................................................... 227

Adding Numbers Using Pointers ............................................................................................................................. 234

Multiplying Numbers Using Pointers .................................................................................................................... 238

Chapter x64.5: Bytes, Words, and Double Words ................................................................................................. 242

Using Hexadecimal Numbers ................................................................................................................................... 242

Byte Granularity............................................................................................................................................................. 243

Bit Granularity ................................................................................................................................................................ 244

Memory Layout .............................................................................................................................................................. 246

Chapter x64.6: Pointers to Memory ............................................................................................................................ 248

Pointers Revisited ......................................................................................................................................................... 248

Addressing Types .......................................................................................................................................................... 249

 Contents 11

Registers Revisited ....................................................................................................................................................... 255

NULL Pointers................................................................................................................................................................. 256

Invalid Pointers .............................................................................................................................................................. 257

Variables as Pointers ................................................................................................................................................... 258

Pointer Initialization .................................................................................................................................................... 259

Note: Initialized and Uninitialized Data ............................................................................................................... 260

More Pseudo Notation................................................................................................................................................. 261

“MemoryPointers” Project: Memory Layout ...................................................................................................... 262

Chapter x64.7: Logical Instructions and EIP ........................................................................................................... 271

Instruction Format........................................................................................................................................................ 271

Logical Shift Instructions ........................................................................................................................................... 272

Logical Operations ........................................................................................................................................................ 273

Zeroing Memory or Registers................................................................................................................................... 274

Instruction Pointer ....................................................................................................................................................... 275

Note: Code Section ........................................................................................................................................................ 277

Chapter x64.8: Reconstructing a Program with Pointers .................................................................................. 279

Example of Disassembly Output: No Optimization ......................................................................................... 279

Reconstructing C/C++ Code: Part 1 ....................................................................................................................... 282

Reconstructing C/C++ Code: Part 2 ....................................................................................................................... 284

Reconstructing C/C++ Code: Part 3 ....................................................................................................................... 285

Reconstructing C/C++ Code: C/C++ program ................................................................................................... 286

Example of Disassembly Output: Optimized Program................................................................................... 287

Chapter x64.9: Memory and Stacks............................................................................................................................. 288

Stack: A Definition......................................................................................................................................................... 288

Stack Implementation in Memory .......................................................................................................................... 289

Things to Remember .................................................................................................................................................... 291

12

PUSH Instruction ........................................................................................................................................................... 292

POP instruction .............................................................................................................................................................. 293

Register Review ............................................................................................................................................................. 294

Application Memory Simplified ............................................................................................................................... 295

Stack Overflow................................................................................................................................................................ 296

Jumps .................................................................................................................................................................................. 298

Calls ..................................................................................................................................................................................... 300

Call Stack ........................................................................................................................................................................... 302

Exploring Stack in WinDbg ........................................................................................................................................ 304

Chapter x64.10: Local Variables ................................................................................................................................... 308

Stack Usage ...................................................................................................................................................................... 308

Addressing Array Elements ...................................................................................................................................... 309

Stack Structure (No Function Parameters) ........................................................................................................ 310

Function Prolog .............................................................................................................................................................. 311

Function Epilog .............................................................................................................................................................. 312

“Local Variables” Project ............................................................................................................................................ 313

Disassembly of Optimized Executable (Release Configuration) ................................................................ 319

Chapter x64.11: Function Parameters ....................................................................................................................... 320

“FunctionParameters” Project ................................................................................................................................. 320

Stack Structure ............................................................................................................................................................... 321

Function Prolog and Epilog ....................................................................................................................................... 323

Project Disassembled Code with Comments...................................................................................................... 325

Parameter Mismatch Problem ................................................................................................................................. 329

Chapter x64.12: More Instructions ............................................................................................................................. 330

CPU Flags Register ........................................................................................................................................................ 330

The Fastest Way to Fill Memory.............................................................................................................................. 331

 Contents 13

Testing for 0..................................................................................................................................................................... 333

TEST - Logical Compare .............................................................................................................................................. 334

CMP – Compare Two Operands ............................................................................................................................... 335

TEST or CMP? .................................................................................................................................................................. 336

Conditional Jumps ......................................................................................................................................................... 337

The Structure of Registers ......................................................................................................................................... 338

Function Return Value ................................................................................................................................................ 339

Using Byte Registers .................................................................................................................................................... 340

Chapter x64.13: Function Pointer Parameters....................................................................................................... 341

“FunctionPointerParameters” Project .................................................................................................................. 341

Commented Disassembly ........................................................................................................................................... 342

Chapter x64.14: Summary of Code Disassembly Patterns ................................................................................ 345

Function Prolog / Epilog ............................................................................................................................................ 345

Parameters and Local Variables .............................................................................................................................. 347

LEA (Load Effective Address) .................................................................................................................................. 349

Accessing Parameters and Local Variables ........................................................................................................ 350

Published by OpenTask, Republic of Ireland

Copyright © 2013 by OpenTask

Copyright © 2013 by Software Diagnostics Services

Copyright © 2013 by Dmitry Vostokov

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or
transmitted, in any form or by any means, without the prior written permission of the
publisher.

You must not circulate this book in any other binding or cover and you must impose the same
condition on any acquirer.

Product and company names mentioned in this book may be trademarks of their owners.

OpenTask books and magazines are available through booksellers and distributors worldwide.
For further information or comments send requests to [email protected].

A CIP catalogue record for this book is available from the British Library.

ISBN-l3: 978-1-908043-67-2 (Paperback)

2
Contents

Presentation Slides and Transcript ................................................................................................................................... 5

Practice Exercises ........................................................................................................................................................... 29
Exercise 0 .................................................................................................................................................................... 34
Exercise R1 .................................................................................................................................................................. 41
Exercise R2 .................................................................................................................................................................. 56
Exercise R3 .................................................................................................................................................................. 73
Exercise R4 .................................................................................................................................................................. 83
Exercise R5 .................................................................................................................................................................. 90
Exercise R6 ................................................................................................................................................................ 101
Memory Cell Diagrams ................................................................................................................................................. 127
MCD-R1..................................................................................................................................................................... 129
MCD-R2..................................................................................................................................................................... 131
MCD-R3..................................................................................................................................................................... 134
MCD-R5..................................................................................................................................................................... 138
MCD-R6..................................................................................................................................................................... 144
Source Code .................................................................................................................................................................. 147
DataTypes.cpp .......................................................................................................................................................... 149
Separate.cpp ............................................................................................................................................................. 154
CPPx64.cpp ............................................................................................................................................................... 155
Selected Q&A................................................................................................................................................................ 161

3
 Exercise R1

Goal: Review x64 assembly fundamentals; learn how to reconstruct stack trace manually.

ADDR Patterns: Universal Pointer, Symbolic Pointer S2, Interpreted Pointer S3, Context Pyramid

Memory Cell Diagrams: Register, Pointer, Stack Frame

1. Launch WinDbg from Windows Kits \ Debugging Tools for Windows (X64)

2. Choose File \ Open Crash Dump… menu option and load \ADDR\MemoryDumps\notepad.dmp.

3. You get the following output:

Microsoft (R) Windows Debugger Version 6.3.9600.16384 AMD64

Copyright (c) Microsoft Corporation. All rights reserved.

Loading Dump File [C:\ADDR\MemoryDumps\notepad.dmp]

User Mini Dump File with Full Memory: Only application data is available

Symbol search path is: *** Invalid ***

****************************************************************************
* Symbol loading may be unreliable without a symbol search path. *
* Use .symfix to have the debugger choose a symbol path. *
* After setting your symbol path, use .reload to refresh symbol locations. *
****************************************************************************
Executable search path is:
Windows 7 Version 7601 (Service Pack 1) MP (4 procs) Free x64
Product: WinNt, suite: SingleUserTS Personal
Machine Name:
Debug session time: Wed Oct 9 20:25:46.000 2013 (UTC + 0:00)
System Uptime: 2 days 23:35:31.218
Process Uptime: 0 days 0:00:53.000
............................
*** ERROR: Symbol file could not be found. Defaulted to export symbols for ntdll.dll -

************* Symbol Loading Error Summary **************

Module name Error
ntdll The system cannot find the file specified

You can troubleshoot most symbol related issues by turning on symbol loading diagnostics (!sym
noisy) and repeating the command that caused symbols to be loaded.
You should also verify that your symbol search path (.sympath) is correct.
*** ERROR: Symbol file could not be found. Defaulted to export symbols for user32.dll -
user32!SfmDxSetSwapChainStats+0x1a:
00000000`77619e6a c3 ret

4. Set up a link to Microsoft symbol server and reload symbol files:

0:000> .symfix c:\mss

0:000> .reload
............................

41
5. We get this stack trace:

0:000> k
Child-SP RetAddr Call Site
00000000`000efdc8 00000000`77619e9e user32!ZwUserGetMessage+0xa
00000000`000efdd0 00000000`ff131064 user32!GetMessageW+0x34
00000000`000efe00 00000000`ff13133c notepad!WinMain+0x182
00000000`000efe80 00000000`7771652d notepad!DisplayNonGenuineDlgWorker+0x2da
00000000`000eff40 00000000`7784c541 kernel32!BaseThreadInitThunk+0xd
00000000`000eff70 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

6. Let’s check the main CPU registers:

0:000> r
rax=00000000000206c0 rbx=00000000000efe40 rcx=000000000d0111c6
rdx=000000000000003b rsi=0000000000000001 rdi=0000000000000000
rip=0000000077619e6a rsp=00000000000efdc8 rbp=00000000ff130000
r8=0000000000000000 r9=ffffffffffffffff r10=0000000000000000
r11=0000000004f424c8 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl zr na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
user32!ZwUserGetMessage+0xa:
00000000`77619e6a c3 ret

Note: The register parts and naming are illustrated in MCD-R1.xlsx A section.

7. The current instruction registers (registers that are used and affected by the current instruction or
semantically tied to it) can be checked by r. command:

0:000> r.
At return instr, rax = 206c0

8. Any register value or its named parts can be checked with ? command:

0:000> ? r11
Evaluate expression: 83109064 = 00000000`04f424c8

0:000> ? r11d
Evaluate expression: 83109064 = 00000000`04f424c8

0:000> ? r11w
Evaluate expression: 9416 = 00000000`000024c8

0:000> ? r11b
Evaluate expression: 200 = 00000000`000000c8

9. Individual parts can also be interpreted using typed r command (here we format them as signed values, see
WinDbg help for all other format types):

0:000> r r9
r9=ffffffffffffffff

0:000> r r9:iq
r9=-1

42
0:000> r r9:id
r9=-1 -1

0:000> r r9:iw
r9=65535 65535 65535 65535

0:000> r r9:ib
r9=255 255 255 255 255 255 255 255

10. Any registry value can be interpreted as a pointer to memory cells, a memory address (Universal Pointer
pattern vs. a pointer that was originally designed to be such). However, memory contents at that address may be
inaccessible or unknown as in the case of RCX and RDI below.

0:000> dp rcx
00000000`0d0111c6 ????????`???????? ????????`????????
00000000`0d0111d6 ????????`???????? ????????`????????
00000000`0d0111e6 ????????`???????? ????????`????????
00000000`0d0111f6 ????????`???????? ????????`????????
00000000`0d011206 ????????`???????? ????????`????????
00000000`0d011216 ????????`???????? ????????`????????
00000000`0d011226 ????????`???????? ????????`????????
00000000`0d011236 ????????`???????? ????????`????????

Note: The following output for R11 is illustrated in MCD-R1.xlsx B section.

0:000> dp r11
00000000`04f424c8 80000710`00020002 50200104`00000a00
00000000`04f424d8 00000000`ff130000 00000000`00000000
00000000`04f424e8 fffff900`c06f2760 00000000`00000000
00000000`04f424f8 fffff900`c06b3ef0 00000000`00000000
00000000`04f42508 00000000`00000000 000000a3`000000ea
00000000`04f42518 000002b9`0000054a 000000a5`000000ec
00000000`04f42528 000002b7`00000537 000007fe`fc00975c
00000000`04f42538 fffff900`c06f23d0 00000000`00000000

0:000> dp rax
00000000`000206c0 00260002`00000000 006e0065`0070004f
00000000`000206d0 0009002e`002e002e 006c0072`00740043
00000000`000206e0 00000000`004f002b 00610053`00260003
00000000`000206f0 00430009`00650076 002b006c`00720074
00000000`00020700 00040000`00000053 00650076`00610053
00000000`00020710 00730041`00260020 0000002e`002e002e
00000000`00020720 00000000`00000000 00670061`00500005
00000000`00020730 00650053`00200065 00700075`00260074

0:000> dp rbx
00000000`000efe40 00000000`0005096e 00000000`00000113
00000000`000efe50 00000000`00000001 00000000`00000000
00000000`000efe60 000002f8`0f5c7a0f 00000000`00000375
00000000`000efe70 00000000`ff13cab0 00000000`ff13133c
00000000`000efe80 00000000`00000000 00000000`00000000
00000000`000efe90 00000000`00000000 00000000`01985022
00000000`000efea0 00000000`00000000 00000000`01985022
00000000`000efeb0 00000000`00000000 00000000`ff13cab0

43
0:000> dp rdi
00000000`00000000 ????????`???????? ????????`????????
00000000`00000010 ????????`???????? ????????`????????
00000000`00000020 ????????`???????? ????????`????????
00000000`00000030 ????????`???????? ????????`????????
00000000`00000040 ????????`???????? ????????`????????
00000000`00000050 ????????`???????? ????????`????????
00000000`00000060 ????????`???????? ????????`????????
00000000`00000070 ????????`???????? ????????`????????

11. We can also specify a range or limit to just one value and use finer granularity for memory dumping:

0:000> dp rax L1
00000000`000206c0 00260002`00000000

Note: The similar output for R11 as below is illustrated in MCD-R1.xlsx C section.

0:000> dd rax
00000000`000206c0 00000000 00260002 0070004f 006e0065
00000000`000206d0 002e002e 0009002e 00740043 006c0072
00000000`000206e0 004f002b 00000000 00260003 00610053
00000000`000206f0 00650076 00430009 00720074 002b006c
00000000`00020700 00000053 00040000 00610053 00650076
00000000`00020710 00260020 00730041 002e002e 0000002e
00000000`00020720 00000000 00000000 00500005 00670061
00000000`00020730 00200065 00650053 00260074 00700075

Note: Visible 00xx00yy pattern in the output of dp command: UNICODE string fragments, an example of Regular
Data memory analysis pattern.

0:000> dw rax
00000000`000206c0 0000 0000 0002 0026 004f 0070 0065 006e
00000000`000206d0 002e 002e 002e 0009 0043 0074 0072 006c
00000000`000206e0 002b 004f 0000 0000 0003 0026 0053 0061
00000000`000206f0 0076 0065 0009 0043 0074 0072 006c 002b
00000000`00020700 0053 0000 0000 0004 0053 0061 0076 0065
00000000`00020710 0020 0026 0041 0073 002e 002e 002e 0000
00000000`00020720 0000 0000 0000 0000 0005 0050 0061 0067
00000000`00020730 0065 0020 0053 0065 0074 0026 0075 0070

0:000> db rax
00000000`000206c0 00 00 00 00 02 00 26 00-4f 00 70 00 65 00 6e 00 ......&.O.p.e.n.
00000000`000206d0 2e 00 2e 00 2e 00 09 00-43 00 74 00 72 00 6c 00 ........C.t.r.l.
00000000`000206e0 2b 00 4f 00 00 00 00 00-03 00 26 00 53 00 61 00 +.O.......&.S.a.
00000000`000206f0 76 00 65 00 09 00 43 00-74 00 72 00 6c 00 2b 00 v.e...C.t.r.l.+.
00000000`00020700 53 00 00 00 00 00 04 00-53 00 61 00 76 00 65 00 S.......S.a.v.e.
00000000`00020710 20 00 26 00 41 00 73 00-2e 00 2e 00 2e 00 00 00 .&.A.s.........
00000000`00020720 00 00 00 00 00 00 00 00-05 00 50 00 61 00 67 00 ..........P.a.g.
00000000`00020730 65 00 20 00 53 00 65 00-74 00 26 00 75 00 70 00 e. .S.e.t.&.u.p.

Note: You may have noticed a slight delay when dumping memory pointed by registers. The faster equivalent
approach is to use @ prefix, for example: @rax:

44
0:000> dp @rax
00000000`000206c0 00260002`00000000 006e0065`0070004f
00000000`000206d0 0009002e`002e002e 006c0072`00740043
00000000`000206e0 00000000`004f002b 00610053`00260003
00000000`000206f0 00430009`00650076 002b006c`00720074
00000000`00020700 00040000`00000053 00650076`00610053
00000000`00020710 00730041`00260020 0000002e`002e002e
00000000`00020720 00000000`00000000 00670061`00500005
00000000`00020730 00650053`00200065 00700075`00260074

12. Notice a difference between a value and its organization in memory stemmed from the little-endian
organization of Intel x86-x64 platform (least significant parts are located at lower addresses):

0:000> dp @rbp L1
00000000`ff130000 00000003`00905a4d

0:000> dd @rbp L2
00000000`ff130000 00905a4d 00000003

Note: The similar double word output for R11 is illustrated in MCD-R1.xlsx C section.

0:000> dp @rbp L1
00000000`ff130000 00000003`00905a4d

0:000> dw @rbp L4
00000000`ff130000 5a4d 0090 0003 0000

0:000> dp @rbp L1
00000000`ff130000 00000003`00905a4d

0:000> db @rbp L8
00000000`ff130000 4d 5a 90 00 03 00 00 00 MZ......

13. Every value can be associated with a symbolic value from PDB symbols files or from the binary (exported
symbols) if available. We call this Symbolic Pointer or S2:

0:000> dps r11

00000000`04f424c8 80000710`00020002
00000000`04f424d0 50200104`00000a00
00000000`04f424d8 00000000`ff130000 notepad!CFileDialogEvents_QueryInterface <PERF>
(notepad+0x0)
00000000`04f424e0 00000000`00000000
00000000`04f424e8 fffff900`c06f2760
00000000`04f424f0 00000000`00000000
00000000`04f424f8 fffff900`c06b3ef0
00000000`04f42500 00000000`00000000
00000000`04f42508 00000000`00000000
00000000`04f42510 000000a3`000000ea
00000000`04f42518 000002b9`0000054a
00000000`04f42520 000000a5`000000ec
00000000`04f42528 000002b7`00000537
00000000`04f42530 000007fe`fc00975c comctl32!Edit_WndProc
00000000`04f42538 fffff900`c06f23d0
00000000`04f42540 00000000`00000000

0:000> ln 000007fe`fc00975c
(000007fe`fc00975c) comctl32!Edit_WndProc | (000007fe`fc00a650)
comctl32!Edit_CalcChangeBlocks
Exact matches:
45
 comctl32!Edit_WndProc (<no parameter info>)

0:000> dt 000007fe`fc00975c
Edit_WndProc
Symbol not found.

Note: The address 00000000`04f42530 that points to 000007fe`fc00975c doesn’t have an associated symbol:

0:000> dt 00000000`04f42530
Symbol not found at address 0000000004f42530.

Note: The next instruction pointer address contained in RIP should have an associated symbol of the current
function in our example, because we have symbols for user32.dll:

0:000> ? @rip
Evaluate expression: 2002886250 = 00000000`77619e6a

0:000> dt @rip
ZwUserGetMessage
Symbol not found.

0:000> r
rax=00000000000206c0 rbx=00000000000efe40 rcx=000000000d0111c6
rdx=000000000000003b rsi=0000000000000001 rdi=0000000000000000
rip=0000000077619e6a rsp=00000000000efdc8 rbp=00000000ff130000
r8=0000000000000000 r9=ffffffffffffffff r10=0000000000000000
r11=0000000004f424c8 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl zr na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
user32!ZwUserGetMessage+0xa:
00000000`77619e6a c3 ret

14. Now we come to the next pointer level after its value and its symbol: its interpretation. We call it an
Interpreted Pointer, S3. Such interpretation is implemented either via typed structures (dt command) or via various
WinDbg extension commands (! Commands) that format information for us. In our example we would like to check
memory pointed to by the value of RBX register. We suspect it might be MSG structure related to get message loop:

typedef struct tagMSG {

HWND hwnd;
UINT message;
WPARAM wParam;
LPARAM lParam;
DWORD time;
POINT pt;
} MSG;

0:000> dp @rbx
00000000`000efe40 00000000`0005096e 00000000`00000113
00000000`000efe50 00000000`00000001 00000000`00000000
00000000`000efe60 000002f8`0f5c7a0f 00000000`00000375
00000000`000efe70 00000000`ff13cab0 00000000`ff13133c
00000000`000efe80 00000000`00000000 00000000`00000000
00000000`000efe90 00000000`00000000 00000000`01985022
00000000`000efea0 00000000`00000000 00000000`01985022
00000000`000efeb0 00000000`00000000 00000000`ff13cab0

46
Note: The raw structure makes sense for WM_TIMER message (0x113) where wParam is a time ID (1) and usually a
callback function (lParam) is NULL (0x0). Also mouse pointer data makes sense. Unfortunately, MSG structure is not
available in symbol files available for notepad memory dump. However, we can load a different unrelated module
with better symbol files, for example, CPUx64.exe from C:\ADDR\MemoryDumps\ExtraSymbols which was compiled
as Windows application with full symbols and so should have structures necessary for thread message loop
processing.

15. We add an additional symbol file path:

0:000> .sympath+ C:\ADDR\MemoryDumps\ExtraSymbols

Symbol search path is: srv*;C:\ADDR\MemoryDumps\ExtraSymbols
Expanded Symbol search path is:
SRV*c:\mss*http://msdl.microsoft.com/download/symbols;c:\addr\memorydumps\extrasymbols

We need to find an address to “load” CPUx64 module with its symbols. We choose a committed address 02000000
from the output of !address command:

0:000> !address

Mapping file section regions...

Mapping module regions...
Mapping PEB regions...
Mapping TEB and stack regions...
Mapping heap regions...
Mapping page heap regions...
Mapping other regions...
Mapping stack trace database regions...
Mapping activation context regions...

BaseAddress EndAddress+1 RegionSize Type State Protect Usage

----------------------------------------------------------------------------------------------------------------------
--
[…]
0`01ffe000 0`01fff000 0`00001000 MEM_PRIVATE MEM_RESERVE
PageHeap [PageHeap: 18f1000; NormalHeap: 2920000]
0`01fff000 0`02000000 0`00001000 MEM_PRIVATE MEM_COMMIT PAGE_NOACCESS
PageHeap [PageHeap: 18f1000; NormalHeap: 2920000]
0`02000000 0`02001000 0`00001000 MEM_PRIVATE MEM_RESERVE
PageHeap [PageHeap: 18f1000; NormalHeap: 2920000]
0`02001000 0`02002000 0`00001000 MEM_PRIVATE MEM_COMMIT PAGE_NOACCESS
PageHeap [PageHeap: 18f1000; NormalHeap: 2920000]
0`02002000 0`02003000 0`00001000 MEM_PRIVATE MEM_RESERVE
PageHeap [PageHeap: 18f1000; NormalHeap: 2920000]
0`02003000 0`02004000 0`00001000 MEM_PRIVATE MEM_COMMIT PAGE_NOACCESS
PageHeap [PageHeap: 18f1000; NormalHeap: 2920000]
[…]

0:000> .reload /f C:\ADDR\MemoryDumps\ExtraSymbols\CPUx64=02000000

0:000> lm m CPU*
start end module name
00000000`02000000 00000000`02000000 CPUx64 (private pdb symbols) c:\addr\memorydumps\extrasymbols\CPUx64.pdb

47
16. Now we are able to use MSG structure:

0:000> dt MSG
CPUx64!MSG
+0x000 hwnd : Ptr64 HWND__
+0x008 message : Uint4B
+0x010 wParam : Uint8B
+0x018 lParam : Int8B
+0x020 time : Uint4B
+0x024 pt : tagPOINT

0:000> dt -r MSG
CPUx64!MSG
+0x000 hwnd : Ptr64 HWND__
+0x000 unused : Int4B
+0x008 message : Uint4B
+0x010 wParam : Uint8B
+0x018 lParam : Int8B
+0x020 time : Uint4B
+0x024 pt : tagPOINT
+0x000 x : Int4B
+0x004 y : Int4B

0:000> dt -r MSG @rbx

CPUx64!MSG
+0x000 hwnd : 0x00000000`0005096e HWND__
+0x000 unused : 0n0
+0x008 message : 0x113
+0x010 wParam : 1
+0x018 lParam : 0n0
+0x020 time : 0xf5c7a0f
+0x024 pt : tagPOINT
+0x000 x : 0n760
+0x004 y : 0n885

17. When we have an exception such as a breakpoint or access violation the values of the thread CPU registers
are saved in the so called exception context structure and valid for the currently executing function and its next
instruction pointed to by RIP register (the topmost frame). In other situations such as a manual memory dump we
can only be sure about some registers such as RIP and RSP:

0:000> k
Child-SP RetAddr Call Site
00000000`000efdc8 00000000`77619e9e user32!ZwUserGetMessage+0xa
00000000`000efdd0 00000000`ff131064 user32!GetMessageW+0x34
00000000`000efe00 00000000`ff13133c notepad!WinMain+0x182
00000000`000efe80 00000000`7771652d notepad!DisplayNonGenuineDlgWorker+0x2da
00000000`000eff40 00000000`7784c541 kernel32!BaseThreadInitThunk+0xd
00000000`000eff70 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

48
0:000> r
rax=00000000000206c0 rbx=00000000000efe40 rcx=000000000d0111c6
rdx=000000000000003b rsi=0000000000000001 rdi=0000000000000000
rip=0000000077619e6a rsp=00000000000efdc8 rbp=00000000ff130000
r8=0000000000000000 r9=ffffffffffffffff r10=0000000000000000
r11=0000000004f424c8 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl zr na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
user32!ZwUserGetMessage+0xa:
00000000`77619e6a c3 ret

18. In any situation when we move down to the next frame, for example, to GetMessageW+0x34 (which points
to the next instruction after ZwUserGetMessage was called), we don’t have its CPU registers values saved previously
(r command gives values only for the topmost frame 0):

0:000> k
Child-SP RetAddr Call Site
00000000`000efdc8 00000000`77619e9e user32!ZwUserGetMessage+0xa
00000000`000efdd0 00000000`ff131064 user32!GetMessageW+0x34
00000000`000efe00 00000000`ff13133c notepad!WinMain+0x182
00000000`000efe80 00000000`7771652d notepad!DisplayNonGenuineDlgWorker+0x2da
00000000`000eff40 00000000`7784c541 kernel32!BaseThreadInitThunk+0xd
00000000`000eff70 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

0:000> ub 00000000`77619e9e
user32!GetMessageW+0xc:
00000000`77619e80 b90000feff mov ecx,0FFFE0000h
00000000`77619e85 410bc1 or eax,r9d
00000000`77619e88 458bd1 mov r10d,r9d
00000000`77619e8b 85c1 test ecx,eax
00000000`77619e8d 0f85968d0100 jne user32!GetMessageW+0x1b (00000000`77632c29)
00000000`77619e93 458bca mov r9d,r10d
00000000`77619e96 488bcb mov rcx,rbx
00000000`77619e99 e8c2ffffff call user32!ZwUserGetMessage (00000000`77619e60)

0:000> u 00000000`77619e9e
user32!GetMessageW+0x34:
00000000`77619e9e 817b0802010000 cmp dword ptr [rbx+8],102h
00000000`77619ea5 448bd0 mov r10d,eax
00000000`77619ea8 0f844e480000 je user32!GetMessageW+0x49 (00000000`7761e6fc)
00000000`77619eae 817b08cc000000 cmp dword ptr [rbx+8],0CCh
00000000`77619eb5 0f8441480000 je user32!GetMessageW+0x49 (00000000`7761e6fc)
00000000`77619ebb 418bc2 mov eax,r10d
00000000`77619ebe 4883c420 add rsp,20h
00000000`77619ec2 5b pop rbx

0:000> kn
# Child-SP RetAddr Call Site
00 00000000`000efdc8 00000000`77619e9e user32!ZwUserGetMessage+0xa
01 00000000`000efdd0 00000000`ff131064 user32!GetMessageW+0x34
02 00000000`000efe00 00000000`ff13133c notepad!WinMain+0x182
03 00000000`000efe80 00000000`7771652d notepad!DisplayNonGenuineDlgWorker+0x2da
04 00000000`000eff40 00000000`7784c541 kernel32!BaseThreadInitThunk+0xd
05 00000000`000eff70 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

0:000> .frame 1
01 00000000`000efdd0 00000000`ff131064 user32!GetMessageW+0x34

49
0:000> r
rax=00000000000206c0 rbx=00000000000efe40 rcx=000000000d0111c6
rdx=000000000000003b rsi=0000000000000001 rdi=0000000000000000
rip=0000000077619e6a rsp=00000000000efdc8 rbp=00000000ff130000
r8=0000000000000000 r9=ffffffffffffffff r10=0000000000000000
r11=0000000004f424c8 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl zr na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
user32!ZwUserGetMessage+0xa:
00000000`77619e6a c3 ret

19. But some CPU registers can be recovered such as RIP (saved address when using call instruction) and RSP
(the stack pointer value that was before saving that RIP address). Other register values can be recovered too if they
were not used in called frames or were saved in temporary memory cells (such as on stack). Let’s recover some
registers for the first few frames.

0:000> r
rax=00000000000206c0 rbx=00000000000efe40 rcx=000000000d0111c6
rdx=000000000000003b rsi=0000000000000001 rdi=0000000000000000
rip=0000000077619e6a rsp=00000000000efdc8 rbp=00000000ff130000
r8=0000000000000000 r9=ffffffffffffffff r10=0000000000000000
r11=0000000004f424c8 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl zr na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
user32!ZwUserGetMessage+0xa:
00000000`77619e6a c3 ret

Let’s disassemble the current function:

0:000> uf user32!ZwUserGetMessage
user32!ZwUserGetMessage:
00000000`77619e60 4c8bd1 mov r10,rcx
00000000`77619e63 b806100000 mov eax,1006h
00000000`77619e68 0f05 syscall
00000000`77619e6a c3 ret

It is a very short function we see it overwrites R10 and EAX. Note that EAX value also don’t correspond to what we
see in the output of r command:

0:000> r @eax
eax=206c0

We see that RSP is not used inside ZwUserGetMessage function and its value should point to the return address of
the caller, GetMessageW function during execution of call instruction:

0:000> dp @rsp
00000000`000efdc8 00000000`77619e9e 00000000`00000000
00000000`000efdd8 00000000`00000000 00000000`00000000
00000000`000efde8 00000000`00000000 00000000`01b20455
00000000`000efdf8 00000000`ff131064 00000000`01950048
00000000`000efe08 00000000`01b20455 000007fe`ff552164
00000000`000efe18 00000000`00000001 00000000`0000193c
00000000`000efe28 000007fe`00000000 00000000`00000000
00000000`000efe38 00000000`00000000 00000000`0005096e

50
0:000> ub 00000000`77619e9e
user32!GetMessageW+0xc:
00000000`77619e80 b90000feff mov ecx,0FFFE0000h
00000000`77619e85 410bc1 or eax,r9d
00000000`77619e88 458bd1 mov r10d,r9d
00000000`77619e8b 85c1 test ecx,eax
00000000`77619e8d 0f85968d0100 jne user32!GetMessageW+0x1b (00000000`77632c29)
00000000`77619e93 458bca mov r9d,r10d
00000000`77619e96 488bcb mov rcx,rbx
00000000`77619e99 e8c2ffffff call user32!ZwUserGetMessage (00000000`77619e60)

0:000> u 00000000`77619e9e
user32!GetMessageW+0x34:
00000000`77619e9e 817b0802010000 cmp dword ptr [rbx+8],102h
00000000`77619ea5 448bd0 mov r10d,eax
00000000`77619ea8 0f844e480000 je user32!GetMessageW+0x49 (00000000`7761e6fc)
00000000`77619eae 817b08cc000000 cmp dword ptr [rbx+8],0CCh
00000000`77619eb5 0f8441480000 je user32!GetMessageW+0x49 (00000000`7761e6fc)
00000000`77619ebb 418bc2 mov eax,r10d
00000000`77619ebe 4883c420 add rsp,20h
00000000`77619ec2 5b pop rbx

This is RIP value but RSP should be the value before call instruction was executed. When a return value is saved RSP
is decremented by 8 so the value of RSP before call should be the value of RSP pointing to the saved return address +
8:

0:000> ? @rsp + 8
Evaluate expression: 982480 = 00000000`000efdd0

0:000> k
Child-SP RetAddr Call Site
00000000`000efdc8 00000000`77619e9e user32!ZwUserGetMessage+0xa
00000000`000efdd0 00000000`ff131064 user32!GetMessageW+0x34
00000000`000efe00 00000000`ff13133c notepad!WinMain+0x182
00000000`000efe80 00000000`7771652d notepad!DisplayNonGenuineDlgWorker+0x2da
00000000`000eff40 00000000`7784c541 kernel32!BaseThreadInitThunk+0xd
00000000`000eff70 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

Let’s now find out RIP and RSP for the next frame (the caller of GetMessageW function). To find out RSP we need see
how it was used in the callee, GetMessageW function before the callee called ZwUserGetMessage. We disassemble
GetMessageW function:

0:000> uf user32!GetMessageW
user32!GetMessageW:
00000000`77619e74 fff3 push rbx
00000000`77619e76 4883ec20 sub rsp,20h
00000000`77619e7a 418bc0 mov eax,r8d
00000000`77619e7d 488bd9 mov rbx,rcx
00000000`77619e80 b90000feff mov ecx,0FFFE0000h
00000000`77619e85 410bc1 or eax,r9d
00000000`77619e88 458bd1 mov r10d,r9d
00000000`77619e8b 85c1 test ecx,eax
00000000`77619e8d 0f85968d0100 jne user32!GetMessageW+0x1b (00000000`77632c29)

user32!GetMessageW+0x29:
00000000`77619e93 458bca mov r9d,r10d
00000000`77619e96 488bcb mov rcx,rbx
00000000`77619e99 e8c2ffffff call user32!ZwUserGetMessage (00000000`77619e60)
00000000`77619e9e 817b0802010000 cmp dword ptr [rbx+8],102h
51
00000000`77619ea5 448bd0 mov r10d,eax
00000000`77619ea8 0f844e480000 je user32!GetMessageW+0x49 (00000000`7761e6fc)

user32!GetMessageW+0x40:
00000000`77619eae 817b08cc000000 cmp dword ptr [rbx+8],0CCh
00000000`77619eb5 0f8441480000 je user32!GetMessageW+0x49 (00000000`7761e6fc)

user32!GetMessageW+0x51:
00000000`77619ebb 418bc2 mov eax,r10d
00000000`77619ebe 4883c420 add rsp,20h
00000000`77619ec2 5b pop rbx
00000000`77619ec3 c3 ret

user32!GetMessageW+0x49:
00000000`7761e6fc 48816310ffff0000 and qword ptr [rbx+10h],0FFFFh
00000000`7761e704 e9b2b7ffff jmp user32!GetMessageW+0x51 (00000000`77619ebb)

user32!GetMessageW+0x1b:
00000000`77632c29 4183f9ff cmp r9d,0FFFFFFFFh
00000000`77632c2d 750d jne user32!GetMessageW+0x5a (00000000`77632c3c)

user32!GetMessageW+0x21:
00000000`77632c2f 4485c1 test ecx,r8d
00000000`77632c32 7508 jne user32!GetMessageW+0x5a (00000000`77632c3c)

user32!GetMessageW+0x26:
00000000`77632c34 4533d2 xor r10d,r10d
00000000`77632c37 e95772feff jmp user32!GetMessageW+0x29 (00000000`77619e93)

user32!GetMessageW+0x5a:
00000000`77632c3c b957000000 mov ecx,57h
00000000`77632c41 ff1561f60400 call qword ptr [user32!_imp_RtlSetLastWin32Error
(00000000`776822a8)]
00000000`77632c47 4533d2 xor r10d,r10d
00000000`77632c4a e96c72feff jmp user32!GetMessageW+0x51 (00000000`77619ebb)

We see that stack pointer was decremented by 0x20 (sub instruction) and also by 8 (push instruction) and so we add
these values to RSP we found out previously for ZwUserGetMessage call, 00000000`000efdd0:

0:000> dps 00000000`000efdd0 + 20 + 8

00000000`000efdf8 00000000`ff131064 notepad!WinMain+0x182
00000000`000efe00 00000000`01950048
00000000`000efe08 00000000`01b20455
00000000`000efe10 000007fe`ff552164 msctf!UIWndProc
00000000`000efe18 00000000`00000001
00000000`000efe20 00000000`0000193c
00000000`000efe28 000007fe`00000000
00000000`000efe30 00000000`00000000
00000000`000efe38 00000000`00000000
00000000`000efe40 00000000`0005096e
00000000`000efe48 00000000`00000113
00000000`000efe50 00000000`00000001
00000000`000efe58 00000000`00000000
00000000`000efe60 000002f8`0f5c7a0f
00000000`000efe68 00000000`00000375
00000000`000efe70 00000000`ff13cab0 notepad!_xi_z

52
We see that GetMessageW was called from WinMain function:

0:000> ub 00000000`ff131064
notepad!WinMain+0xf5:
00000000`ff131046 ff1544b40000 call qword ptr [notepad!_imp_SetWinEventHook
(00000000`ff13c490)]
00000000`ff13104c 488bd8 mov rbx,rax
00000000`ff13104f eb00 jmp notepad!WinMain+0x16f (00000000`ff131051)
00000000`ff131051 488d4c2440 lea rcx,[rsp+40h]
00000000`ff131056 4533c9 xor r9d,r9d
00000000`ff131059 4533c0 xor r8d,r8d
00000000`ff13105c 33d2 xor edx,edx
00000000`ff13105e ff1524b40000 call qword ptr [notepad!_imp_GetMessageW
(00000000`ff13c488)]

The value of RSP before call should be adjusted by 8 due to saved return address:

0:000> ? 00000000`000efdf8 + 8
Evaluate expression: 982528 = 00000000`000efe00

0:000> k
Child-SP RetAddr Call Site
00000000`000efdc8 00000000`77619e9e user32!ZwUserGetMessage+0xa
00000000`000efdd0 00000000`ff131064 user32!GetMessageW+0x34
00000000`000efe00 00000000`ff13133c notepad!WinMain+0x182
00000000`000efe80 00000000`7771652d notepad!DisplayNonGenuineDlgWorker+0x2da
00000000`000eff40 00000000`7784c541 kernel32!BaseThreadInitThunk+0xd
00000000`000eff70 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

And so on we are able to reconstruct the stack trace like a debugger. Note that we are able to correctly disassemble
functions using uf command because function boundaries are saved in PDB symbol files or the start of the function is
available from image file as an exported function. If such information is not available we would most likely have a
truncated stack trace.

20. Other registers and memory values are reused and overwritten when we move down the frames so less and
less information can be recovered. We call this ADDR pattern (Inverse) Context Pyramid.

21. We also introduce special Stack Frame memory cell diagrams. The case of stack frame for GetMessageW
function before calling ZwUserGetMessage is illustrated in MCD-R1.xlsx section D.

22. To avoid possible confusion and glitches we recommend exiting WinDbg after each exercise.

53
 MCD-R1

A. Main Registers

RAX

RAX EAX

RAX EAX AX

RAX EAX AH | AL

RSI

RSI ESI

RSI ESI SI

RSI ESI | SIL

R8

R8 R8D

R8 R8D R8W

R8 R8D |R8B

129
B. Universal Pointer
We use a similar color for the value it points to

R11

C. Pointing to a double word

R11

D. Stack Frame

RSP
8
10
18
20
28
30
38
40
48
50

130
Published by OpenTask, Republic of Ireland

Copyright © 2013 by OpenTask

Copyright © 2013 by Software Diagnostics Services

Copyright © 2013 by Dmitry Vostokov

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or
transmitted, in any form or by any means, without the prior written permission of the
publisher.

You must not circulate this book in any other binding or cover, and you must impose the same
condition on any acquirer.

Product and company names mentioned in this book may be trademarks of their owners.

OpenTask books and magazines are available through booksellers and distributors worldwide.
For further information or comments send requests to [email protected].

A CIP catalogue record for this book is available from the British Library.

ISBN-l3: 978-1-908043-42-9 (Paperback)

Revision 2 (February 2016)

2
Contents

Presentation Slides and Transcript .................................................................................................................................5

Practice Exercises .......................................................................................................................................................111
App Source Code ........................................................................................................................................................125

3
7
Published by OpenTask, Republic of Ireland

Copyright © 2017 by OpenTask

Copyright © 2017 by Software Diagnostics Services

Copyright © 2017 by Dmitry Vostokov

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or
transmitted, in any form or by any means, without the prior written permission of the
publisher.

You must not circulate this book in any other binding or cover, and you must impose the same
condition on any acquirer.

Product and company names mentioned in this book may be trademarks of their owners.

OpenTask books and magazines are available through booksellers and distributors worldwide.
For further information or comments send requests to [email protected].

A CIP catalog record for this book is available from the British Library.

ISBN-l3: 978-1-908043-84-9 (Paperback)

Version 3, 2017

Revision 3.00 (June 2017)

2
Contents

About the Author ...........................................................................................................................................................5

Presentation Slides and Transcript .................................................................................................................................7
Practice Exercises .........................................................................................................................................................13
Exercise 0: Download, setup and verify your WinDbg installation ............................................................................18
Exercise C1: Stack Trace Collection (64-bit) ..............................................................................................................25
Exercise C2: Memory Search (64-bit) ........................................................................................................................66
Exercise C3: Linked Lists (64-bit) ...............................................................................................................................80
Exercise C4A: WinDbg Built-in Scripting (64-bit) .....................................................................................................133
Exercise C4B: WinDbg JavaScript Scripting (64-bit) ................................................................................................151
Exercise C5: Registry (64-bit) ..................................................................................................................................167
Exercise C6: Module Variables (64-bit) ...................................................................................................................176
Exercise C7: System Objects (64-bit) ......................................................................................................................181
Exercise C8: Network (64-bit) .................................................................................................................................191
Exercise C9: Device Drivers (64-bit) ........................................................................................................................205
Exercise C10: Storage and File System (64-bit) .......................................................................................................221
Exercise C11: Window Messaging (64-bit) ..............................................................................................................226
Legacy Exercises .........................................................................................................................................................239
Exercise Legacy.0: Download, setup and verify your WinDbg installation ..............................................................241
Exercise Legacy.C1: Stack Trace Collection (64-bit) ................................................................................................246
Exercise Legacy.C2: Memory Search (64-bit) ..........................................................................................................271
Exercise Legacy.C3: Linked Lists (64-bit) .................................................................................................................282
Exercise Legacy.C4: Scripting (64-bit) .....................................................................................................................311
Exercise Legacy.C5: Registry (64-bit) ......................................................................................................................328
Exercise Legacy.C6: Module Variables (64-bit) .......................................................................................................336
Exercise Legacy.C7: System Objects (64-bit) ...........................................................................................................340
Exercise Legacy.C8: Network (64-bit) .....................................................................................................................346
Exercise Legacy.C9: Device Drivers (64-bit) ............................................................................................................354
Selected Q&A .............................................................................................................................................................365

3
 Exercise C1: Stack Trace Collection (64-bit)

Goal: Learn how to get stack traces related to sessions, processes, and threads; diagnose different thread types; get
stack traces from WOW64 processes.

Patterns: Stack Trace Collection (unmanaged space); Passive Thread; Coupled Processes (weak); Coupled Processes
(strong); Wait Chain (ALPC); Virtualized Process; Truncated Stack Trace.

1. Launch WinDbg from Windows Kits \ WinDbg (X64).

2. Open \AdvMDA-Dumps\x64\MEMORY-Normal.DMP

3. We get the dump file loaded:

Microsoft (R) Windows Debugger Version 10.0.15063.137 AMD64

Copyright (c) Microsoft Corporation. All rights reserved.

Loading Dump File [F:\AdvWMDA-Dumps\x64\MEMORY-Normal.DMP]

Kernel Bitmap Dump File: Full address space is available

Symbol search path is: srv*

Executable search path is:
Windows 10 Kernel Version 10586 MP (4 procs) Free x64
Product: WinNt, suite: TerminalServer SingleUserTS Personal
Built by: 10586.103.amd64fre.th2_release.160126-1819
Machine Name:
Kernel base = 0xfffff801`4868a000 PsLoadedModuleList = 0xfffff801`48968cf0
Debug session time: Thu May 19 00:13:25.654 2016 (UTC + 1:00)
System Uptime: 0 days 0:02:48.462
Loading Kernel Symbols
...............................................................
................................................................
......................................
Loading User Symbols
...........................................
Loading unloaded module list
.............
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck D1, {ffffc000dd71a800, 2, 0, fffff801c17a1385}

*** ERROR: Module load completed but symbols could not be loaded for myfault.sys
*** ERROR: Module load completed but symbols could not be loaded for NotMyfault.exe
Probably caused by : myfault.sys ( myfault+1385 )

Followup: MachineOwner
-----------

Note: Probably caused by myfault.sys. We used NotMyFault tool from Windows Internals:

http://technet.microsoft.com/en-us/sysinternals/bb963901

http://download.sysinternals.com/files/NotMyFault.zip
25
4. We open a log file, set up symbols and reload them:

3: kd> .logopen F:\AdvWMDA-Dumps\x64\C1.log

Opened log file 'F:\AdvWMDA-Dumps\x64\C1.log'

3: kd> .symfix c:\mss

3: kd> .reload
Loading Kernel Symbols
...............................................................
................................................................
......................................
Loading User Symbols
...........................................
Loading unloaded module list
.............

5. We list running sessions:

3: kd> !session
Sessions on machine: 2
Valid Sessions: 0 1
Current Session 1

6. We check the current process:

3: kd> !process
PROCESS ffffe000ec09a080
SessionId: 1 Cid: 1594 Peb: 00379000 ParentCid: 0c64
DirBase: 3cfce000 ObjectTable: ffffc000dd91c2c0 HandleCount: <Data Not Accessible>
Image: NotMyfault.exe
VadRoot ffffe000eb3fb4b0 Vads 92 Clone 0 Private 473. Modified 6. Locked 0.
DeviceMap ffffc000db0667a0
Token ffffc000dd9d5a90
ElapsedTime 00:00:05.488
UserTime 00:00:00.000
KernelTime 00:00:00.000
QuotaPoolUsage[PagedPool] 224896
QuotaPoolUsage[NonPagedPool] 12632
Working Set Sizes (now,min,max) (3220, 50, 345) (12880KB, 200KB, 1380KB)
PeakWorkingSetSize 3149
VirtualSize 115 Mb
PeakVirtualSize 115 Mb
PageFaultCount 3323
MemoryPriority FOREGROUND
BasePriority 8
CommitCharge 539
Job ffffe000ec07ead0

THREAD ffffe000ecab7080 Cid 1594.08cc Teb: 000000000037a000 Win32Thread: ffffe000ebbfee30

RUNNING on processor 3
THREAD ffffe000ec360080 Cid 1594.1538 Teb: 000000000037c000 Win32Thread: 0000000000000000 WAIT:
(WrQueue) UserMode Alertable
ffffe000ebdf0200 QueueObject

THREAD ffffe000ec16e080 Cid 1594.1540 Teb: 000000000037e000 Win32Thread: 0000000000000000 WAIT:

(WrQueue) UserMode Alertable
ffffe000ebdf0200 QueueObject

THREAD ffffe000ec97c840 Cid 1594.1544 Teb: 0000000000380000 Win32Thread: 0000000000000000 WAIT:

(WrQueue) UserMode Alertable
ffffe000ebdf0200 QueueObject

26
 THREAD ffffe000ec41f040 Cid 1594.154c Teb: 0000000000382000 Win32Thread: 0000000000000000 WAIT:
(WrQueue) UserMode Alertable
ffffe000ec00bb40 QueueObject

THREAD ffffe000ec43a080 Cid 1594.0614 Teb: 0000000000384000 Win32Thread: 0000000000000000 WAIT:

(WrQueue) UserMode Alertable
ffffe000ec00bb40 QueueObject

THREAD ffffe000ec474080 Cid 1594.17b0 Teb: 0000000000386000 Win32Thread: 0000000000000000 WAIT:

(UserRequest) UserMode Non-Alertable
ffffe000ec658730 SynchronizationTimer

THREAD ffffe000ec475080 Cid 1594.17ac Teb: 0000000000388000 Win32Thread: 0000000000000000 WAIT:

(UserRequest) UserMode Alertable
ffffe000ec08c9c0 SynchronizationEvent
ffffe000ec675c50 SynchronizationTimer

7. We set the current session 0 and examine its implicit process:

3: kd> !session -s 0
Sessions on machine: 2
Implicit process is now ffffe000`eb239080
Using session 0

3: kd> !process ffffe000`eb239080 3f

PROCESS ffffe000eb239080
SessionId: 0 Cid: 0180 Peb: 61467f1000 ParentCid: 0174
DirBase: 04466000 ObjectTable: ffffc000daca8040 HandleCount: <Data Not Accessible>
Image: csrss.exe
VadRoot ffffe000eb14ac00 Vads 90 Clone 0 Private 216. Modified 444. Locked 0.
DeviceMap ffffc000da21a760
Token ffffc000dacb0060
ElapsedTime 00:02:39.778
UserTime 00:00:00.000
KernelTime 00:00:00.093
QuotaPoolUsage[PagedPool] 149960
QuotaPoolUsage[NonPagedPool] 12696
Working Set Sizes (now,min,max) (314, 50, 345) (1256KB, 200KB, 1380KB)
PeakWorkingSetSize 985
VirtualSize 2097199 Mb
PeakVirtualSize 2097200 Mb
PageFaultCount 2633
MemoryPriority BACKGROUND
BasePriority 13
CommitCharge 323

PEB at 00000061467f1000
InheritedAddressSpace: No
ReadImageFileExecOptions: No
BeingDebugged: No
ImageBaseAddress: 00007ff71e540000
Ldr 00007ff8ed365200
Ldr.Initialized: Yes
Ldr.InInitializationOrderModuleList: 0000022132102ee0 . 000002213211b4e0
Ldr.InLoadOrderModuleList: 0000022132103050 . 000002213211b4c0
Ldr.InMemoryOrderModuleList: 0000022132103060 . 000002213211b4d0
Base TimeStamp Module
7ff71e540000 5632d16d Oct 30 02:09:49 2015 C:\Windows\system32\csrss.exe
7ff8ed220000 56a8483f Jan 27 04:31:59 2016 C:\Windows\SYSTEM32\ntdll.dll
7ff8e9820000 5632d16f Oct 30 02:09:51 2015 C:\Windows\system32\CSRSRV.dll
7ff8e9800000 5632d166 Oct 30 02:09:42 2015 C:\Windows\system32\basesrv.DLL
7ff8e97c0000 5632d722 Oct 30 02:34:10 2015 C:\Windows\system32\winsrv.DLL
7ff8eb3e0000 565423d2 Nov 24 08:46:10 2015 C:\Windows\system32\USER32.dll
7ff8e9c80000 56a8489c Jan 27 04:33:32 2016 C:\Windows\system32\kernelbase.dll
7ff8eb0b0000 5632d5aa Oct 30 02:27:54 2015 C:\Windows\system32\kernel32.dll
7ff8ed090000 568b2035 Jan 05 01:45:25 2016 C:\Windows\system32\GDI32.dll
7ff8e97b0000 5632d888 Oct 30 02:40:08 2015 C:\Windows\system32\sxssrv.DLL
27
 7ff8e9670000 5632d5f0 Oct 30 02:29:04 2015 C:\Windows\system32\sxs.dll
7ff8ea890000 5632d515 Oct 30 02:25:25 2015 C:\Windows\system32\RPCRT4.dll
7ff8e9bb0000 5632d756 Oct 30 02:35:02 2015 C:\Windows\system32\bcryptPrimitives.dll
SubSystemData: 0000000000000000
ProcessHeap: 00000221320e0000
ProcessParameters: 0000022132102550
CurrentDirectory: 'C:\Windows\system32\'
WindowTitle: '< Name not readable >'
ImageFile: 'C:\Windows\system32\csrss.exe'
CommandLine: '%SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,20480,768
Windows=On SubSystemType=Windows ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3
ServerDll=sxssrv,4 ProfileControl=Off MaxRequestThreads=16'
DllPath: '< Name not readable >'
Environment: 0000022132102080
ComSpec=C:\Windows\system32\cmd.exe
NUMBER_OF_PROCESSORS=4
OS=Windows_NT

Path=C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
PROCESSOR_ARCHITECTURE=AMD64
PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 58 Stepping 9, GenuineIntel
PROCESSOR_LEVEL=6
PROCESSOR_REVISION=3a09

PSModulePath=%ProgramFiles%\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules
SystemDrive=C:
SystemRoot=C:\Windows
TEMP=C:\Windows\TEMP
TMP=C:\Windows\TEMP
USERNAME=SYSTEM
windir=C:\Windows

THREAD ffffe000eb23f080 Cid 0180.0190 Teb: 00000061467f8000 Win32Thread: ffffe000eb75e260 WAIT:

(WrLpcReceive) UserMode Non-Alertable
ffffe000eb23f6b8 Semaphore Limit 0x1
Not impersonating
DeviceMap ffffc000da21a760
Owning Process ffffe000eb239080 Image: csrss.exe
Attached Process N/A Image: N/A
Wait Start TickCount 10750 Ticks: 31 (0:00:00:00.484)
Context Switch Count 467 IdealProcessor: 3
UserTime 00:00:00.015
KernelTime 00:00:00.000
Win32 Start Address CSRSRV!CsrApiRequestThread (0x00007ff8e9825380)
Stack Init ffffd00024a14c90 Current ffffd00024a14410
Base ffffd00024a15000 Limit ffffd00024a0f000 Call 0000000000000000
Priority 14 BasePriority 13 PriorityDecrement 0 IoPriority 2 PagePriority 5
*** ERROR: Module load completed but symbols could not be loaded for myfault.sys
Child-SP RetAddr Call Site
ffffd000`24a14450 fffff801`487003ea nt!KiSwapContext+0x76
ffffd000`24a14590 fffff801`486ffe79 nt!KiSwapThread+0x15a
ffffd000`24a14640 fffff801`486ffae5 nt!KiCommitThreadWait+0x149
ffffd000`24a146d0 fffff801`487585e6 nt!KeWaitForSingleObject+0x375
ffffd000`24a14790 fffff801`48a7a4aa nt!AlpcpWaitForSingleObject+0x3e
ffffd000`24a147d0 fffff801`48a77b32 nt!AlpcpReceiveMessagePort+0x45a
ffffd000`24a14860 fffff801`48a776b3 nt!AlpcpReceiveMessage+0x322
ffffd000`24a149d0 fffff801`487d6ca3 nt!NtAlpcSendWaitReceivePort+0x103
ffffd000`24a14a90 00007ff8`ed2c6174 nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @
ffffd000`24a14b00)
00000061`465df7d8 00007ff8`e9825602 ntdll!NtAlpcSendWaitReceivePort+0x14
00000061`465df7e0 00007ff8`ed27c585 CSRSRV!CsrApiRequestThread+0x282
00000061`465dfc70 00000000`00000000 ntdll!RtlUserThreadStart+0x45

28
 THREAD ffffe000eb74d080 Cid 0180.01b0 Teb: 00000061467fc000 Win32Thread: ffffe000ebf95c60 WAIT:
(WrLpcReply) UserMode Non-Alertable
ffffe000eb74d6b8 Semaphore Limit 0x1
Waiting for reply to ALPC Message ffffc000dae5fb30 : queued at port ffffe000eb73a800 : owned by
process ffffe000eb83e840
Not impersonating
DeviceMap ffffc000da21a760
Owning Process ffffe000eb239080 Image: csrss.exe
Attached Process N/A Image: N/A
Wait Start TickCount 2203 Ticks: 8578 (0:00:02:14.031)
Context Switch Count 7 IdealProcessor: 1
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address winsrv!TerminalServerRequestThread (0x00007ff8e97c1320)
Stack Init ffffd000250bcc90 Current ffffd000250bc3f0
Base ffffd000250bd000 Limit ffffd000250b7000 Call 0000000000000000
Priority 15 BasePriority 15 PriorityDecrement 0 IoPriority 2 PagePriority 5
Kernel stack not resident.

THREAD ffffe000eb74e080 Cid 0180.01b4 Teb: 00000061467fe000 Win32Thread: ffffe000eb971090 WAIT:

(UserRequest) UserMode Alertable
ffffe000eb245c00 SynchronizationEvent
ffffe000eb245d00 SynchronizationEvent
ffffe000eb245c80 SynchronizationEvent
ffffe000eb245b80 SynchronizationEvent
Not impersonating
DeviceMap ffffc000da21a760
Owning Process ffffe000eb239080 Image: csrss.exe
Attached Process N/A Image: N/A
Wait Start TickCount 845 Ticks: 9936 (0:00:02:35.250)
Context Switch Count 2 IdealProcessor: 2
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address winsrv!NotificationThread (0x00007ff8e97c2150)
Stack Init ffffd0002531ac90 Current ffffd00025319f80
Base ffffd0002531b000 Limit ffffd00025315000 Call 0000000000000000
Priority 14 BasePriority 13 PriorityDecrement 0 IoPriority 2 PagePriority 5
Kernel stack not resident.

THREAD ffffe000eb752840 Cid 0180.01b8 Teb: 0000006146600000 Win32Thread: ffffe000eb76fa90 WAIT:

(WrQueue) UserMode Alertable
ffffe000eb23cac0 QueueObject
Not impersonating
DeviceMap ffffc000da21a760
Owning Process ffffe000eb239080 Image: csrss.exe
Attached Process N/A Image: N/A
Wait Start TickCount 2196 Ticks: 8585 (0:00:02:14.140)
Context Switch Count 40 IdealProcessor: 3
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address ntdll!TppWorkerThread (0x00007ff8ed24b290)
Stack Init ffffd000251a7c90 Current ffffd000251a73e0
Base ffffd000251a8000 Limit ffffd000251a2000 Call 0000000000000000
Priority 15 BasePriority 13 PriorityDecrement 0 IoPriority 2 PagePriority 5
Kernel stack not resident.

THREAD ffffe000eb75a080 Cid 0180.01bc Teb: 0000006146602000 Win32Thread: 0000000000000000 WAIT:

(WrLpcReceive) UserMode Non-Alertable
ffffe000eb75a6b8 Semaphore Limit 0x1
Not impersonating
DeviceMap ffffc000da21a760
Owning Process ffffe000eb239080 Image: csrss.exe
Attached Process N/A Image: N/A
Wait Start TickCount 570 Ticks: 10211 (0:00:02:39.546)
Context Switch Count 3 IdealProcessor: 0
UserTime 00:00:00.000
KernelTime 00:00:00.000

29
 Win32 Start Address CSRSRV!CsrSbApiRequestThread (0x00007ff8e9824ed0)
Stack Init ffffd00025331c90 Current ffffd00025331490
Base ffffd00025332000 Limit ffffd0002532c000 Call 0000000000000000
Priority 14 BasePriority 13 PriorityDecrement 0 IoPriority 2 PagePriority 5
Kernel stack not resident.

THREAD ffffe000eb78a080 Cid 0180.01fc Teb: 0000006146604000 Win32Thread: ffffe000eb7df420 WAIT:

(WrLpcReceive) UserMode Non-Alertable
ffffe000eb78a6b8 Semaphore Limit 0x1
Not impersonating
DeviceMap ffffc000da21a760
Owning Process ffffe000eb239080 Image: csrss.exe
Attached Process N/A Image: N/A
Wait Start TickCount 10750 Ticks: 31 (0:00:00:00.484)
Context Switch Count 515 IdealProcessor: 1
UserTime 00:00:00.046
KernelTime 00:00:00.093
Win32 Start Address CSRSRV!CsrApiRequestThread (0x00007ff8e9825380)
Stack Init ffffd00024ac6c90 Current ffffd00024ac6410
Base ffffd00024ac7000 Limit ffffd00024ac1000 Call 0000000000000000
Priority 14 BasePriority 13 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
ffffd000`24ac6450 fffff801`487003ea nt!KiSwapContext+0x76
ffffd000`24ac6590 fffff801`486ffe79 nt!KiSwapThread+0x15a
ffffd000`24ac6640 fffff801`486ffae5 nt!KiCommitThreadWait+0x149
ffffd000`24ac66d0 fffff801`487585e6 nt!KeWaitForSingleObject+0x375
ffffd000`24ac6790 fffff801`48a7a4aa nt!AlpcpWaitForSingleObject+0x3e
ffffd000`24ac67d0 fffff801`48a77b32 nt!AlpcpReceiveMessagePort+0x45a
ffffd000`24ac6860 fffff801`48a776b3 nt!AlpcpReceiveMessage+0x322
ffffd000`24ac69d0 fffff801`487d6ca3 nt!NtAlpcSendWaitReceivePort+0x103
ffffd000`24ac6a90 00007ff8`ed2c6174 nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @
ffffd000`24ac6b00)
00000061`4697f5d8 00007ff8`e9825602 ntdll!NtAlpcSendWaitReceivePort+0x14
00000061`4697f5e0 00007ff8`ed27c585 CSRSRV!CsrApiRequestThread+0x282
00000061`4697fa70 00000000`00000000 ntdll!RtlUserThreadStart+0x45

THREAD ffffe000eb7cf080 Cid 0180.023c Teb: 0000006146606000 Win32Thread: ffffe000eb22a7c0 WAIT:

(WrUserRequest) KernelMode Alertable
ffffe000eb7b2610 SynchronizationEvent
ffffe000eb7c5870 NotificationTimer
ffffe000eb7b5af0 SynchronizationTimer
fffff80148965dc0 NotificationEvent
ffffe000eb737fe0 SynchronizationEvent
ffffe000eb737f60 SynchronizationEvent
ffffe000eb73cab0 SynchronizationEvent
ffffe000eb737ba0 SynchronizationEvent
ffffe000eb737aa0 SynchronizationEvent
ffffe000eb737a20 SynchronizationEvent
ffffe000eb7379a0 SynchronizationEvent
ffffe000eb737800 SynchronizationTimer
ffffe000eb737660 SynchronizationTimer
ffffe000eb7375e0 SynchronizationEvent
ffffe000eb737560 SynchronizationEvent
ffffe000eb7374e0 SynchronizationEvent
Not impersonating
DeviceMap ffffc000da21a760
Owning Process ffffe000eb239080 Image: csrss.exe
Attached Process N/A Image: N/A
Wait Start TickCount 10718 Ticks: 63 (0:00:00:00.984)
Context Switch Count 17 IdealProcessor: 2
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address winsrv!StartCreateSystemThreads (0x00007ff8e97c5680)
Stack Init ffffd00024f7ac90 Current ffffd00024f7a5e0
Base ffffd00024f7b000 Limit ffffd00024f75000 Call 0000000000000000
Priority 16 BasePriority 16 PriorityDecrement 0 IoPriority 2 PagePriority 5

30
 Child-SP RetAddr Call Site
ffffd000`24f7a620 fffff801`487003ea nt!KiSwapContext+0x76
ffffd000`24f7a760 fffff801`486ffe79 nt!KiSwapThread+0x15a
ffffd000`24f7a810 fffff801`48701a1e nt!KiCommitThreadWait+0x149
ffffd000`24f7a8a0 fffff961`7f61947a nt!KeWaitForMultipleObjects+0x24e
ffffd000`24f7a960 fffff961`7f9f3010 win32kfull!RawInputThread+0x9aa
ffffd000`24f7aa90 fffff961`7f62a83d win32kbase!xxxCreateSystemThreads+0x70
ffffd000`24f7aad0 fffff801`487d6ca3 win32kfull!NtUserCallNoParam+0x2d
ffffd000`24f7ab00 00007ff8`e97c7274 nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @
ffffd000`24f7ab00)
00000061`469bfed8 00000000`00000000 winsrv!NtUserCallNoParam+0x14

THREAD ffffe000eb7d0080 Cid 0180.0240 Teb: 0000006146608000 Win32Thread: ffffe000eb7b3260 WAIT:

(WrUserRequest) UserMode Non-Alertable
ffffe000eb70b360 SynchronizationEvent
ffffe000eae0f1e0 SynchronizationEvent
ffffe000eb226570 SynchronizationEvent
ffffe000eb7387e0 SynchronizationEvent
ffffe000eb738760 SynchronizationEvent
ffffe000eb7385c0 SynchronizationTimer
ffffe000eb7340d0 SynchronizationEvent
ffffe000eb7383a0 SynchronizationEvent
ffffe000eb737060 SynchronizationEvent
ffffe000eb7ce340 SynchronizationEvent
Not impersonating
DeviceMap ffffc000da21a760
Owning Process ffffe000eb239080 Image: csrss.exe
Attached Process N/A Image: N/A
Wait Start TickCount 2110 Ticks: 8671 (0:00:02:15.484)
Context Switch Count 31 IdealProcessor: 3
UserTime 00:00:00.000
KernelTime 00:00:00.015
Win32 Start Address winsrv!StartCreateSystemThreads (0x00007ff8e97c5680)
Stack Init ffffd00025006c90 Current ffffd00025006550
Base ffffd00025007000 Limit ffffd00025001000 Call 0000000000000000
Priority 15 BasePriority 13 PriorityDecrement 0 IoPriority 2 PagePriority 5
Kernel stack not resident.

THREAD ffffe000eb8d0080 Cid 0180.03a0 Teb: 000000614660a000 Win32Thread: ffffe000eabce820 WAIT:

(WrUserRequest) UserMode Non-Alertable
ffffe000eb8cd640 SynchronizationEvent
Not impersonating
DeviceMap ffffc000da21a760
Owning Process ffffe000eb239080 Image: csrss.exe
Attached Process N/A Image: N/A
Wait Start TickCount 819 Ticks: 9962 (0:00:02:35.656)
Context Switch Count 4 IdealProcessor: 0
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address winsrv!StartCreateSystemThreads (0x00007ff8e97c5680)
Stack Init ffffd000250e2c90 Current ffffd000250e2550
Base ffffd000250e3000 Limit ffffd000250dd000 Call 0000000000000000
Priority 15 BasePriority 13 PriorityDecrement 0 IoPriority 2 PagePriority 5
Kernel stack not resident.

THREAD ffffe000eb94a3c0 Cid 0180.040c Teb: 000000614660c000 Win32Thread: ffffe000eb7db0e0 WAIT:

(WrLpcReceive) UserMode Non-Alertable
ffffe000eb94a9f8 Semaphore Limit 0x1
Not impersonating
DeviceMap ffffc000da21a760
Owning Process ffffe000eb239080 Image: csrss.exe
Attached Process N/A Image: N/A
Wait Start TickCount 10750 Ticks: 31 (0:00:00:00.484)
Context Switch Count 384 IdealProcessor: 1
UserTime 00:00:00.015
KernelTime 00:00:00.093
Win32 Start Address CSRSRV!CsrApiRequestThread (0x00007ff8e9825380)

31
 Stack Init ffffd0002538bc90 Current ffffd0002538b410
Base ffffd0002538c000 Limit ffffd00025386000 Call 0000000000000000
Priority 14 BasePriority 13 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
ffffd000`2538b450 fffff801`487003ea nt!KiSwapContext+0x76
ffffd000`2538b590 fffff801`486ffe79 nt!KiSwapThread+0x15a
ffffd000`2538b640 fffff801`486ffae5 nt!KiCommitThreadWait+0x149
ffffd000`2538b6d0 fffff801`487585e6 nt!KeWaitForSingleObject+0x375
ffffd000`2538b790 fffff801`48a7a4aa nt!AlpcpWaitForSingleObject+0x3e
ffffd000`2538b7d0 fffff801`48a77b32 nt!AlpcpReceiveMessagePort+0x45a
ffffd000`2538b860 fffff801`48a776b3 nt!AlpcpReceiveMessage+0x322
ffffd000`2538b9d0 fffff801`487d6ca3 nt!NtAlpcSendWaitReceivePort+0x103
ffffd000`2538ba90 00007ff8`ed2c6174 nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @
ffffd000`2538bb00)
00000061`46a7f358 00007ff8`e9825602 ntdll!NtAlpcSendWaitReceivePort+0x14
00000061`46a7f360 00007ff8`ed27c585 CSRSRV!CsrApiRequestThread+0x282
00000061`46a7f7f0 00000000`00000000 ntdll!RtlUserThreadStart+0x45

THREAD ffffe000eba97080 Cid 0180.0788 Teb: 000000614660e000 Win32Thread: ffffe000eba78c50 WAIT:

(WrLpcReceive) UserMode Non-Alertable
ffffe000eba976b8 Semaphore Limit 0x1
Not impersonating
DeviceMap ffffc000da21a760
Owning Process ffffe000eb239080 Image: csrss.exe
Attached Process N/A Image: N/A
Wait Start TickCount 10750 Ticks: 31 (0:00:00:00.484)
Context Switch Count 311 IdealProcessor: 2
UserTime 00:00:00.078
KernelTime 00:00:00.046
Win32 Start Address CSRSRV!CsrApiRequestThread (0x00007ff8e9825380)
Stack Init ffffd00025b99c90 Current ffffd00025b99410
Base ffffd00025b9a000 Limit ffffd00025b94000 Call 0000000000000000
Priority 14 BasePriority 13 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
ffffd000`25b99450 fffff801`487003ea nt!KiSwapContext+0x76
ffffd000`25b99590 fffff801`486ffe79 nt!KiSwapThread+0x15a
ffffd000`25b99640 fffff801`486ffae5 nt!KiCommitThreadWait+0x149
ffffd000`25b996d0 fffff801`487585e6 nt!KeWaitForSingleObject+0x375
ffffd000`25b99790 fffff801`48a7a4aa nt!AlpcpWaitForSingleObject+0x3e
ffffd000`25b997d0 fffff801`48a77b32 nt!AlpcpReceiveMessagePort+0x45a
ffffd000`25b99860 fffff801`48a776b3 nt!AlpcpReceiveMessage+0x322
ffffd000`25b999d0 fffff801`487d6ca3 nt!NtAlpcSendWaitReceivePort+0x103
ffffd000`25b99a90 00007ff8`ed2c6174 nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @
ffffd000`25b99b00)
00000061`46abf998 00007ff8`e9825602 ntdll!NtAlpcSendWaitReceivePort+0x14
00000061`46abf9a0 00007ff8`ed27c585 CSRSRV!CsrApiRequestThread+0x282
00000061`46abfe30 00000000`00000000 ntdll!RtlUserThreadStart+0x45

Note: We see that the current process has changed. We specified 3f flags to have the process context changed to
that of csrss.exe during the execution of !process command. We also notice passive threads waiting for ALPC
notification, for example, ffffe000eb23f080 (weakly coupled processes) and ffffe000eb74d080 thread waiting for
ALPC request reply from svchost.exe process (strongly coupled processes):

3: kd> !alpc /m ffffc000dae5fb30

Message ffffc000dae5fb30
MessageID : 0x0068 (104)
CallbackID : 0x0267 (615)
SequenceNumber : 0x00000003 (3)
Type : LPC_REQUEST
DataLength : 0x4048 (16456)
TotalLength : 0x4070 (16496)
Canceled : No
Release : No

32
 ReplyWaitReply : No
Continuation : Yes
OwnerPort : ffffe000eb884610 [ALPC_CLIENT_COMMUNICATION_PORT]
WaitingThread : ffffe000eb74d080
QueueType : ALPC_MSGQUEUE_PENDING
QueuePort : ffffe000eb73a800 [ALPC_CONNECTION_PORT]
QueuePortOwnerProcess : ffffe000eb83e840 (svchost.exe)
ServerThread : ffffe000ebda8300
QuotaCharged : Yes
CancelQueuePort : 0000000000000000
CancelSequencePort : 0000000000000000
CancelSequenceNumber : 0x00000000 (0)
ClientContext : 0000000000000000
ServerContext : 0000000000000000
PortContext : 000001eaa7f10bd0
CancelPortContext : 0000000000000000
SecurityData : 0000000000000000
View : 0000000000000000
HandleData : 0000000000000000

3: kd> !thread ffffe000ebda8300 3f

THREAD ffffe000ebda8300 Cid 02b4.0c24 Teb: 000000016cc14000 Win32Thread: 0000000000000000 WAIT:
(WrQueue) UserMode Alertable
ffffe000eb840d40 QueueObject
Not impersonating
DeviceMap ffffc000da21a760
Owning Process ffffe000eb83e840 Image: svchost.exe
Attached Process N/A Image: N/A
Wait Start TickCount 10745 Ticks: 36 (0:00:00:00.562)
Context Switch Count 1832 IdealProcessor: 0
UserTime 00:00:00.046
KernelTime 00:00:00.046
Win32 Start Address ntdll!TppWorkerThread (0x00007ff8ed24b290)
Stack Init ffffd00026ca8c90 Current ffffd00026ca83e0
Base ffffd00026ca9000 Limit ffffd00026ca3000 Call 0000000000000000
Priority 13 BasePriority 8 PriorityDecrement 80 IoPriority 2 PagePriority 5

Child-SP RetAddr Call Site

ffffd000`26ca8420 fffff801`487003ea nt!KiSwapContext+0x76
ffffd000`26ca8560 fffff801`486ffe79 nt!KiSwapThread+0x15a
ffffd000`26ca8610 fffff801`487025ea nt!KiCommitThreadWait+0x149
ffffd000`26ca86a0 fffff801`487021ba nt!KeRemoveQueueEx+0x22a
ffffd000`26ca8740 fffff801`48702e6b nt!IoRemoveIoCompletion+0x8a
ffffd000`26ca8850 fffff801`487d6ca3 nt!NtWaitForWorkViaWorkerFactory+0x30b
ffffd000`26ca8a90 00007ff8`ed2c8794 nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ ffffd000`26ca8b00)
00000001`6e67f898 00007ff8`ed24b528 ntdll!NtWaitForWorkViaWorkerFactory+0x14
00000001`6e67f8a0 00007ff8`eb0c8102 ntdll!TppWorkerThread+0x298
00000001`6e67fcb0 00007ff8`ed27c574 KERNEL32!BaseThreadInitThunk+0x22
00000001`6e67fce0 00000000`00000000 ntdll!RtlUserThreadStart+0x34

Note: ALPC wait chains in csrss.exe are normal and expected.

We can get the list of ALPC receiver threads and threads waiting for reply using Microsoft MEX Debugging Extension:

https://www.microsoft.com/en-us/download/details.aspx?id=53304

After downloading, extracting, and unzipping, we copy \x64\mex.dll to WinDbg installation folder (For example,
C:\Program Files (x86)\Windows Kits\10\Debuggers\x64).

3: kd> .load mex

Mex External 3.0.0.7172 Loaded!

33
3: kd> !help
Mex currently has 255 extensions available. Please specify a keyword to search.
Or browse by category:

All PowerShell[6] SystemCenter[3] Networking[12] Process[5] Mex[2] Kernel[27] DotNet[32] Decompile[15] Utility[40] Thread[27] Binaries[6] General[22]

3: kd> !mex.help -all

[...]

3: kd> !mex.wrlpcreceive
Process PID Thread Id CSwitches User Kernel State Time Reason Wait Function
===================== === ================ ==== ========= ==== ====== ======= ========= ============ ========================================
System 4 ffffe000e9cf1040 114 46 0 0 Waiting 35s.703 WrLpcReceive nt!AlpcpSignalAndWait+0x1d9
csrss.exe 180 ffffe000eb23f080 190 467 16ms 0 Waiting 484ms WrLpcReceive CSRSRV!CsrApiRequestThread+0x282
csrss.exe 180 ffffe000eb75a080 1bc 3 0 0 Waiting 2m:39.546 WrLpcReceive Kernel stack not resident
csrss.exe 180 ffffe000eb78a080 1fc 515 47ms 94ms Waiting 484ms WrLpcReceive CSRSRV!CsrApiRequestThread+0x282
csrss.exe 180 ffffe000eb94a3c0 40c 384 16ms 94ms Waiting 484ms WrLpcReceive CSRSRV!CsrApiRequestThread+0x282
csrss.exe 180 ffffe000eba97080 788 311 78ms 47ms Waiting 484ms WrLpcReceive CSRSRV!CsrApiRequestThread+0x282
csrss.exe 1d0 ffffe000eb76a080 1e0 365 47ms 125ms Waiting 578ms WrLpcReceive CSRSRV!CsrApiRequestThread+0x282
csrss.exe 1d0 ffffe000eb7a2080 218 3 0 0 Waiting 2m:39.453 WrLpcReceive Kernel stack not resident
csrss.exe 1d0 ffffe000eb7c5080 230 374 31ms 109ms Waiting 578ms WrLpcReceive CSRSRV!CsrApiRequestThread+0x282
csrss.exe 1d0 ffffe000eb8863c0 328 2 0 0 Waiting 2m:32.312 WrLpcReceive Kernel stack not resident
csrss.exe 1d0 ffffe000eb8a7080 35c 336 47ms 94ms Waiting 578ms WrLpcReceive CSRSRV!CsrApiRequestThread+0 x282
csrss.exe 1d0 ffffe000ebfda840 123c 184 31ms 16ms Waiting 578ms WrLpcReceive CSRSRV!CsrApiRequestThread+0x282
csrss.exe 1d0 ffffe000ebfd8840 1240 173 16ms 31ms Waiting 281ms WrLpcReceive CSRSRV!CsrApiRequestThread+0x282
lsass.exe 25c ffffe000eb7f8080 26c 2 0 0 Waiting 2m:12.750 WrLpcReceive nt!AlpcpReceiveMessagePort+0x45a
svchost.exe (netsvcs) 388 ffffe000eb9e9340 4cc 178 0 16ms Waiting 578ms WrLpcReceive themeservice!CAPIConnection::Listen+0x8b
svchost.exe 484 ffffe000eba3a780 538 182 0 16ms Waiting 49s.203 WrLpcReceive nt!AlpcpReceiveMessagePort+0x45a
taskhostw.exe ac4 ffffe000ebd86080 974 371 16ms 31ms Waiting 62ms WrLpcReceive MSCTF!CCtfServerPort::ServerLoop+0x18a
Count: 17

0: kd> !mex.wrlpcreply
Process PID Thread Id CSwitches User Kernel State Time Reason Waiting On
Wait Function
============================ === ================ ==== ========= ==== ====== ======= ========= ==========
============================================================ =========================
csrss.exe 180 ffffe000eb74d080 1b0 7 0 0 Waiting 2m:14.031 WrLpcReply Thread: ffffe000ebda8300 in svchost.exe (DcomLaunch)
(0n692) Kernel stack not resident
csrss.exe 1d0 ffffe000eb79e080 20c 276 0 0 Waiting 2m:14.031 WrLpcReply Thread: ffffe000ebda8300 in svchost.exe (DcomLaunch)
(0n692) Kernel stack not resident
svchost.exe (netsvcs) 388 ffffe000ebe2e080 cb8 3 0 0 Waiting 2m:12.750 WrLpcReply Thread: ffffe000ebc143c0 in svchost.exe (0n1012)
svchost.exe 3f4 ffffe000e9054600 b54 2 0 0 Waiting 2m:12.734 WrLpcReply Thread: ffffe000eb8863c0 in csrss.exe (0n464)
svchost.exe (NetworkService) 4dc ffffe000e9278540 b78 2 0 0 Waiting 2m:12.734 WrLpcReply Thread: ffffe000eb8ed040 in svchost.exe (0n1012)
explorer.exe c64 ffffe000ec19d080 1764 10 0 0 Waiting 1m:16.484 WrLpcReply Message queued to ShellExpe rienceHost.exe (0n3484)

Note: MEX command changed the current CPU from 3 to 0.

8. Now we list processes and threads from the session 1:

0: kd> !sprocess 1 3f
Dumping Session 1

_MM_SESSION_SPACE ffffd000251ac000
_MMSESSION ffffd000251acb40
PROCESS ffffe000eb21d840
SessionId: 1 Cid: 01d0 Peb: 27d00b2000 ParentCid: 01c0
DirBase: 2685f000 ObjectTable: ffffc000dad6fac0 HandleCount: <Data Not Accessible>
Image: csrss.exe
VadRoot ffffe000eb79ed60 Vads 80 Clone 0 Private 212. Modified 2761. Locked 0.
DeviceMap ffffc000da21a760
Token ffffc000dad84b30
ElapsedTime 00:02:39.544
UserTime 00:00:00.000
KernelTime 00:00:00.078
QuotaPoolUsage[PagedPool] 148992
QuotaPoolUsage[NonPagedPool] 16200
Working Set Sizes (now,min,max) (548, 50, 345) (2192KB, 200KB, 1380KB)
PeakWorkingSetSize 2499
VirtualSize 2097199 Mb
PeakVirtualSize 2097208 Mb
PageFaultCount 6214
MemoryPriority BACKGROUND
BasePriority 13
CommitCharge 344

PEB at 00000027d00b2000
34
 InheritedAddressSpace: No
ReadImageFileExecOptions: No
BeingDebugged: No
ImageBaseAddress: 00007ff71e540000
Ldr 00007ff8ed365200
Ldr.Initialized: Yes
Ldr.InInitializationOrderModuleList: 0000018bfb602ee0 . 0000018bfb626870
Ldr.InLoadOrderModuleList: 0000018bfb603050 . 0000018bfb626850
Ldr.InMemoryOrderModuleList: 0000018bfb603060 . 0000018bfb626860
Base TimeStamp Module
7ff71e540000 5632d16d Oct 30 02:09:49 2015 C:\Windows\system32\csrss.exe
7ff8ed220000 56a8483f Jan 27 04:31:59 2016 C:\Windows\SYSTEM32\ntdll.dll
7ff8e9820000 5632d16f Oct 30 02:09:51 2015 C:\Windows\system32\CSRSRV.dll
7ff8e9800000 5632d166 Oct 30 02:09:42 2015 C:\Windows\system32\basesrv.DLL
7ff8e97c0000 5632d722 Oct 30 02:34:10 2015 C:\Windows\system32\winsrv.DLL
7ff8eb3e0000 565423d2 Nov 24 08:46:10 2015 C:\Windows\system32\USER32.dll
7ff8e9c80000 56a8489c Jan 27 04:33:32 2016 C:\Windows\system32\kernelbase.dll
7ff8eb0b0000 5632d5aa Oct 30 02:27:54 2015 C:\Windows\system32\kernel32.dll
7ff8ed090000 568b2035 Jan 05 01:45:25 2016 C:\Windows\system32\GDI32.dll
7ff8e97b0000 5632d888 Oct 30 02:40:08 2015 C:\Windows\system32\sxssrv.DLL
7ff8e9670000 5632d5f0 Oct 30 02:29:04 2015 C:\Windows\system32\sxs.dll
7ff8ea890000 5632d515 Oct 30 02:25:25 2015 C:\Windows\system32\RPCRT4.dll
7ff8e9bb0000 5632d756 Oct 30 02:35:02 2015 C:\Windows\system32\bcryptPrimitives.dll
SubSystemData: 0000000000000000
ProcessHeap: 0000018bfb500000
ProcessParameters: 0000018bfb602550
CurrentDirectory: 'C:\Windows\system32\'
WindowTitle: '< Name not readable >'
ImageFile: 'C:\Windows\system32\csrss.exe'
CommandLine: '%SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,20480,768
Windows=On SubSystemType=Windows ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3
ServerDll=sxssrv,4 ProfileControl=Off MaxRequestThreads=16'
DllPath: '< Name not readable >'
Environment: 0000018bfb602080
ComSpec=C:\Windows\system32\cmd.exe
NUMBER_OF_PROCESSORS=4
OS=Windows_NT

Path=C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
PROCESSOR_ARCHITECTURE=AMD64
PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 58 Stepping 9, GenuineIntel
PROCESSOR_LEVEL=6
PROCESSOR_REVISION=3a09

PSModulePath=%ProgramFiles%\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules
SystemDrive=C:
SystemRoot=C:\Windows
TEMP=C:\Windows\TEMP
TMP=C:\Windows\TEMP
USERNAME=SYSTEM
windir=C:\Windows

THREAD ffffe000eb76a080 Cid 01d0.01e0 Teb: 00000027d00b9000 Win32Thread: ffffe000eaec6ef0 WAIT:

(WrLpcReceive) UserMode Non-Alertable
ffffe000eb76a6b8 Semaphore Limit 0x1
Not impersonating
DeviceMap ffffc000da21a760
Owning Process ffffe000eb21d840 Image: csrss.exe
Attached Process N/A Image: N/A
Wait Start TickCount 10744 Ticks: 37 (0:00:00:00.578)
Context Switch Count 365 IdealProcessor: 0
UserTime 00:00:00.046
KernelTime 00:00:00.125
Win32 Start Address CSRSRV!CsrApiRequestThread (0x00007ff8e9825380)
Stack Init ffffd00025576c90 Current ffffd00025576410

35
 Base ffffd00025577000 Limit ffffd00025571000 Call 0000000000000000
Priority 14 BasePriority 13 PriorityDecrement 0 IoPriority 2 PagePriority 5

Child-SP RetAddr Call Site

ffffd000`25576450 fffff801`487003ea nt!KiSwapContext+0x76
ffffd000`25576590 fffff801`486ffe79 nt!KiSwapThread+0x15a
ffffd000`25576640 fffff801`486ffae5 nt!KiCommitThreadWait+0x149
ffffd000`255766d0 fffff801`487585e6 nt!KeWaitForSingleObject+0x375
ffffd000`25576790 fffff801`48a7a4aa nt!AlpcpWaitForSingleObject+0x3e
ffffd000`255767d0 fffff801`48a77b32 nt!AlpcpReceiveMessagePort+0x45a
ffffd000`25576860 fffff801`48a776b3 nt!AlpcpReceiveMessage+0x322
ffffd000`255769d0 fffff801`487d6ca3 nt!NtAlpcSendWaitReceivePort+0x103
ffffd000`25576a90 00007ff8`ed2c6174 nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @
ffffd000`25576b00)
00000027`d027f858 00007ff8`e9825602 ntdll!NtAlpcSendWaitReceivePort+0x14
00000027`d027f860 00007ff8`ed27c585 CSRSRV!CsrApiRequestThread+0x282
00000027`d027fcf0 00000000`00000000 ntdll!RtlUserThreadStart+0x45

[...]

PROCESS ffffe000ec373080
SessionId: 1 Cid: 12b0 Peb: 00516000 ParentCid: 0c64
DirBase: 27369000 ObjectTable: ffffc000dd3bc840 HandleCount: <Data Not Accessible>
Image: OneDrive.exe
VadRoot ffffe000ec3e5b00 Vads 168 Clone 0 Private 918. Modified 1753. Locked 0.
DeviceMap ffffc000db94eec0
Token ffffc000dccf26c0
ElapsedTime 00:02:08.766
UserTime 00:00:00.000
KernelTime 00:00:00.031
QuotaPoolUsage[PagedPool] 255608
QuotaPoolUsage[NonPagedPool] 23256
Working Set Sizes (now,min,max) (713, 50, 345) (2852KB, 200KB, 1380KB)
PeakWorkingSetSize 4842
VirtualSize 134 Mb
PeakVirtualSize 139 Mb
PageFaultCount 6191
MemoryPriority BACKGROUND
BasePriority 8
CommitCharge 1235

PEB at 0000000000516000
error 1 InitTypeRead( nt!_PEB at 0000000000516000)...

THREAD ffffe000ec367080 Cid 12b0.12b4 Teb: 0000000000518000 Win32Thread: ffffe000ec2d44a0 WAIT:

(WrUserRequest) UserMode Non-Alertable
ffffe000ec5941e0 SynchronizationEvent
Not impersonating
DeviceMap ffffc000db94eec0
Owning Process ffffe000ec373080 Image: OneDrive.exe
Attached Process N/A Image: N/A
Wait Start TickCount 7705 Ticks: 3076 (0:00:00:48.062)
Context Switch Count 215 IdealProcessor: 3
UserTime 00:00:00.062
KernelTime 00:00:00.078
Win32 Start Address 0x000000000037e2c6
Stack Init ffffd00027df8c90 Current ffffd00027df8480
Base ffffd00027df9000 Limit ffffd00027df3000 Call 0000000000000000
Priority 10 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
ffffd000`27df84c0 fffff801`487003ea nt!KiSwapContext+0x76
ffffd000`27df8600 fffff801`486ffe79 nt!KiSwapThread+0x15a
ffffd000`27df86b0 fffff801`486ffae5 nt!KiCommitThreadWait+0x149
ffffd000`27df8740 fffff961`7f6de5c5 nt!KeWaitForSingleObject+0x375
ffffd000`27df8800 fffff961`7f6de1c8 win32kfull!xxxRealSleepThread+0x355
ffffd000`27df88f0 fffff961`7f6dcd9d win32kfull!xxxSleepThread2+0x98

36
 ffffd000`27df8940 fffff961`7f6dc1e0 win32kfull!xxxRealInternalGetMessage+0xb4d
ffffd000`27df8a70 fffff801`487d6ca3 win32kfull!NtUserGetMessage+0x90
ffffd000`27df8b00 00000000`6c393824 nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @
ffffd000`27df8b00)
00000000`0008e398 00000000`00000000 0x6c393824

THREAD ffffe000ec5ef080 Cid 12b0.12c4 Teb: 0000000000524000 Win32Thread: 0000000000000000 WAIT:

(UserRequest) UserMode Non-Alertable
ffffe000ec4c2350 SynchronizationEvent
Not impersonating
DeviceMap ffffc000db94eec0
Owning Process ffffe000ec373080 Image: OneDrive.exe
Attached Process N/A Image: N/A
Wait Start TickCount 2556 Ticks: 8225 (0:00:02:08.515)
Context Switch Count 1 IdealProcessor: 3
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address 0x00000000777be7f0
Stack Init ffffd000270bcc90 Current ffffd000270bc710
Base ffffd000270bd000 Limit ffffd000270b7000 Call 0000000000000000
Priority 10 BasePriority 10 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
ffffd000`270bc750 fffff801`487003ea nt!KiSwapContext+0x76
ffffd000`270bc890 fffff801`486ffe79 nt!KiSwapThread+0x15a
ffffd000`270bc940 fffff801`486ffae5 nt!KiCommitThreadWait+0x149
ffffd000`270bc9d0 fffff801`48a9f032 nt!KeWaitForSingleObject+0x375
ffffd000`270bca90 fffff801`487d6ca3 nt!NtWaitForSingleObject+0xf2
ffffd000`270bcb00 00000000`6c4021bc nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @
ffffd000`270bcb00)
00000000`00aef018 00000000`00000000 0x6c4021bc

THREAD ffffe000ec487840 Cid 12b0.12c8 Teb: 0000000000527000 Win32Thread: 0000000000000000 WAIT:

(WrQueue) UserMode Alertable
ffffe000ec59a8c0 QueueObject
Not impersonating
DeviceMap ffffc000db94eec0
Owning Process ffffe000ec373080 Image: OneDrive.exe
Attached Process N/A Image: N/A
Wait Start TickCount 2571 Ticks: 8210 (0:00:02:08.281)
Context Switch Count 37 IdealProcessor: 0
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address 0x000000007777c6d0
Stack Init ffffd000274cfc90 Current ffffd000274cf3e0
Base ffffd000274d0000 Limit ffffd000274ca000 Call 0000000000000000
Priority 9 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
ffffd000`274cf420 fffff801`487003ea nt!KiSwapContext+0x76
ffffd000`274cf560 fffff801`486ffe79 nt!KiSwapThread+0x15a
ffffd000`274cf610 fffff801`487025ea nt!KiCommitThreadWait+0x149
ffffd000`274cf6a0 fffff801`487021ba nt!KeRemoveQueueEx+0x22a
ffffd000`274cf740 fffff801`48702e6b nt!IoRemoveIoCompletion+0x8a
ffffd000`274cf850 fffff801`487d6ca3 nt!NtWaitForWorkViaWorkerFactory+0x30b
ffffd000`274cfa90 00007ff8`ed2c8794 nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @
ffffd000`274cfb00)
00000000`00b3e628 00000000`00000000 0x00007ff8`ed2c8794

THREAD ffffe000ec5de080 Cid 12b0.12cc Teb: 000000000052a000 Win32Thread: ffffe000ec26aba0 WAIT:

(WrQueue) UserMode Alertable
ffffe000ec59a8c0 QueueObject
Not impersonating
DeviceMap ffffc000db94eec0
Owning Process ffffe000ec373080 Image: OneDrive.exe
Attached Process N/A Image: N/A
Wait Start TickCount 10254 Ticks: 527 (0:00:00:08.234)
Context Switch Count 76 IdealProcessor: 1
UserTime 00:00:00.031

37
 KernelTime 00:00:00.015
Win32 Start Address 0x000000007777c6d0
Stack Init ffffd000273a2c90 Current ffffd000273a23e0
Base ffffd000273a3000 Limit ffffd0002739d000 Call 0000000000000000
Priority 8 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
ffffd000`273a2420 fffff801`487003ea nt!KiSwapContext+0x76
ffffd000`273a2560 fffff801`486ffe79 nt!KiSwapThread+0x15a
ffffd000`273a2610 fffff801`487025ea nt!KiCommitThreadWait+0x149
ffffd000`273a26a0 fffff801`487021ba nt!KeRemoveQueueEx+0x22a
ffffd000`273a2740 fffff801`48702e6b nt!IoRemoveIoCompletion+0x8a
ffffd000`273a2850 fffff801`487d6ca3 nt!NtWaitForWorkViaWorkerFactory+0x30b
ffffd000`273a2a90 00007ff8`ed2c8794 nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @
ffffd000`273a2b00)
00000000`00b7ea38 00000000`6c33686e 0x00007ff8`ed2c8794
00000000`00b7ea40 00000000`6c403500 0x6c33686e
00000000`00b7ea48 00000000`00000001 0x6c403500
00000000`00b7ea50 00002326`00000100 0x1
00000000`00b7ea58 00000000`0298fbc8 0x00002326`00000100
00000000`00b7ea60 00000000`00b7eaa0 0x298fbc8
00000000`00b7ea68 00000000`0298faa4 0xb7eaa0
00000000`00b7ea70 00000000`00b7ea40 0x298faa4
00000000`00b7ea78 00000000`6c334185 0xb7ea40
00000000`00b7ea80 00000000`00b7eaa0 0x6c334185
00000000`00b7ea88 00000000`0000003c 0xb7eaa0
00000000`00b7ea90 00000000`007422d8 0x3c
00000000`00b7ea98 00000000`0298fdd8 0x7422d8
00000000`00b7eaa0 00000000`00000000 0x298fdd8

THREAD ffffe000ec5cf080 Cid 12b0.12d4 Teb: 0000000000530000 Win32Thread: 0000000000000000 WAIT:

(WrQueue) UserMode Non-Alertable
ffffe000ec2ce080 QueueObject
Not impersonating
DeviceMap ffffc000db94eec0
Owning Process ffffe000ec373080 Image: OneDrive.exe
Attached Process N/A Image: N/A
Wait Start TickCount 2563 Ticks: 8218 (0:00:02:08.406)
Context Switch Count 1 IdealProcessor: 3
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address 0x0000000071304ccc
Stack Init ffffd0002556bc90 Current ffffd0002556b560
Base ffffd0002556c000 Limit ffffd00025566000 Call 0000000000000000
Priority 8 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
ffffd000`2556b5a0 fffff801`487003ea nt!KiSwapContext+0x76
ffffd000`2556b6e0 fffff801`486ffe79 nt!KiSwapThread+0x15a
ffffd000`2556b790 fffff801`487025ea nt!KiCommitThreadWait+0x149
ffffd000`2556b820 fffff801`487021ba nt!KeRemoveQueueEx+0x22a
ffffd000`2556b8c0 fffff801`48af2964 nt!IoRemoveIoCompletion+0x8a
ffffd000`2556b9d0 fffff801`487d6ca3 nt!NtRemoveIoCompletion+0x134
ffffd000`2556ba90 00000000`6c4021bc nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @
ffffd000`2556bb00)
00000000`035dec78 00000000`00000000 0x6c4021bc

THREAD ffffe000ec5eb080 Cid 12b0.12d8 Teb: 0000000000533000 Win32Thread: 0000000000000000 WAIT:

(WrQueue) UserMode Non-Alertable
ffffe000ec2ce080 QueueObject
Not impersonating
DeviceMap ffffc000db94eec0
Owning Process ffffe000ec373080 Image: OneDrive.exe
Attached Process N/A Image: N/A
Wait Start TickCount 2563 Ticks: 8218 (0:00:02:08.406)
Context Switch Count 1 IdealProcessor: 0
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address 0x0000000071304ccc

38
 Stack Init ffffd000268c7c90 Current ffffd000268c7560
Base ffffd000268c8000 Limit ffffd000268c2000 Call 0000000000000000
Priority 8 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
ffffd000`268c75a0 fffff801`487003ea nt!KiSwapContext+0x76
ffffd000`268c76e0 fffff801`486ffe79 nt!KiSwapThread+0x15a
ffffd000`268c7790 fffff801`487025ea nt!KiCommitThreadWait+0x149
ffffd000`268c7820 fffff801`487021ba nt!KeRemoveQueueEx+0x22a
ffffd000`268c78c0 fffff801`48af2964 nt!IoRemoveIoCompletion+0x8a
ffffd000`268c79d0 fffff801`487d6ca3 nt!NtRemoveIoCompletion+0x134
ffffd000`268c7a90 00000000`6c4021bc nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @
ffffd000`268c7b00)
00000000`0371ebe8 00000000`00000000 0x6c4021bc

THREAD ffffe000ec441840 Cid 12b0.12dc Teb: 0000000000536000 Win32Thread: 0000000000000000 WAIT:

(UserRequest) UserMode Non-Alertable
ffffe000ec361580 NotificationEvent
Not impersonating
DeviceMap ffffc000db94eec0
Owning Process ffffe000ec373080 Image: OneDrive.exe
Attached Process N/A Image: N/A
Wait Start TickCount 2563 Ticks: 8218 (0:00:02:08.406)
Context Switch Count 1 IdealProcessor: 1
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address 0x00000000719bbfb4
Stack Init ffffd000279e6c90 Current ffffd000279e6710
Base ffffd000279e7000 Limit ffffd000279e1000 Call 0000000000000000
Priority 8 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
ffffd000`279e6750 fffff801`487003ea nt!KiSwapContext+0x76
ffffd000`279e6890 fffff801`486ffe79 nt!KiSwapThread+0x15a
ffffd000`279e6940 fffff801`486ffae5 nt!KiCommitThreadWait+0x149
ffffd000`279e69d0 fffff801`48a9f032 nt!KeWaitForSingleObject+0x375
ffffd000`279e6a90 fffff801`487d6ca3 nt!NtWaitForSingleObject+0xf2
ffffd000`279e6b00 00000000`6c4021bc nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @
ffffd000`279e6b00)
00000000`0385edf8 00000000`00000000 0x6c4021bc

THREAD ffffe000ec5c2840 Cid 12b0.12e0 Teb: 0000000000539000 Win32Thread: 0000000000000000 WAIT:

(UserRequest) UserMode Non-Alertable
ffffe000ea90f760 SynchronizationEvent
Not impersonating
DeviceMap ffffc000db94eec0
Owning Process ffffe000ec373080 Image: OneDrive.exe
Attached Process N/A Image: N/A
Wait Start TickCount 10254 Ticks: 527 (0:00:00:08.234)
Context Switch Count 8 IdealProcessor: 2
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address 0x00000000719bbfb4
Stack Init ffffd000255a9c90 Current ffffd000255a9710
Base ffffd000255aa000 Limit ffffd000255a4000 Call 0000000000000000
Priority 9 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
ffffd000`255a9750 fffff801`487003ea nt!KiSwapContext+0x76
ffffd000`255a9890 fffff801`486ffe79 nt!KiSwapThread+0x15a
ffffd000`255a9940 fffff801`486ffae5 nt!KiCommitThreadWait+0x149
ffffd000`255a99d0 fffff801`48a9f032 nt!KeWaitForSingleObject+0x375
ffffd000`255a9a90 fffff801`487d6ca3 nt!NtWaitForSingleObject+0xf2
ffffd000`255a9b00 00000000`6c4021bc nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @
ffffd000`255a9b00)
00000000`0399ed08 00000000`6c402021 0x6c4021bc
00000000`0399ed10 00000023`777c854c 0x6c402021
00000000`0399ed18 00000000`00000023 0x00000023`777c854c
00000000`0399ed20 00000000`ee556126 0x23
00000000`0399ed28 00000000`03a9f37c 0xee556126

39
 00000000`0399ed30 00000000`0399ed58 0x3a9f37c
00000000`0399ed38 00000000`006e1b88 0x399ed58
00000000`0399ed40 00000000`0000006e 0x6e1b88
00000000`0399ed48 00000000`00000000 0x6e

THREAD ffffe000ec5ab080 Cid 12b0.12e4 Teb: 000000000053c000 Win32Thread: ffffe000ec268970 WAIT:

(UserRequest) UserMode Non-Alertable
ffffe000ec4b2480 NotificationEvent
ffffe000eb793be0 SynchronizationEvent
Not impersonating
DeviceMap ffffc000db94eec0
Owning Process ffffe000ec373080 Image: OneDrive.exe
Attached Process N/A Image: N/A
Wait Start TickCount 2572 Ticks: 8209 (0:00:02:08.265)
Context Switch Count 5 IdealProcessor: 3
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address 0x00000000719bbfb4
Stack Init ffffd0002763cc90 Current ffffd0002763bf80
Base ffffd0002763d000 Limit ffffd00027637000 Call 0000000000000000
Priority 9 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
ffffd000`2763bfc0 fffff801`487003ea nt!KiSwapContext+0x76
ffffd000`2763c100 fffff801`486ffe79 nt!KiSwapThread+0x15a
ffffd000`2763c1b0 fffff801`48701a1e nt!KiCommitThreadWait+0x149
ffffd000`2763c240 fffff801`48a9c21d nt!KeWaitForMultipleObjects+0x24e
ffffd000`2763c300 fffff801`48af40a7 nt!ObWaitForMultipleObjects+0x2bd
ffffd000`2763c810 fffff801`487d6ca3 nt!NtWaitForMultipleObjects32+0xf7
ffffd000`2763ca90 00000000`6c4021bc nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @
ffffd000`2763cb00)
00000000`03adec48 00000000`00000000 0x6c4021bc

THREAD ffffe000ec59b840 Cid 12b0.12e8 Teb: 000000000053f000 Win32Thread: ffffe000ec15fb00 WAIT:

(UserRequest) UserMode Non-Alertable
ffffe000ebc4afa0 NotificationEvent
ffffe000ec2e4db0 SynchronizationEvent
Not impersonating
DeviceMap ffffc000db94eec0
Owning Process ffffe000ec373080 Image: OneDrive.exe
Attached Process N/A Image: N/A
Wait Start TickCount 2572 Ticks: 8209 (0:00:02:08.265)
Context Switch Count 20 IdealProcessor: 0
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address 0x00000000719bbfb4
Stack Init ffffd00027908c90 Current ffffd00027907f80
Base ffffd00027909000 Limit ffffd00027903000 Call 0000000000000000
Priority 8 BasePriority 6 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
ffffd000`27907fc0 fffff801`487003ea nt!KiSwapContext+0x76
ffffd000`27908100 fffff801`486ffe79 nt!KiSwapThread+0x15a
ffffd000`279081b0 fffff801`48701a1e nt!KiCommitThreadWait+0x149
ffffd000`27908240 fffff801`48a9c21d nt!KeWaitForMultipleObjects+0x24e
ffffd000`27908300 fffff801`48af40a7 nt!ObWaitForMultipleObjects+0x2bd
ffffd000`27908810 fffff801`487d6ca3 nt!NtWaitForMultipleObjects32+0xf7
ffffd000`27908a90 00000000`6c4021bc nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @
ffffd000`27908b00)
00000000`03c1ee68 00000000`00000000 0x6c4021bc

THREAD ffffe000ec59e840 Cid 12b0.12ec Teb: 0000000000542000 Win32Thread: ffffe000ec2385a0 WAIT:

(UserRequest) UserMode Non-Alertable
ffffe000ec282e20 NotificationEvent
ffffe000eba64ba0 SynchronizationEvent
Not impersonating
DeviceMap ffffc000db94eec0
Owning Process ffffe000ec373080 Image: OneDrive.exe
Attached Process N/A Image: N/A

40
 Wait Start TickCount 2568 Ticks: 8213 (0:00:02:08.328)
Context Switch Count 12 IdealProcessor: 1
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address 0x00000000719bbfb4
Stack Init ffffd00027422c90 Current ffffd00027421f80
Base ffffd00027423000 Limit ffffd0002741d000 Call 0000000000000000
Priority 8 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
ffffd000`27421fc0 fffff801`487003ea nt!KiSwapContext+0x76
ffffd000`27422100 fffff801`486ffe79 nt!KiSwapThread+0x15a
ffffd000`274221b0 fffff801`48701a1e nt!KiCommitThreadWait+0x149
ffffd000`27422240 fffff801`48a9c21d nt!KeWaitForMultipleObjects+0x24e
ffffd000`27422300 fffff801`48af40a7 nt!ObWaitForMultipleObjects+0x2bd
ffffd000`27422810 fffff801`487d6ca3 nt!NtWaitForMultipleObjects32+0xf7
ffffd000`27422a90 00000000`6c4021bc nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @
ffffd000`27422b00)
00000000`03d5f0a8 00000000`00000000 0x6c4021bc

THREAD ffffe000ec58b040 Cid 12b0.12f0 Teb: 0000000000545000 Win32Thread: 0000000000000000 WAIT:

(WrQueue) UserMode Alertable
ffffe000ec59a8c0 QueueObject
Not impersonating
DeviceMap ffffc000db94eec0
Owning Process ffffe000ec373080 Image: OneDrive.exe
Attached Process N/A Image: N/A
Wait Start TickCount 2569 Ticks: 8212 (0:00:02:08.312)
Context Switch Count 3 IdealProcessor: 2
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address 0x000000007777c6d0
Stack Init ffffd000276fbc90 Current ffffd000276fb3e0
Base ffffd000276fc000 Limit ffffd000276f6000 Call 0000000000000000
Priority 8 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
ffffd000`276fb420 fffff801`487003ea nt!KiSwapContext+0x76
ffffd000`276fb560 fffff801`486ffe79 nt!KiSwapThread+0x15a
ffffd000`276fb610 fffff801`487025ea nt!KiCommitThreadWait+0x149
ffffd000`276fb6a0 fffff801`487021ba nt!KeRemoveQueueEx+0x22a
ffffd000`276fb740 fffff801`48702e6b nt!IoRemoveIoCompletion+0x8a
ffffd000`276fb850 fffff801`487d6ca3 nt!NtWaitForWorkViaWorkerFactory+0x30b
ffffd000`276fba90 00007ff8`ed2c8794 nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @
ffffd000`276fbb00)
00000000`03e9e5c8 00000000`00000000 0x00007ff8`ed2c8794

THREAD ffffe000ec585840 Cid 12b0.12f4 Teb: 0000000000548000 Win32Thread: ffffe000ebb31ec0 WAIT:

(DelayExecution) UserMode Non-Alertable
ffffffffffffffff NotificationEvent
Not impersonating
DeviceMap ffffc000db94eec0
Owning Process ffffe000ec373080 Image: OneDrive.exe
Attached Process N/A Image: N/A
Wait Start TickCount 10251 Ticks: 530 (0:00:00:08.281)
Context Switch Count 27 IdealProcessor: 3
UserTime 00:00:00.015
KernelTime 00:00:00.000
Win32 Start Address 0x000000007777c6d0
Stack Init ffffd00027d5ac90 Current ffffd00027d5a790
Base ffffd00027d5b000 Limit ffffd00027d55000 Call 0000000000000000
Priority 9 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
ffffd000`27d5a7d0 fffff801`487003ea nt!KiSwapContext+0x76
ffffd000`27d5a910 fffff801`486ffe79 nt!KiSwapThread+0x15a
ffffd000`27d5a9c0 fffff801`486944ac nt!KiCommitThreadWait+0x149
ffffd000`27d5aa50 fffff801`48a5101c nt!KeDelayExecutionThread+0x28c
ffffd000`27d5aad0 fffff801`487d6ca3 nt!NtDelayExecution+0x5c

41
 ffffd000`27d5ab00 00000000`6c4021bc nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @
ffffd000`27d5ab00)
00000000`040decc8 00000000`6c40209d 0x6c4021bc
00000000`040decd0 00000023`777c6ccc 0x6c40209d
00000000`040decd8 00000000`00000023 0x00000023`777c6ccc
00000000`040dece0 00000000`007aee30 0x23
00000000`040dece8 00000000`041df314 0x7aee30
00000000`040decf0 00000000`040ded20 0x41df314
00000000`040decf8 00000000`000120bf 0x40ded20
00000000`040ded00 00000000`041dc5a8 0x120bf
00000000`040ded08 00000000`00000018 0x41dc5a8
00000000`040ded10 00000000`00784900 0x18
00000000`040ded18 00000000`00000004 0x784900
00000000`040ded20 00000000`041dc518 0x4
00000000`040ded28 00000000`041df300 0x41dc518
00000000`040ded30 00000000`041dc518 0x41df300
00000000`040ded38 00000000`040dede0 0x41dc518
00000000`040ded40 00000000`00000000 0x40dede0

[...]

Note: Incorrect and truncated stack traces with the presence of 32-bit return addresses may point to a virtualized
WOW64 process in case wow64* module information was paged out. Please see exercise Legacy.C1 for iexplore.exe
example.

We can double check the process bitness by using MEX extention tasklist command:

0: kd> !tasklist -s 1
PID Address Name Ses
============= ================ ============================= ===
0x1d0 0n464 ffffe000eb21d840 csrss.exe 1
0x21c 0n540 ffffe000eb7a52c0 winlogon.exe 1
0x354 0n852 ffffe000eb8a3080 dwm.exe 1
0xb0c 0n2828 ffffe000eb601080 sihost.exe 1
0xac4 0n2756 ffffe000ebd4e840 taskhostw.exe 1
0xc5c 0n3164 ffffe000ebdca840 RuntimeBroker.exe 1
0xc64 0n3172 ffffe000ebdc7840 explorer.exe 1
0xca4 0n3236 ffffe000ebdcd840 SkypeHost.exe*32 1
0xd9c 0n3484 ffffe000ebf0a840 ShellExperienceHost.exe 1
0xe58 0n3672 ffffe000ebf00840 SearchUI.exe 1
0xfbc 0n4028 ffffe000ec252080 TabTip.exe 1
0xff0 0n4080 ffffe000ec121080 TabTip32.exe*32 1
0x1228 0n4648 ffffe000eb6c3080 vmtoolsd.exe 1
0x12b0 0n4784 ffffe000ec373080 OneDrive.exe*32 1
0x1050 0n4176 ffffe000ec24a080 ApplicationFrameHost.exe 1
0x10f8 0n4344 ffffe000ec491080 MicrosoftEdge.exe 1
0x1208 0n4616 ffffe000ec220080 browser_broker.exe 1
0x1354 0n4948 ffffe000ec62c840 MicrosoftEdgeCP.exe 1
0x1378 0n4984 ffffe000ec6a8640 SearchProtocolHost.exe 1
0x105c 0n4188 ffffe000ec77d840 MicrosoftEdgeCP.exe 1
0x1430 0n5168 ffffe000ec88d840 MicrosoftEdgeCP.exe 1
0x14c0 0n5312 ffffe000ec944840 MicrosoftEdgeCP.exe 1
0x2c4 0n708 ffffe000ec156840 notepad.exe 1
0x3a8 0n936 ffffe000eca66840 svchost.exe(UnistackSvcGroup) 1
0x1594 0n5524 ffffe000ec09a080 NotMyfault.exe 1
============= ================ ============================= ===
PID Address Name Ses

Warning! Zombie process(es) detected (not displayed). Count: 2 [zombie report]

42
Note: For the complete list or tasklist command options, please use -? parameter.

9. Suppose, we are interested in the last OneDrive.exe thread ffffe000ec585840 (here we need /w switch):

0: kd> .load wow64exts

0: kd> .thread /w ffffe000ec585840

Implicit thread is now ffffe000`ec585840
WARNING: WOW context retrieval requires
switching to the thread's process context.
Use .process /p ffffe000`ec121080 to switch back.
Implicit process is now ffffe000`ec373080
x86 context set

0: kd:x86> k
*** Stack trace for last set context - .thread/.cxr resets it
# ChildEBP RetAddr
WARNING: Frame IP not in any known module. Following frames may be wrong.
00 041df7cc 748ea56f 0x777c6f3c
01 041df7dc 70700d58 0x748ea56f
02 041df804 7070124a 0x70700d58
03 041df848 7777933a 0x7070124a
04 041df8b4 7777929a 0x7777933a
05 041df8d4 7777cd32 0x7777929a
06 041dfa8c 75f538f4 0x7777cd32
07 041dfaa0 777b5e13 0x75f538f4
08 041dfae8 777b5dde 0x777b5e13
09 041dfaf8 00000000 0x777b5dde

0: kd:x86> .reload
Loading Kernel Symbols
...............................................................
................................................................
......................................
Loading User Symbols
PEB is paged out (Peb.Ldr = 00000000`00516018). Type ".hh dbgerr001" for details
Loading unloaded module list
.............Unable to enumerate user-mode unloaded modules, NTSTATUS 0xC0000147
Loading Wow64 Symbols
......................Unable to read NT module Base Name string at 00000000`006bb938 - NTSTATUS
0xC0000147
.Unable to read NT module Base Name string at 00000000`006d6d2c - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006c1460 - NTSTATUS 0xC0000147
....Unable to read NT module Base Name string at 00000000`006c1850 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006c13d0 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006c1a90 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006da118 - NTSTATUS 0xC0000147
Missing image name, possible paged-out or corrupt data.
.Unable to read NT module Base Name string at 00000000`00707204 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006e6518 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006ddd30 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`00707bbc - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`00707d54 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006ddc58 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006e6568 - NTSTATUS 0xC0000147
Missing image name, possible paged-out or corrupt data.
.Unable to read NT module Base Name string at 00000000`006ddca0 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006dd118 - NTSTATUS 0xC0000147
...Unable to read NT module Base Name string at 00000000`0070b68e - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`0070c17c - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006dd598 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006dd2c8 - NTSTATUS 0xC0000147
Missing image name, possible paged-out or corrupt data.
.Unable to read NT module Base Name string at 00000000`0070c680 - NTSTATUS 0xC0000147
Missing image name, possible paged-out or corrupt data.
.Unable to read NT module Base Name string at 00000000`006dd238 - NTSTATUS 0xC0000147
43
.Unable to read NT module Base Name string at 00000000`006df948 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006dd820 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006dd550 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006dd280 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`0070fa74 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006e6248 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006dd478 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006df3c8 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006e5f78 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006dd1a8 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006dd5e0 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006e5ed8 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006dd6b8 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006dd628 - NTSTATUS 0xC0000147
..
.Unable to read NT module Base Name string at 00000000`0070b574 - NTSTATUS 0xC0000147
Missing image name, possible paged-out or corrupt data.
.Unable to read NT module Base Name string at 00000000`0072ee18 - NTSTATUS 0xC0000147
..Unable to read NT module Base Name string at 00000000`0074bb98 - NTSTATUS 0xC0000147
....Unable to read NT module Base Name string at 00000000`00755830 - NTSTATUS 0xC0000147
Missing image name, possible paged-out or corrupt data.
........

************* Symbol Loading Error Summary **************

Module name Error
SharedUserData No error - symbol load deferred

You can troubleshoot most symbol related issues by turning on symbol loading diagnostics (!sym noisy) and
repeating the command that caused symbols to be loaded.
You should also verify that your symbol search path (.sympath) is correct.

0: kd:x86> k
*** Stack trace for last set context - .thread/.cxr resets it
# ChildEBP RetAddr
00 041df764 748ea619 ntdll!NtDelayExecution+0xc
01 041df7cc 748ea56f KERNELBASE!SleepEx+0x99
02 041df7dc 70700d58 KERNELBASE!Sleep+0xf
03 041df804 7070124a WINHTTP!SafeTerminateDll+0xa8
04 041df848 7777933a WINHTTP!FailFastThreadpoolWaitCallback<&SafeTerminateDll>+0x2a
05 041df8b4 7777929a ntdll!TppExecuteWaitCallback+0x7a
06 041df8d4 7777cd32 ntdll!TppWaitCompletion+0x8a
07 041dfa8c 75f538f4 ntdll!TppWorkerThread+0x662
08 041dfaa0 777b5e13 KERNEL32!BaseThreadInitThunk+0x24
09 041dfae8 777b5dde ntdll!__RtlUserThreadStart+0x2f
0a 041dfaf8 00000000 ntdll!_RtlUserThreadStart+0x1b

Note: To switch back to our native processor architecture we use .effmach or !sw commands:

0: kd:x86> .effmach AMD64

Effective machine: x64 (AMD64)

0: kd> .thread /w ffffe000ec585840

Implicit thread is now ffffe000`ec585840
x86 context set

0: kd:x86> !sw
Switched to Host mode

10. Another way to list all stack traces is to use !for_each_thread command where we can customize stack trace
output:
0: kd> !for_each_thread ".thread /r /p @#Thread; kv"
Implicit thread is now ffffe000`e9058600
Implicit process is now ffffe000`e9040700
Loading User Symbols

44
************* Symbol Loading Error Summary **************
Module name Error
SharedUserData No error - symbol load deferred

You can troubleshoot most symbol related issues by turning on symbol loading diagnostics (!sym noisy) and repeating the command that caused symbols to be
loaded.
You should also verify that your symbol search path (.sympath) is correct.
*** Stack trace for last set context - .thread/.cxr resets it
# Child-SP RetAddr : Args to Child : Call Site
00 ffffd000`23db2890 fffff801`487003ea : 08488b2f`00000000 00000000`00000001 f67501ee`83c80301 72c13bc1`0301b70f : nt!KiSwapContext+0x76
01 ffffd000`23db29d0 fffff801`486ffe79 : 458b0448`8bf4458b e58bfc45`8b088908 00000000`00000000 53f8458d`0cec83ec : nt!KiSwapThread+0x15a
02 ffffd000`23db2a80 fffff801`486ffae5 : 0173850f`c085c10b 24fe835c`ee830000 b70f0000`0167820f 00000000`00000000 : nt!KiCommitThreadWait+0x149
03 ffffd000`23db2b10 fffff801`487ba48e : fffff801`48964140 8b66fc45`00000000 75c33b66`02c18300 d118ee83`fc4d2b00 : nt!KeWaitForSingleObject +0x375
04 ffffd000`23db2bd0 fffff801`4876d5a5 : 3b0247b7`0f7a7504 404b88b9`4575f445 66108b66`06478d00 66267508`558b113b : nt!PopIrpWorkerControl+0x22
05 ffffd000`23db2c10 fffff801`487d1626 : ffffd000`28840180 ffffe000`e9058600 fffff801`4876d564 e8502847`8d5213eb : nt!PspSystemThreadStartup+0x41
06 ffffd000`23db2c60 00000000`00000000 : ffffd000`23db3000 ffffd000`23dad000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x16
Implicit thread is now ffffe000`e90cb040
Implicit process is now ffffe000`e9040700
Loading User Symbols

[...]

Implicit thread is now ffffe000`ec475080

Implicit process is now ffffe000`ec09a080
Loading User Symbols
...........................................

************* Symbol Loading Error Summary **************

Module name Error
SharedUserData No error - symbol load deferred
msrpc The system cannot find the file specified
vmci The system cannot find the file specified
vsock The system cannot find the file specified
vmhgfs Symbol loading cancelled
vmmemctl The system cannot find the file specified
myfault The system cannot find the file specified

You can troubleshoot most symbol related issues by turning on symbol loading diagnostics (!sym noisy) and repeating the command that caused symbols to be
loaded.
You should also verify that your symbol search path (.sympath) is correct.
*** Stack trace for last set context - .thread/.cxr resets it
# Child-SP RetAddr : Args to Child : Call Site
00 ffffd000`257fefc0 fffff801`487003ea : ffffe000`00000000 00000000`00000001 00000000`00000000 fffff801`00000000 : nt!KiSwapContext+0x76
01 ffffd000`257ff100 fffff801`486ffe79 : ffffc000`dd8e2ef0 fffff801`488ba2a1 00000000`00000000 fffff801`486fb0ae : nt!KiSwapThread+0x15a
02 ffffd000`257ff1b0 fffff801`48701a1e : ffffd000`00000000 ffffc000`dd9992f4 00000000`00000008 00000000`00000002 : nt!KiCommitThreadWait+0x149
03 ffffd000`257ff240 fffff801`48a9c21d : ffffd000`00000002 ffffd000`257ff3c0 00000000`00000000 ffffd000`00000006 : nt!KeWaitForMultipleObjects+0x24e
04 ffffd000`257ff300 fffff801`48af1c26 : fffff801`4899eb01 00000000`00000000 00000000`00000000 ffffd000`257ffad8 : nt!ObWaitForMultipleObjects+0x2bd
05 ffffd000`257ff810 fffff801`487d6ca3 : 00000000`00000000 ffffd000`00000000 ffffe000`ec475080 00000000`03a5f8f8 : nt!NtWaitForMultipleObject s+0xf6
06 ffffd000`257ffa90 00007ff8`ed2c5bd4 : 00007ff8`e9cc3b2f 00007ff8`ecf68210 00000000`00000002 00000000`10000010 : nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @
ffffd000`257ffb00)
07 00000000`03a5f8d8 00007ff8`e9cc3b2f : 00007ff8`ecf68210 00000000`00000002 00000000`10000010 00000000`00000000 : ntdll!NtWaitForMultipleObjects+0x14
08 00000000`03a5f8e0 00007ff8`eab1727f : 00000000`00000001 00007ff8`eac21148 00000000`00000001 00000000`0062d800 : KERNELBASE!WaitForMultipleObjectsEx+0xef
09 00000000`03a5fbe0 00007ff8`eab170e7 : 00000000`0062d800 00000000`00000000 00000000`00644f40 00000000`000017ac : combase!WaitCoalesced+0xb3
[d:\th\com\published\comutils\coalescedwait.cxx @ 72]
0a 00000000`03a5fe70 00007ff8`eab27c4c : 00000000`ffffffff 00000000`0062d800 00000000`00644f40 00000000`00000000 : combase!CRpcThread::WorkerLoop+0x11f
[d:\th\com\combase\dcomrem\threads.cxx @ 321]
0b 00000000`03a5fee0 00007ff8`eb0c8102 : 00007ff8`eab27bd0 00000000`00000000 00000000`00000000 00000000`00000000 :
combase!CRpcThreadCache::RpcWorkerThreadEntry+0x7c [d:\th\com\combase\dcomrem\threads.cxx @ 76]
0c 00000000`03a5ff10 00007ff8`ed27c574 : 00007ff8`eb0c80e0 00000000`00000000 00000000`00000000 00000000`00000000 : KERNEL32!BaseThreadInitThunk+0x22
0d 00000000`03a5ff40 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x34

Note: We can use this script to list all processes and threads including 32-bit stack traces when it is possible:

0: kd> !for_each_thread "!thread @#Thread 1f;.thread /w @#Thread; .reload; kb 256; .effmach AMD64"
!thread @#Thread 1f;.thread /w @#Thread; .reload; kb 256; .effmach AMD64
Setting context for owner process...
.process /p /r ffffe000e9040700

THREAD ffffe000e9058600 Cid 0004.000c Teb: 0000000000000000 Win32Thread: 0000000000000000 WAIT: (Executive)
KernelMode Non-Alertable
fffff80148964140 SynchronizationEvent
Not impersonating
DeviceMap ffffc000da21a760
Owning Process ffffe000e9040700 Image: System
Attached Process N/A Image: N/A
Wait Start TickCount 19 Ticks: 10762 (0:00:02:48.156)
Context Switch Count 1 IdealProcessor: 0
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address nt!PopIrpWorkerControl (0xfffff801487ba46c)
Stack Init ffffd00023db2c90 Current ffffd00023db2850
Base ffffd00023db3000 Limit ffffd00023dad000 Call 0000000000000000
Priority 15 BasePriority 13 PriorityDecrement 32 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
ffffd000`23db2890 fffff801`487003ea nt!KiSwapContext+0x76
ffffd000`23db29d0 fffff801`486ffe79 nt!KiSwapThread+0x15a
ffffd000`23db2a80 fffff801`486ffae5 nt!KiCommitThreadWait+0x149
ffffd000`23db2b10 fffff801`487ba48e nt!KeWaitForSingleObject+0x375
45
ffffd000`23db2bd0 fffff801`4876d5a5 nt!PopIrpWorkerControl+0x22
ffffd000`23db2c10 fffff801`487d1626 nt!PspSystemThreadStartup+0x41
ffffd000`23db2c60 00000000`00000000 nt!KiStartSystemThread+0x16

.process /p /r 0
Implicit thread is now ffffe000`e9058600
The context is partially valid. Only x86 user-mode context is available.
x86 context set
Loading Kernel Symbols
...............................................................
................................................................
......................................
Loading User Symbols

Loading unloaded module list

.............Unable to enumerate user-mode unloaded modules, NTSTATUS 0xC0000147

************* Symbol Loading Error Summary **************

Module name Error
SharedUserData No error - symbol load deferred

You can troubleshoot most symbol related issues by turning on symbol loading diagnostics (!sym noisy) and repeating
the command that caused symbols to be loaded.
You should also verify that your symbol search path (.sympath) is correct.
# ChildEBP RetAddr Args to Child
WARNING: Frame IP not in any known module. Following frames may be wrong.
00 00000000 00000000 00000000 00000000 00000000 0x487d0f16
Effective machine: x64 (AMD64)
!thread @#Thread 1f;.thread /w @#Thread; .reload; kb 256; .effmach AMD64
Setting context for owner process...
.process /p /r ffffe000e9040700

[...]

THREAD ffffe000ec5c2840 Cid 12b0.12e0 Teb: 0000000000539000 Win32Thread: 0000000000000000 WAIT: (UserRequest)
UserMode Non-Alertable
ffffe000ea90f760 SynchronizationEvent
Not impersonating
DeviceMap ffffc000db94eec0
Owning Process ffffe000ec373080 Image: OneDrive.exe
Attached Process N/A Image: N/A
Wait Start TickCount 10254 Ticks: 527 (0:00:00:08.234)
Context Switch Count 8 IdealProcessor: 2
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address 0x00000000719bbfb4
Stack Init ffffd000255a9c90 Current ffffd000255a9710
Base ffffd000255aa000 Limit ffffd000255a4000 Call 0000000000000000
Priority 9 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
ffffd000`255a9750 fffff801`487003ea nt!KiSwapContext+0x76
ffffd000`255a9890 fffff801`486ffe79 nt!KiSwapThread+0x15a
ffffd000`255a9940 fffff801`486ffae5 nt!KiCommitThreadWait+0x149
ffffd000`255a99d0 fffff801`48a9f032 nt!KeWaitForSingleObject+0x375
ffffd000`255a9a90 fffff801`487d6ca3 nt!NtWaitForSingleObject+0xf2
ffffd000`255a9b00 00000000`6c4021bc nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ ffffd000`255a9b00)
00000000`0399ed08 00000000`6c402021 0x6c4021bc
00000000`0399ed10 00000023`777c854c 0x6c402021
00000000`0399ed18 00000000`00000023 0x00000023`777c854c
00000000`0399ed20 00000000`ee556126 0x23
00000000`0399ed28 00000000`03a9f37c 0xee556126
00000000`0399ed30 00000000`0399ed58 0x3a9f37c
00000000`0399ed38 00000000`006e1b88 0x399ed58
00000000`0399ed40 00000000`0000006e 0x6e1b88
00000000`0399ed48 00000000`00000000 0x6e

.process /p /r 0
Implicit thread is now ffffe000`ec5c2840
WARNING: WOW context retrieval requires
switching to the thread's process context.
Use .process /p ffffe000`e9040700 to switch back.
Implicit process is now ffffe000`ec373080
x86 context set
Loading Kernel Symbols
...............................................................
46
................................................................
......................................
Loading User Symbols
PEB is paged out (Peb.Ldr = 00000000`00516018). Type ".hh dbgerr001" for details
Loading unloaded module list
.............Unable to enumerate user-mode unloaded modules, NTSTATUS 0xC0000147
Loading Wow64 Symbols
......................Unable to read NT module Base Name string at 00000000`006bb938 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006d6d2c - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006c1460 - NTSTATUS 0xC0000147
....Unable to read NT module Base Name string at 00000000`006c1850 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006c13d0 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006c1a90 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006da118 - NTSTATUS 0xC0000147
Missing image name, possible paged-out or corrupt data.
.Unable to read NT module Base Name string at 00000000`00707204 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006e6518 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006ddd30 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`00707bbc - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`00707d54 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006ddc58 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006e6568 - NTSTATUS 0xC0000147
Missing image name, possible paged-out or corrupt data.
.Unable to read NT module Base Name string at 00000000`006ddca0 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006dd118 - NTSTATUS 0xC0000147
...Unable to read NT module Base Name string at 00000000`0070b68e - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`0070c17c - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006dd598 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006dd2c8 - NTSTATUS 0xC0000147
Missing image name, possible paged-out or corrupt data.
.Unable to read NT module Base Name string at 00000000`0070c680 - NTSTATUS 0xC0000147
Missing image name, possible paged-out or corrupt data.
.Unable to read NT module Base Name string at 00000000`006dd238 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006df948 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006dd820 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006dd550 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006dd280 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`0070fa74 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006e6248 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006dd478 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006df3c8 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006e5f78 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006dd1a8 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006dd5e0 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006e5ed8 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006dd6b8 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`006dd628 - NTSTATUS 0xC0000147
..
.Unable to read NT module Base Name string at 00000000`0070b574 - NTSTATUS 0xC0000147
Missing image name, possible paged-out or corrupt data.
.Unable to read NT module Base Name string at 00000000`0072ee18 - NTSTATUS 0xC0000147
..Unable to read NT module Base Name string at 00000000`0074bb98 - NTSTATUS 0xC0000147
....Unable to read NT module Base Name string at 00000000`00755830 - NTSTATUS 0xC0000147
Missing image name, possible paged-out or corrupt data.
........

************* Symbol Loading Error Summary **************

Module name Error
SharedUserData No error - symbol load deferred

You can troubleshoot most symbol related issues by turning on symbol loading diagnostics (!sym noisy) and repeating
the command that caused symbols to be loaded.
You should also verify that your symbol search path (.sympath) is correct.
# ChildEBP RetAddr Args to Child
00 03a9f824 748de111 00000434 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
01 03a9f898 719fcba5 00000434 ffffffff 00000000 KERNELBASE!WaitForSingleObjectEx+0x91
02 03a9f8ac 719fb506 86add51e 007665f0 007665ec MSVCR120!Concurrency::details::ExternalContextBase::Block+0x37
[f:\dd\vctools\crt\crtw32\concrt\externalcontextbase.cpp @ 145]
03 03a9f918 7193ea79 00780c94 03fa25a8 007665e0 MSVCR120!Concurrency::details::_Condition_variable::wait+0xab
[f:\dd\vctools\crt\crtw32\concrt\event.cpp @ 595]
04 03a9f94c 7193eb58 007665ec 007665f0 00000000 MSVCP120!do_wait+0x42 [f:\dd\vctools\crt\crtw32\stdcpp\thr\cond.c @
56]
*** ERROR: Symbol file could not be found. Defaulted to export symbols for SyncEngine.DLL -
05 03a9f960 712fdaa8 007665ec 007665f0 6038d983 MSVCP120!_Cnd_wait+0x10 [f:\dd\vctools\crt\crtw32\stdcpp\thr\cond.c @
81]
WARNING: Stack unwind information not available. Following frames may be wrong.
47
06 03a9f994 71144017 00007530 00007530 007664c8 SyncEngine!QoS::ScenarioQosWrapper::GetApiId+0x1d999d
07 03a9f9ec 71145105 00007530 71375228 00007530 SyncEngine!QoS::ScenarioQosWrapper::GetApiId+0x1ff0c
08 03a9fa0c 711769ec 00007530 719bbfb4 71077444 SyncEngine!QoS::ScenarioQosWrapper::GetApiId+0x20ffa
09 03a9fa34 7117669b 03a9fa5b 6038da7b 719bbfb4 SyncEngine!QoS::ScenarioQosWrapper::GetApiId+0x528e1
0a 03a9fa6c 719bc01d 00000000 86add702 719bbfb4 SyncEngine!QoS::ScenarioQosWrapper::GetApiId+0x52590
0b 03a9faa4 719bc001 719bbfb4 03a9fac4 75f538f4 MSVCR120!_callthreadstartex+0x1b
[f:\dd\vctools\crt\crtw32\startup\threadex.c @ 376]
0c 03a9fab0 75f538f4 00786200 75f538d0 63db2846 MSVCR120!_threadstartex+0x7c
[f:\dd\vctools\crt\crtw32\startup\threadex.c @ 354]
0d 03a9fac4 777b5e13 00786200 4f093fe7 00000000 KERNEL32!BaseThreadInitThunk+0x24
0e 03a9fb0c 777b5dde ffffffff 777db7e8 00000000 ntdll!__RtlUserThreadStart+0x2f
0f 03a9fb1c 00000000 719bbfb4 00786200 00000000 ntdll!_RtlUserThreadStart+0x1b
Effective machine: x64 (AMD64)
!thread @#Thread 1f;.thread /w @#Thread; .reload; kb 256; .effmach AMD64
Setting context for owner process...
.process /p /r ffffe000ec373080

[...]

11. Yet another way is to use !stacks command (the default version omits paged out stacks):

0: kd> !stacks
Proc.Thread .Thread Ticks ThreadState Blocker
[fffff80148a1ca40 Idle]
0.000000 fffff80148a1d740 ffffd5e3 RUNNING hal!HalProcessorIdle+0xf
0.000000 ffffd00023f1abc0 0000000 RUNNING hal!HalProcessorIdle+0xf
0.000000 ffffd00023f99bc0 0000000 RUNNING hal!HalProcessorIdle+0xf
0.000000 ffffd0002884cbc0 0000000 RUNNING nt!KiIdleLoop+0x11d
[ffffe000e9040700 System]
4.000018 ffffe000e90f0040 ffffffed Blocked nt!PopFxEmergencyWorker+0x29
4.000020 ffffe000e90fa040 fffff7ca Blocked nt!KeRemovePriQueue+0x1b7
4.000024 ffffe000e90f6040 ffffd602 Blocked nt!ExpWorkerFactoryManagerThread+0x28
4.00003c ffffe000e90f7040 ffffd604 Blocked nt!MiModifiedPageWriter+0x3c2
4.000048 ffffe000e9150040 ffffd5f5 Blocked nt!MiZeroPageThread+0x752
4.000050 ffffe000e9172040 ffffd661 Blocked nt!CcQueueLazyWriteScanThread+0x96
4.000054 ffffe000e9176040 ffffe1f7 Blocked nt!CcAsyncReadWorker+0x231
4.000058 ffffe000e9175040 ffffffe8 Blocked nt!CcAsyncReadWorker+0x231
4.00005c ffffe000e9174040 ffffffe8 Blocked nt!CcAsyncReadWorker+0x231
4.000068 ffffe000e9177040 fffff106 Blocked nt!KeRemovePriQueue+0x1b7
4.000070 ffffe000e92c5040 fffffc46 Blocked nt!EtwpLogger+0xcb
4.000074 ffffe000e92e6040 ffffd657 Blocked nt!EtwpLogger+0xcb
4.000078 ffffe000e92ef040 ffffd707 Blocked nt!EtwpLogger+0xcb
4.00007c ffffe000e92f0040 ffffd759 Blocked nt!EtwpLogger+0xcb
4.000080 ffffe000e9306240 ffffd897 Blocked nt!EtwpLogger+0xcb
4.000084 ffffe000e9327840 ffffd6d6 Blocked nt!EtwpLogger+0xcb
4.000088 ffffe000e93d6040 ffffffe2 Blocked nt!EtwpLogger+0xcb
4.00008c ffffe000e93da440 ffffe310 Blocked nt!EtwpLogger+0xcb
4.000090 ffffe000e93dd040 ffffe552 Blocked nt!EtwpLogger+0xcb
4.000094 ffffe000e93fe040 ffffe23f Blocked nt!EtwpLogger+0xcb
4.000098 ffffe000e93ff040 ffffffe2 Blocked nt!EtwpLogger+0xcb
4.00009c ffffe000e9ca3500 ffffe1ed Blocked +0xffffe000e9d885c9
4.0000a0 ffffe000e9072040 ffffdec0 Blocked nt!KeRemovePriQueue+0x1b7
4.0000a4 ffffe000e905a040 ffffffe1 Blocked nt!IopPassiveInterruptRealtimeWorker+0x16
4.0000a8 ffffe000e9cb0040 ffffffe1 Blocked nt!IopPassiveInterruptRealtimeWorker+0x16
4.0000ac ffffe000e92a2040 ffffffe1 Blocked nt!IopPassiveInterruptRealtimeWorker+0x16
4.0000b0 ffffe000e92a1040 ffffffe1 Blocked nt!IopPassiveInterruptRealtimeWorker+0x16
4.0000b8 ffffe000e9d17040 fffffd16 Blocked ACPI!ACPIWorkerThread+0x74
4.0000bc ffffe000ea92b040 fffff613 Blocked nt!KeRemovePriQueue+0x1b7
4.0000c0 ffffe000ea90e040 ffffdec0 Blocked nt!KeRemovePriQueue+0x1b7
4.0000c4 ffffe000ea939040 ffffff66 Blocked pci!RootPmeEventDispatcher+0x8b
4.0000c8 ffffe000ea92e040 ffffff66 Blocked ACPI!PciRootBusBiosMethodDispatcherOnResume+0x51
*** ERROR: Module load completed but symbols could not be loaded for vmci.sys
4.0000cc ffffe000ea9d5040 ffffd6e9 Blocked vmci+0x8110
*** ERROR: Module load completed but symbols could not be loaded for vsock.sys
4.0000d0 ffffe000ea9d7380 fffff5ef Blocked vsock+0x4387
4.0000d4 ffffe000eaafb840 ffffd602 Blocked WdFilter!MpAsyncpWorkerThread+0x13d
4.0000d8 ffffe000eab33340 ffffde7d Blocked ndis!ndisThreadPoolTimerHandler+0x1f
4.0000e0 ffffe000eab36840 fffffe63 Blocked ndis!ndisReceiveWorkerThread+0xb7
48
 4.0000e4 ffffe000eab37040 fffffe63 Blocked ndis!ndisReceiveWorkerThread+0xb7
4.0000e8 ffffe000eab38040 fffffe63 Blocked ndis!ndisReceiveWorkerThread+0xb7
4.0000ec ffffe000eab39040 fffffe63 Blocked ndis!ndisReceiveWorkerThread+0xb7
4.000108 ffffe000e9ca7040 fffffe50 Blocked watchdog!SMgrGdiCalloutThread+0x43
*** ERROR: Module load completed but symbols could not be loaded for vmhgfs.sys
4.00010c ffffe000e9ca8040 fffffe4c Blocked vmhgfs+0xd394
4.000110 ffffe000e9cc1040 fffffe4c Blocked vmhgfs+0xd394
4.000114 ffffe000e9cf1040 ffffded0 Blocked nt!AlpcpSignalAndWait+0x1d9
4.00012c ffffe000e9cbe840 fffffe43 Blocked dxgkrnl!DpiPowerArbiterThread+0x67
4.00014c ffffe000eae13080 ffffd705 Blocked nt!CmpLazyWriteWorker+0x3a
4.000150 ffffe000eae14080 ffffd8b8 Blocked nt!CmpLazyWriteWorker+0x3a
4.00015c ffffe000eae5e5c0 ffffdec0 Blocked nt!KeRemovePriQueue+0x1b7
4.000160 ffffe000eae62040 ffffd5ff Blocked nt!KeRemovePriQueue+0x1b7
4.000164 ffffe000eae664c0 fffff106 Blocked nt!KeRemovePriQueue+0x1b7
4.000168 ffffe000eae68040 ffffdec0 Blocked nt!KeRemovePriQueue+0x1b7
4.00016c ffffe000eae6a040 ffffdec0 Blocked nt!KeRemovePriQueue+0x1b7
4.00019c ffffe000e92ea080 ffffd5e5 Blocked dxgmms1!VidSchiWaitForSchedulerEvents+0x1cc
4.0001a0 ffffe000e9cee080 fffff9ce Blocked dxgkrnl!BLTQUEUE::BltQueueWorker+0x1ae
4.0001a4 ffffe000eb1db080 ffffd6db Blocked BasicRender!WARPKMADAPTER::RunGPU+0x14d
4.0001a8 ffffe000eb1f8840 ffffd73b Blocked dxgmms2!VidSchiWaitForSchedulerEvents+0x21e
4.0001ac ffffe000eb70b840 ffffd795 Blocked dxgmms2!VIDMM_WORKER_THREAD::Run+0x117
*** WARNING: Unable to verify timestamp for msrpc.sys
*** ERROR: Module load completed but symbols could not be loaded for msrpc.sys
4.000204 ffffe000eb78e080 ffffd608 Blocked nt!IoRemoveIoCompletion+0x8a
4.000294 ffffe000eb934840 ffffd6d6 Blocked luafv!SynchronousFsControl+0x175
4.0002cc ffffe000eb93c040 fffffd75 Blocked storqosflt!SqosJobDispatcherThreadRoutine+0x51
4.0005f0 ffffe000e9097040 fffffd4c Blocked HTTP!UlpScavengerThread+0xfc
4.000674 ffffe000eb65c840 fffffb1a Blocked mpsdrv!NseQueryExportTable+0x124
4.0006bc ffffe000eb69b080 fffffd2f Blocked nt!EtwpLogger+0xcb
*** ERROR: Module load completed but symbols could not be loaded for vmmemctl.sys
4.0006d4 ffffe000eb6d5840 ffffd622 Blocked vmmemctl+0x22ea
4.000714 ffffe000eb6fd040 fffffd25 Blocked Ndu!NduTokenComputeTokensWorkerRoutine+0x7a
4.000728 ffffe000eba63040 ffffd5e8 Blocked mmcss!CiSchedulerThreadFunction+0x5a7
4.0007d4 ffffe000ebae3280 ffffd606 Blocked nt!SmKmStoreHelperWorker+0x46
4.0007d8 ffffe000ebae9840 ffffd606 Blocked nt!SmKmStoreHelperWorker+0x46
4.0007dc ffffe000ebb05080 ffffd607 Blocked nt!SMKM_STORE<SM_TRAITS>::SmStReadThread+0xaa
4.0007e0 ffffe000ebb07840 ffffd5f4 Blocked nt!SMKM_STORE<SM_TRAITS>::SmStWorker+0xe1
4.0007e4 ffffe000ebb08080 ffffee7c Blocked nt!KeRemovePriQueue+0x1b7
4.0007e8 ffffe000ebb09080 ffffd607 Blocked nt!MiStoreEvictThread+0xfa
4.00044c ffffe000ebb66840 fffffcff Blocked srv2!RfspThreadPoolNodeManagerRun+0x7a
4.000340 ffffe000ebb67040 fffffcff Blocked srv2!RfspThreadPoolNodeWorkerProcessWorkItems+0xc1
4.000494 ffffe000ebb68040 fffffcff Blocked srv2!RfspThreadPoolNodeManagerRun+0x7a
4.0004b0 ffffe000ebb69040 fffffcff Blocked srv2!RfspThreadPoolNodeWorkerProcessWorkItems+0xc1
4.0004d0 ffffe000ebb6a040 fffffcff Blocked srv2!RfspThreadPoolNodeManagerRun+0x7a
4.0004fc ffffe000ebb6b040 fffffcff Blocked srv2!RfspThreadPoolNodeWorkerProcessWorkItems+0xc1
4.000814 ffffe000ebbe4080 fffffcf2 Blocked nt!EtwpLogger+0xcb
4.00092c ffffe000eb4d5080 ffffd724 Blocked nt!EtwpLogger+0xcb
4.0009cc ffffe000eb588200 fffffc74 Blocked nt!EtwpLogger+0xcb
4.000b18 ffffe000ebc1d080 fffff686 Blocked nt!EtwpLogger+0xcb
4.000c94 ffffe000ebdfb840 ffffd6d6 Blocked nt!SmKmStoreHelperWorker+0x46
4.000c98 ffffe000ebdfc080 ffffd6d6 Blocked nt!SmKmStoreHelperWorker+0x46
4.000c9c ffffe000ebdff080 ffffe320 Blocked nt!SMKM_STORE<SM_TRAITS>::SmStReadThread+0xaa
4.000ca0 ffffe000ebe1c840 ffffd606 Blocked nt!SMKM_STORE<SM_TRAITS>::SmStWorker+0xe1
4.000d74 ffffe000ebecc300 fffff106 Blocked nt!KeRemovePriQueue+0x1b7
4.000d78 ffffe000ebece840 ffffdec0 Blocked nt!KeRemovePriQueue+0x1b7
4.000d7c ffffe000ebed8040 ffffdec0 Blocked nt!KeRemovePriQueue+0x1b7
4.000d80 ffffe000ebed9040 fffff106 Blocked nt!KeRemovePriQueue+0x1b7
4.000d84 ffffe000ebede300 fffff106 Blocked nt!KeRemovePriQueue+0x1b7
4.000d88 ffffe000ebee2040 fffff106 Blocked nt!KeRemovePriQueue+0x1b7
4.000d8c ffffe000ebf0e840 ffffd8dc Blocked nt!SmKmStoreHelperWorker+0x46
4.000d90 ffffe000ebf11080 ffffdce3 Blocked nt!SmKmStoreHelperWorker+0x46
4.000d94 ffffe000ebf2d080 ffffd8de Blocked nt!SMKM_STORE<SM_TRAITS>::SmStReadThread+0xaa
4.000d98 ffffe000ebf30840 ffffd8c4 Blocked nt!SMKM_STORE<SM_TRAITS>::SmStWorker+0xe1
4.000da8 ffffe000eae4f840 ffffd732 Blocked nt!KeRemovePriQueue+0x1b7
4.000dac ffffe000ebf46840 ffffdec0 Blocked nt!KeRemovePriQueue+0x1b7
4.000db0 ffffe000ebf0f840 ffffdebf Blocked nt!KeRemovePriQueue+0x1b7
4.000db4 ffffe000ebee5040 fffff106 Blocked nt!KeRemovePriQueue+0x1b7

49
 4.000e48 ffffe000ebf57840 ffffd605 Blocked nt!SmKmStoreHelperWorker+0x46
4.000e4c ffffe000ebeee080 ffffd605 Blocked nt!SmKmStoreHelperWorker+0x46
4.000e50 ffffe000ebd34080 ffffd8de Blocked nt!SMKM_STORE<SM_TRAITS>::SmStReadThread+0xaa
4.000e54 ffffe000ebefc080 ffffd604 Blocked nt!SMKM_STORE<SM_TRAITS>::SmStWorker+0xe1
4.000e64 ffffe000ebf7e040 fffff106 Blocked nt!KeRemovePriQueue+0x1b7
4.000e68 ffffe000ebcec240 fffff613 Blocked nt!KeRemovePriQueue+0x1b7
4.000e6c ffffe000ebc53040 fffff106 Blocked nt!KeRemovePriQueue+0x1b7
4.000e70 ffffe000ebc7c040 ffffd732 Blocked nt!KeRemovePriQueue+0x1b7
4.000e74 ffffe000ebcf0840 fffff7c9 Blocked nt!KeRemovePriQueue+0x1b7
4.000e78 ffffe000ec115040 fffff943 Blocked nt!KeRemovePriQueue+0x1b7
4.000e7c ffffe000ec122040 ffffdec0 Blocked nt!KeRemovePriQueue+0x1b7
4.000e80 ffffe000ec123040 ffffd5f4 Blocked nt!KeRemovePriQueue+0x1b7
4.000e84 ffffe000ec124040 ffffdebe Blocked nt!KeRemovePriQueue+0x1b7
4.000e88 ffffe000ec125840 ffffdec0 Blocked nt!KeRemovePriQueue+0x1b7
4.000e8c ffffe000ec126040 fffff613 Blocked nt!KeRemovePriQueue+0x1b7
4.000e90 ffffe000ec128040 ffffdebe Blocked nt!KeRemovePriQueue+0x1b7
4.000e94 ffffe000ec12a040 fffff7c9 Blocked nt!KeRemovePriQueue+0x1b7
4.000e98 ffffe000ec12b840 fffff106 Blocked nt!KeRemovePriQueue+0x1b7
4.000e9c ffffe000ec135840 fffff106 Blocked nt!KeRemovePriQueue+0x1b7
4.0013a4 ffffe000ebe2f040 ffffd657 Blocked nt!KeRemovePriQueue+0x1b7
4.0013a8 ffffe000eb682840 ffffd8d8 Blocked nt!KeRemovePriQueue+0x1b7
4.0013ac ffffe000eb8ab300 ffffd8d8 Blocked nt!KeRemovePriQueue+0x1b7
4.0013b0 ffffe000eb8f4040 ffffd8d8 Blocked nt!KeRemovePriQueue+0x1b7
4.001138 ffffe000ec4e3080 ffffd7b4 Blocked nt!SmKmStoreHelperWorker+0x46
4.001140 ffffe000ebfde080 ffffd7b4 Blocked nt!SmKmStoreHelperWorker+0x46
4.001100 ffffe000ec4bd080 ffffe31e Blocked nt!SMKM_STORE<SM_TRAITS>::SmStReadThread+0xaa
4.0010fc ffffe000ec1eb840 ffffd63e Blocked nt!SMKM_STORE<SM_TRAITS>::SmStWorker+0xe1
4.001344 ffffe000ec62a080 ffffde28 Blocked nt!SmKmStoreHelperWorker+0x46
4.001348 ffffe000ec6d3080 ffffde28 Blocked nt!SmKmStoreHelperWorker+0x46
4.00134c ffffe000ec6cf080 ffffd8dd Blocked nt!SMKM_STORE<SM_TRAITS>::SmStReadThread+0xaa
4.001350 ffffe000ec5da180 ffffd782 Blocked nt!SMKM_STORE<SM_TRAITS>::SmStWorker+0xe1
4.000ef4 ffffe000ec7cd840 ffffd6db Blocked nt!SmKmStoreHelperWorker+0x46
4.00109c ffffe000ec410840 ffffd6db Blocked nt!SmKmStoreHelperWorker+0x46
4.0010a0 ffffe000ec48c840 ffffd8dd Blocked nt!SMKM_STORE<SM_TRAITS>::SmStReadThread+0xaa
4.001098 ffffe000ec7ea840 ffffd6db Blocked nt!SMKM_STORE<SM_TRAITS>::SmStWorker+0xe1
4.001420 ffffe000ec88f840 ffffd6dc Blocked nt!SmKmStoreHelperWorker+0x46
4.001424 ffffe000ec872080 ffffd6dc Blocked nt!SmKmStoreHelperWorker+0x46
4.001428 ffffe000ec3ca840 ffffd8dd Blocked nt!SMKM_STORE<SM_TRAITS>::SmStReadThread+0xaa
4.00142c ffffe000ec8ac840 ffffd6d7 Blocked nt!SMKM_STORE<SM_TRAITS>::SmStWorker+0xe1
4.0014b0 ffffe000ec95d840 ffffd793 Blocked nt!SmKmStoreHelperWorker+0x46
4.0014b4 ffffe000ec941840 ffffd6d9 Blocked nt!SmKmStoreHelperWorker+0x46
4.0014b8 ffffe000ec95b080 ffffd8dd Blocked nt!SMKM_STORE<SM_TRAITS>::SmStReadThread+0xaa
4.0014bc ffffe000ec97a840 ffffd6d8 Blocked nt!SMKM_STORE<SM_TRAITS>::SmStWorker+0xe1
4.001574 ffffe000eb8b4040 ffffee7c Blocked nt!KeRemovePriQueue+0x1b7
4.001028 ffffe000ec647840 ffffdee0 Blocked nt!EtwpLogger+0xcb

[ffffe000ead78840 smss.exe]

[ffffe000eb239080 csrss.exe]
180.000190 ffffe000eb23f080 ffffd602 Blocked nt!AlpcpWaitForSingleObject+0x3e
180.0001fc ffffe000eb78a080 ffffd602 Blocked nt!AlpcpWaitForSingleObject+0x3e
180.00023c ffffe000eb7cf080 ffffd622 Blocked win32kfull!RawInputThread+0x9aa
180.00040c ffffe000eb94a3c0 ffffd602 Blocked nt!AlpcpWaitForSingleObject+0x3e
180.000788 ffffe000eba97080 ffffd602 Blocked nt!AlpcpWaitForSingleObject+0x3e

[ffffe000eb764840 wininit.exe]
1c8.000208 ffffe000eb78f080 ffffee8a Blocked nt!IoRemoveIoCompletion+0x8a

[...]

[ffffe000ec09a080 NotMyfault.exe]
1594.0008cc ffffe000ecab7080 ffffd5e3 RUNNING nt!KeBugCheckEx
1594.001538 ffffe000ec360080 ffffd705 Blocked nt!IoRemoveIoCompletion+0x8a
1594.001540 ffffe000ec16e080 ffffd705 Blocked nt!IoRemoveIoCompletion+0x8a
1594.001544 ffffe000ec97c840 ffffd704 Blocked nt!IoRemoveIoCompletion+0x8a
1594.00154c ffffe000ec41f040 ffffd704 Blocked nt!IoRemoveIoCompletion+0x8a

50
1594.000614 ffffe000ec43a080 ffffd704 Blocked nt!IoRemoveIoCompletion+0x8a
1594.0017b0 ffffe000ec474080 ffffd704 Blocked nt!ObWaitForMultipleObjects+0x2bd
1594.0017ac ffffe000ec475080 ffffd704 Blocked nt!ObWaitForMultipleObjects+0x2bd

Threads Processed: 1185

12. Let’s now check processes that were waiting for user input:

0: kd> !stacks 2 NtUserGetMessage

Proc.Thread .Thread Ticks ThreadState Blocker
[fffff80148a1ca40 Idle]
[ffffe000e9040700 System]

[ffffe000ead78840 smss.exe]

[ffffe000eb239080 csrss.exe]

[ffffe000eb764840 wininit.exe]

[ffffe000eb21d840 csrss.exe]

[ffffe000eb7a52c0 winlogon.exe]

[ffffe000eb7e7080 services.exe]

[ffffe000eb7f4080 lsass.exe]

[ffffe000eb83e840 svchost.exe]

[ffffe000eb84e080 svchost.exe]

[ffffe000eb8a3080 dwm.exe]
354.000358 ffffe000eb8a6080 fffff713 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14

[ffffe000eb8c06c0 svchost.exe]

[ffffe000eb8c1400 svchost.exe]

[ffffe000eb8fa780 svchost.exe]

[ffffe000eb93f840 svchost.exe]

[ffffe000eb9426c0 vmacthlp.exe]

[ffffe000eb958840 WUDFHost.exe]

[ffffe000eb95b840 svchost.exe]

[ffffe000eb9a2840 svchost.exe]
484.000574 ffffe000eba4f080 fffff712 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
51
 win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14

[ffffe000eb9f4080 svchost.exe]

[ffffe000e90d6840 spoolsv.exe]

[ffffe000e90dd840 svchost.exe]

[ffffe000eb6f5080 svchost.exe]

[ffffe000eba89840 svchost.exe]

[ffffe000eba61080 vmtoolsd.exe]

[ffffe000eba8f840 MsMpEng.exe]
774.0003e4 ffffe000eb667080 ffffde3c Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14

[ffffe000eba8e740 VGAuthService.]

[ffffe000eb4a6840 dllhost.exe]
8bc.000908 ffffe000eb4cb080 fffff711 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14

[ffffe000eb4a2840 WmiPrvSE.exe]
8c4.0008c8 ffffe000eb4ac080 fffff711 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14

[ffffe000eb561080 dllhost.exe]
938.000968 ffffe000eb575080 fffff711 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14

52
 [ffffe000e92795c0 msdtc.exe]
9ac.0009c8 ffffe000eb46e6c0 fffff711 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14

[ffffe000e926d840 NisSrv.exe]

[ffffe000ebd12840 VSSVC.exe]

[ffffe000eb601080 sihost.exe]
b0c.000a78 ffffe000ebd6e080 fffff711 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14

[ffffe000ebd4e840 taskhostw.exe]
ac4.000af8 ffffe000ebd85080 ffffd608 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14

[ffffe000ebdb36c0 userinit.exe]

[ffffe000ebdca840 RuntimeBroker.]
c5c.0011d8 ffffe000ec6be840 fffff0f6 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14
c5c.001780 ffffe000eb956080 ffffe1e7 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14

[ffffe000ebdc7840 explorer.exe]
c64.000ce4 ffffe000ebe4d080 ffffe1e7 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a

53
 nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14
c64.000f64 ffffe000ec313080 ffffdb03 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14
c64.000f68 ffffe000ec304080 ffffd867 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14
c64.00088c ffffe000ec209840 ffffeaf5 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14
c64.0010a4 ffffe000ec305740 fffff17d Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14
c64.001070 ffffe000ec2d9080 ffffd86c Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14
c64.00106c ffffe000eba82840 ffffd7b4 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13

54
 USER32!NtUserGetMessage+0x14
c64.001124 ffffe000ec593080 ffffe1e7 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForMultipleObjects+0x24e
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14
c64.000718 ffffe000ecaa45c0 ffffd7b4 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14

[ffffe000ebdcd840 SkypeHost.exe]

[ffffe000eb4195c0 SearchIndexer.]

[ffffe000ebf0a840 ShellExperienc]
d9c.000ea4 ffffe000ec129840 fffff710 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14

[ffffe000ebf00840 SearchUI.exe]
e58.000f50 ffffe000ec32b080 ffffe1e7 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14
e58.000f54 ffffe000ec31c840 fffff710 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14

[ffffe000ec252080 TabTip.exe]
fbc.000fc0 ffffe000ec263840 fffff710 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98

55
 win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14
fbc.000a30 ffffe000ec227840 ffffd5e3 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14

[ffffe000ec121080 TabTip32.exe]

[ffffe000ebd73840 svchost.exe]

[ffffe000ec2b4840 WmiPrvSE.exe]
10d8.0010dc ffffe000ec2b6080 fffff710 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14
10d8.001630 ffffe000eb9a8440 ffffd8cb Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14

[ffffe000eb6c3080 vmtoolsd.exe]

[ffffe000ec373080 OneDrive.exe]
12b0.0012b4 ffffe000ec367080 ffffe1e7 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
+0x6c393824

[ffffe000ec24a080 ApplicationFra]
1050.000b5c ffffe000ec634080 ffffd735 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForMultipleObjects+0x24e
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14

56
 [ffffe000ec491080 MicrosoftEdge.]

[ffffe000ec220080 browser_broker]

[ffffe000ec62c840 MicrosoftEdgeC]
1354.000cec ffffe000ec5bd080 fffff0d1 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14

[ffffe000ec6a8640 SearchProtocol]

[ffffe000ec6c4080 SearchFilterHo]

[ffffe000ec77d840 MicrosoftEdgeC]
105c.00101c ffffe000ec84d080 fffff044 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14

[ffffe000ec88d840 MicrosoftEdgeC]
1430.00169c ffffe000ebdbd080 ffffe1e7 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14
1430.0016a4 ffffe000ebe76080 ffffeb33 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14

[ffffe000ec944840 MicrosoftEdgeC]
14c0.0014f0 ffffe000ec99d080 ffffd5e7 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14
14c0.001504 ffffe000ec9cf080 ffffe1e7 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a

57
 nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14
14c0.001508 ffffe000ec9d0080 ffffeff5 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14

[ffffe000ec156840 notepad.exe]
2c4.0002bc ffffe000ec563080 ffffd842 Blocked nt!KiSwapContext+0x76
nt!KiSwapThread+0x15a
nt!KiCommitThreadWait+0x149
nt!KeWaitForSingleObject+0x375
win32kfull!xxxRealSleepThread+0x355
win32kfull!xxxSleepThread2+0x98
win32kfull!xxxRealInternalGetMessage+0xb4d
win32kfull!NtUserGetMessage+0x90
nt!KiSystemServiceCopyEnd+0x13
USER32!NtUserGetMessage+0x14

[ffffe000ec9aa540 audiodg.exe]

[ffffe000eca66840 svchost.exe]

[ffffe000ec089080 WmiApSrv.exe]

[ffffe000ec8db080 TabTip.exe]

[ffffe000ec09a080 NotMyfault.exe]

Threads Processed: 1185

13. We can also list processes grouped by session id:

0: kd> !sprocess -4
Total sessions : 2

Session 0
_MM_SESSION_SPACE ffffd000250f3000
_MMSESSION ffffd000250f3b40
PROCESS ffffe000eb239080
SessionId: 0 Cid: 0180 Peb: 61467f1000 ParentCid: 0174
DirBase: 04466000 ObjectTable: ffffc000daca8040 HandleCount: <Data Not Accessible>
Image: csrss.exe

PROCESS ffffe000eb764840
SessionId: 0 Cid: 01c8 Peb: 8e26a86000 ParentCid: 0174
DirBase: 26eac000 ObjectTable: ffffc000dad79e80 HandleCount: <Data Not Accessible>
Image: wininit.exe

58
PROCESS ffffe000eb7e7080
SessionId: 0 Cid: 0250 Peb: 721f3eb000 ParentCid: 01c8
DirBase: 03be0000 ObjectTable: ffffc000e2351040 HandleCount: <Data Not Accessible>
Image: services.exe

PROCESS ffffe000eb7f4080
SessionId: 0 Cid: 025c Peb: ea96f64000 ParentCid: 01c8
DirBase: 2f4fd000 ObjectTable: ffffc000e236b340 HandleCount: <Data Not Accessible>
Image: lsass.exe

PROCESS ffffe000eb83e840
SessionId: 0 Cid: 02b4 Peb: 16cdd6000 ParentCid: 0250
DirBase: 364d2000 ObjectTable: ffffc000dae60440 HandleCount: <Data Not Accessible>
Image: svchost.exe

PROCESS ffffe000eb84e080
SessionId: 0 Cid: 02ec Peb: baa1131000 ParentCid: 0250
DirBase: 3636b000 ObjectTable: ffffc000daeb6d80 HandleCount: <Data Not Accessible>
Image: svchost.exe

PROCESS ffffe000eb8c06c0
SessionId: 0 Cid: 0388 Peb: a6e152b000 ParentCid: 0250
DirBase: 35129000 ObjectTable: ffffc000daf83e80 HandleCount: <Data Not Accessible>
Image: svchost.exe

PROCESS ffffe000eb8c1400
SessionId: 0 Cid: 0390 Peb: 5f9c0e8000 ParentCid: 0250
DirBase: 34cb3000 ObjectTable: ffffc000daf8bdc0 HandleCount: <Data Not Accessible>
Image: svchost.exe

PROCESS ffffe000eb8fa780
SessionId: 0 Cid: 03f4 Peb: a47f49000 ParentCid: 0250
DirBase: 343d8000 ObjectTable: ffffc000db068040 HandleCount: <Data Not Accessible>
Image: svchost.exe

PROCESS ffffe000eb93f840
SessionId: 0 Cid: 0318 Peb: 60c92fc000 ParentCid: 0250
DirBase: 34e26000 ObjectTable: ffffc000db0ac840 HandleCount: <Data Not Accessible>
Image: svchost.exe

PROCESS ffffe000eb9426c0
SessionId: 0 Cid: 03d4 Peb: 002dd000 ParentCid: 0250
DirBase: 266ac000 ObjectTable: ffffc000db13ae80 HandleCount: <Data Not Accessible>
Image: vmacthlp.exe

PROCESS ffffe000eb958840
SessionId: 0 Cid: 0420 Peb: 8c3d62b000 ParentCid: 0390
DirBase: 2e32f000 ObjectTable: ffffc000db0c7d80 HandleCount: <Data Not Accessible>
Image: WUDFHost.exe

PROCESS ffffe000eb95b840
SessionId: 0 Cid: 0428 Peb: 110cd9c000 ParentCid: 0250
DirBase: 26779000 ObjectTable: ffffc000db0c5700 HandleCount: <Data Not Accessible>
Image: svchost.exe

PROCESS ffffe000eb9a2840
SessionId: 0 Cid: 0484 Peb: b5f9f4a000 ParentCid: 0250
DirBase: 2dbc4000 ObjectTable: ffffc000db0ff580 HandleCount: <Data Not Accessible>

59
 Image: svchost.exe

PROCESS ffffe000eb9f4080
SessionId: 0 Cid: 04dc Peb: 781ee21000 ParentCid: 0250
DirBase: 2c612000 ObjectTable: ffffc000db1c56c0 HandleCount: <Data Not Accessible>
Image: svchost.exe

PROCESS ffffe000e90d6840
SessionId: 0 Cid: 05f8 Peb: 003fa000 ParentCid: 0250
DirBase: 2a489000 ObjectTable: ffffc000db3648c0 HandleCount: <Data Not Accessible>
Image: spoolsv.exe

PROCESS ffffe000e90dd840
SessionId: 0 Cid: 0634 Peb: 7ed35ba000 ParentCid: 0250
DirBase: 2a3d4000 ObjectTable: ffffc000db1dde80 HandleCount: <Data Not Accessible>
Image: svchost.exe

PROCESS ffffe000eb6f5080
SessionId: 0 Cid: 0708 Peb: dd28a8a000 ParentCid: 0250
DirBase: 22a1d000 ObjectTable: ffffc000db53b040 HandleCount: <Data Not Accessible>
Image: svchost.exe

PROCESS ffffe000eba89840
SessionId: 0 Cid: 0748 Peb: 78deb31000 ParentCid: 0250
DirBase: 22373000 ObjectTable: ffffc000db55a940 HandleCount: <Data Not Accessible>
Image: svchost.exe

PROCESS ffffe000eba61080
SessionId: 0 Cid: 0754 Peb: 00392000 ParentCid: 0250
DirBase: 21e8b000 ObjectTable: ffffc000db55d780 HandleCount: <Data Not Accessible>
Image: vmtoolsd.exe

PROCESS ffffe000eba8f840
SessionId: 0 Cid: 0774 Peb: dc8dd81000 ParentCid: 0250
DirBase: 21fd9000 ObjectTable: ffffc000db568580 HandleCount: <Data Not Accessible>
Image: MsMpEng.exe

PROCESS ffffe000eba8e740
SessionId: 0 Cid: 077c Peb: 00233000 ParentCid: 0250
DirBase: 21a1e000 ObjectTable: ffffc000db572e80 HandleCount: <Data Not Accessible>
Image: VGAuthService.exe

PROCESS ffffe000eb4a6840
SessionId: 0 Cid: 08bc Peb: f1d1a3b000 ParentCid: 0250
DirBase: 05f4b000 ObjectTable: ffffc000daf36680 HandleCount: <Data Not Accessible>
Image: dllhost.exe

PROCESS ffffe000eb4a2840
SessionId: 0 Cid: 08c4 Peb: 35a123000 ParentCid: 02b4
DirBase: 01314000 ObjectTable: ffffc000db2b3240 HandleCount: <Data Not Accessible>
Image: WmiPrvSE.exe

PROCESS ffffe000eb561080
SessionId: 0 Cid: 0938 Peb: b01aa10000 ParentCid: 0250
DirBase: 055a4000 ObjectTable: ffffc000db9d8d80 HandleCount: <Data Not Accessible>
Image: dllhost.exe

60
PROCESS ffffe000e92795c0
SessionId: 0 Cid: 09ac Peb: 6db75d4000 ParentCid: 0250
DirBase: 08fb8000 ObjectTable: ffffc000dba8e8c0 HandleCount: <Data Not Accessible>
Image: msdtc.exe

PROCESS ffffe000e926d840
SessionId: 0 Cid: 0a8c Peb: f58d62c000 ParentCid: 0250
DirBase: 166d8000 ObjectTable: ffffc000db7a9480 HandleCount: <Data Not Accessible>
Image: NisSrv.exe

PROCESS ffffe000ebd12840
SessionId: 0 Cid: 0bd8 Peb: bbb79eb000 ParentCid: 0250
DirBase: 00a69000 ObjectTable: ffffc000db7cc540 HandleCount: <Data Not Accessible>
Image: VSSVC.exe

PROCESS ffffe000eb4195c0
SessionId: 0 Cid: 0d6c Peb: abc12fc000 ParentCid: 0250
DirBase: 0cb17000 ObjectTable: ffffc000dc22be80 HandleCount: <Data Not Accessible>
Image: SearchIndexer.exe

PROCESS ffffe000ebd73840
SessionId: 0 Cid: 1090 Peb: 87553cb000 ParentCid: 0250
DirBase: 382e7000 ObjectTable: ffffc000dc62f8c0 HandleCount: <Data Not Accessible>
Image: svchost.exe

PROCESS ffffe000ec2b4840
SessionId: 0 Cid: 10d8 Peb: 2a11a5c000 ParentCid: 02b4
DirBase: 1f9af000 ObjectTable: ffffc000dc714340 HandleCount: <Data Not Accessible>
Image: WmiPrvSE.exe

PROCESS ffffe000ec6c4080
SessionId: 0 Cid: 1364 Peb: 4bddb6a000 ParentCid: 0d6c
DirBase: 21a68000 ObjectTable: ffffc000dd698c40 HandleCount: <Data Not Accessible>
Image: SearchFilterHost.exe

PROCESS ffffe000ec9aa540
SessionId: 0 Cid: 0bd4 Peb: 9076394000 ParentCid: 0318
DirBase: 36e80000 ObjectTable: ffffc000dd7f4e80 HandleCount: <Data Not Accessible>
Image: audiodg.exe

PROCESS ffffe000ec089080
SessionId: 0 Cid: 0be0 Peb: cd7c56f000 ParentCid: 0250
DirBase: 17c24000 ObjectTable: ffffc000dd89e6c0 HandleCount: <Data Not Accessible>
Image: WmiApSrv.exe

Session 1
_MM_SESSION_SPACE ffffd000251ac000
_MMSESSION ffffd000251acb40
PROCESS ffffe000eb21d840
SessionId: 1 Cid: 01d0 Peb: 27d00b2000 ParentCid: 01c0
DirBase: 2685f000 ObjectTable: ffffc000dad6fac0 HandleCount: <Data Not Accessible>
Image: csrss.exe

PROCESS ffffe000eb7a52c0
SessionId: 1 Cid: 021c Peb: 9668399000 ParentCid: 01c0
DirBase: 01165000 ObjectTable: ffffc000dad8fe80 HandleCount: <Data Not Accessible>
Image: winlogon.exe

61
PROCESS ffffe000eb8a3080
SessionId: 1 Cid: 0354 Peb: e4fdc6b000 ParentCid: 021c
DirBase: 3483c000 ObjectTable: ffffc000daf1e3c0 HandleCount: <Data Not Accessible>
Image: dwm.exe

PROCESS ffffe000eb601080
SessionId: 1 Cid: 0b0c Peb: d8ab44a000 ParentCid: 0388
DirBase: 3424e000 ObjectTable: ffffc000dbe8fb00 HandleCount: <Data Not Accessible>
Image: sihost.exe

PROCESS ffffe000ebd4e840
SessionId: 1 Cid: 0ac4 Peb: 4b7b3c1000 ParentCid: 0388
DirBase: 0a8da000 ObjectTable: ffffc000dbeaad00 HandleCount: <Data Not Accessible>
Image: taskhostw.exe

PROCESS ffffe000ebdb36c0
SessionId: 1 Cid: 0c40 Peb: f248fb000 ParentCid: 021c
DirBase: 08820000 ObjectTable: 00000000 HandleCount: 0.
Image: userinit.exe

PROCESS ffffe000ebdc7840
SessionId: 1 Cid: 0c64 Peb: 0036a000 ParentCid: 0c40
DirBase: 08c52000 ObjectTable: ffffc000dbf4c880 HandleCount: <Data Not Accessible>
Image: explorer.exe

PROCESS ffffe000ebdca840
SessionId: 1 Cid: 0c5c Peb: 10ca437000 ParentCid: 02b4
DirBase: 08cad000 ObjectTable: ffffc000dbf687c0 HandleCount: <Data Not Accessible>
Image: RuntimeBroker.exe

PROCESS ffffe000ebdcd840
SessionId: 1 Cid: 0ca4 Peb: 0032e000 ParentCid: 02b4
DirBase: 0a510000 ObjectTable: ffffc000dbc48d80 HandleCount: <Data Not Accessible>
Image: SkypeHost.exe

PROCESS ffffe000ebf0a840
SessionId: 1 Cid: 0d9c Peb: 30f35e8000 ParentCid: 02b4
DeepFreeze
DirBase: 0f0b5000 ObjectTable: ffffc000dc267840 HandleCount: <Data Not Accessible>
Image: ShellExperienceHost.exe

PROCESS ffffe000ebf00840
SessionId: 1 Cid: 0e58 Peb: fc6f501000 ParentCid: 02b4
DeepFreeze
DirBase: 02e3a000 ObjectTable: ffffc000dc32f880 HandleCount: <Data Not Accessible>
Image: SearchUI.exe

PROCESS ffffe000ec252080
SessionId: 1 Cid: 0fbc Peb: d460f56000 ParentCid: 0390
DirBase: 16d28000 ObjectTable: ffffc000dc556880 HandleCount: <Data Not Accessible>
Image: TabTip.exe

PROCESS ffffe000ec121080
SessionId: 1 Cid: 0ff0 Peb: 04490000 ParentCid: 0fbc
DirBase: 17c60000 ObjectTable: ffffc000dc583200 HandleCount: <Data Not Accessible>
Image: TabTip32.exe

62
PROCESS ffffe000eb6c3080
SessionId: 1 Cid: 1228 Peb: 00307000 ParentCid: 0c64
DirBase: 28ac7000 ObjectTable: ffffc000dca7b440 HandleCount: <Data Not Accessible>
Image: vmtoolsd.exe

PROCESS ffffe000ec373080
SessionId: 1 Cid: 12b0 Peb: 00516000 ParentCid: 0c64
DirBase: 27369000 ObjectTable: ffffc000dd3bc840 HandleCount: <Data Not Accessible>
Image: OneDrive.exe

PROCESS ffffe000ec24a080
SessionId: 1 Cid: 1050 Peb: c4977ce000 ParentCid: 02b4
DirBase: 21d8b000 ObjectTable: ffffc000dd286c80 HandleCount: <Data Not Accessible>
Image: ApplicationFrameHost.exe

PROCESS ffffe000ec491080
SessionId: 1 Cid: 10f8 Peb: 9023376000 ParentCid: 02b4
DirBase: 13390000 ObjectTable: ffffc000dd546040 HandleCount: <Data Not Accessible>
Image: MicrosoftEdge.exe

PROCESS ffffe000ec220080
SessionId: 1 Cid: 1208 Peb: 6db6d66000 ParentCid: 02b4
DirBase: 1d195000 ObjectTable: ffffc000dd5ab6c0 HandleCount: <Data Not Accessible>
Image: browser_broker.exe

PROCESS ffffe000ec62c840
SessionId: 1 Cid: 1354 Peb: b3574a5000 ParentCid: 0c5c
DeepFreeze
DirBase: 1d545000 ObjectTable: ffffc000dd602bc0 HandleCount: <Data Not Accessible>
Image: MicrosoftEdgeCP.exe

PROCESS ffffe000ec6a8640
SessionId: 1 Cid: 1378 Peb: da2c2ac000 ParentCid: 0d6c
DirBase: 17e34000 ObjectTable: ffffc000dd68b480 HandleCount: <Data Not Accessible>
Image: SearchProtocolHost.exe

PROCESS ffffe000ec77d840
SessionId: 1 Cid: 105c Peb: 35b6605000 ParentCid: 0c5c
DeepFreeze
DirBase: 2baf9000 ObjectTable: ffffc000dd680200 HandleCount: <Data Not Accessible>
Image: MicrosoftEdgeCP.exe

PROCESS ffffe000ec88d840
SessionId: 1 Cid: 1430 Peb: 9b37cc6000 ParentCid: 0c5c
DirBase: 39846000 ObjectTable: ffffc000dca0b040 HandleCount: <Data Not Accessible>
Image: MicrosoftEdgeCP.exe

PROCESS ffffe000ec944840
SessionId: 1 Cid: 14c0 Peb: 9af0d18000 ParentCid: 0c5c
DirBase: 05df7000 ObjectTable: ffffc000dc647440 HandleCount: <Data Not Accessible>
Image: MicrosoftEdgeCP.exe

PROCESS ffffe000ec156840
SessionId: 1 Cid: 02c4 Peb: f3e927f000 ParentCid: 0c64
DirBase: 04d22000 ObjectTable: ffffc000dd7a2d40 HandleCount: <Data Not Accessible>
Image: notepad.exe

63
PROCESS ffffe000eca66840
SessionId: 1 Cid: 03a8 Peb: 6389def000 ParentCid: 0250
DirBase: 388ee000 ObjectTable: ffffc000db8fde80 HandleCount: <Data Not Accessible>
Image: svchost.exe

PROCESS ffffe000ec8db080
SessionId: 1 Cid: 05ac Peb: b6c6ea8000 ParentCid: 0390
DirBase: 3a507000 ObjectTable: 00000000 HandleCount: 0.
Image: TabTip.exe

PROCESS ffffe000ec09a080
SessionId: 1 Cid: 1594 Peb: 00379000 ParentCid: 0c64
DirBase: 3cfce000 ObjectTable: ffffc000dd91c2c0 HandleCount: <Data Not Accessible>
Image: NotMyfault.exe

14. We close logging before exiting WinDbg:

0: kd> .logclose
Closing open log file F:\AdvWMDA-Dumps\x64\C1.log

Note: To avoid possible confusion and glitches we recommend exiting WinDbg after each exercise.

64