1、Introduction and Case Studies,Tate Calhoun Platforms Global Escalation Services Core Team,(Xperf),Agenda,What is Xperf?How does it work?What can it do? Case Studies!Where can I get it?,What is Xperf?,No application or driver is an island! Triaging system wide performance issues via multiple trace st
2、reams introduces complexity. Data must be aggregated, visualized, and understood to determine root cause. Xperf is the long term investment Microsoft is making to meet this diagnostic need.Support Engineer Trends/Challenges:1.) Memory.dmps are not getting smaller! (yet) 2.) Classic yet challenging a
3、pplication and system hangs, slow I/O, and other system wide issues 3.) User observation to code behavior. Stackwalking! 4.) The visualization of complex ETW data,What is Xperf?,An ETW controller and consumer Created by the Windows Fundamentals team Designed for analyzing system and application perf
4、ormance on Windows Vista, Windows Server 2008, and later.,What is Xperf?,Capture and analysis model follows this general flow:ETW tracing is enabled using xperf (or xbootmgr). Scenario of interest is performed. ETW tracing is stopped using xperf (or automatically by xbootmgr) and the data is saved t
5、o an ETL trace file. Trace files can then further be processed with xperf and/or viewed with xperfview. Traces can be processed on the machine on which they were taken, or copied to another machine for analysis (including cross-architecture). Everything needed for analysis is stored in the trace fil
6、e.,How does this work?,Performance Analyzer is built on top of the Event Tracing for Windows (ETW) infrastructure. ETW enables Windows and applications to efficiently generate events, which can be enabled and disabled at any time without requiring system or process restarts. ETW collects requested k
7、ernel events and saves to an etl file.Provides extensive details about the system. Context switches, interrupts, deferred procedure calls, process and thread creation and destruction, disk I/Os, hard faults, processor P-State transitions, and registry operations, etc.One of the great features of ETW
8、, supported in WPT, is the support of symbol decoding, sample profiling, and capture of call stacks on kernel events.,ETW Overview,ETW, or Event Tracing for Windows, is a high performance kernel level tracing system that made its first appearance in Windows 2000 and has found widespread use since. M
9、ost operations throughout the operating system that are interesting to a performance analyst are already instrumented. ETW is comprised of four key components: Providers: Components that generate events. They are instrumented with the ETW EventWrite() API. Sessions: Kernel objects that collect event
10、s into buffers and send those events to a file or directly a consuming application. Controllers: Programs that create and control sessions, and that control providers. Controlling providers consists of enabling and disabling events, or groups of events. Consumer: Programs that consume event data fro
11、m ETW sessions. The data can be consumed from a file, or in real time streaming mode. If you are interested in instrumenting your own applications, please see the ETW documentation in MSDN,ETW Components,ETW Advantages for Performance Tracing,Events can be enabled or disabled dynamically, no system
12、or process restartsBy default all events are disabled and the call to EventWrite() incurs a very minimal overhead just a flag check. This allows retail (shipping) code to be fully instrumented and enables tracing to be enabled in shipping components at any time. Even when enabled, event tracing has
13、a very low run-time overhead, even when large numbers of events are generated per second. Once begun, an ETW logging session has a defined maximum memory footprint. The disk footprint can be similarly limited. CPU load is directly proportional to the event rate and scales very well. Events from mult
14、iple providers, sessions and the kernel logger can be merged seamlessly when collected simultaneously on the same machine. This allows different kinds of events and settings to be used on different logging sessions and then merged together after the fact. The Windows kernel and multiple components a
15、re already extensively instrumented. ETW is the instrumentation method of choice on Windows.,Key Benefits to Support Engineers,1.) Lightweight/Non-Invasive, only about 2-3% CPU, 5% Memory, configurable. 2.) STACK TRACING! 3.) Independent Data Gathering vs. Analysis (Easily understatedthis is almost
16、like being able to share a live debug session) 4.) We are trying to move away from “how do I reproduce this problem internally?“ to “What was the problem and how to fix it“ 5.) A differenttemporal debugger?,More about stacks,In Vista+, stack walking can be enabled for certain kernel events. When com
17、bined with symbols on the processing side this extends analysis using system events to correlate back into the code context that triggered the corresponding event, from up in kernel mode to deep inside applications. This is a very powerful tool!,What can it do?,More and more each release and in the
18、upcoming Win7 beta releases of the tool you will see improved data capturing ability. For now, here are some scenarios that we have explored.1.) CPU Consumption and Scheduling Issues 2.) Storage Delay, Disk I/O Issues 3.) Power Management Events,Lets get started,Download and install of the tools (pr
19、eferably custom to c:xperf) and an elevated CMD. http:/ get started,Lets take a view of the advanced stackwalk options and other Kernel groups (xperf -providers K),Lets get started,Here are the stackwalk flags (xperf -help stackwalk),Lets get started,Any Easy Syntax to start out!?!Using the followin
20、g is a good head start and we see that massive customization is possible with stackwalk, etc.Xperf -on DiagEasyFor profiling issues to get stacks one can use:Xperf -on Latency -stackwalk Profile,Case Study 1 IE Hang,Xperf -on latency -stackwalk profile Repro IE Hang Xperf -d ielatency.etl(-d MergedE
21、TL Merge the ETL files of stopped logging sessions into MergedETL; if no session is stopped explicitly, the “NT Kernel Logger“ stopped implicitly.),Case Study 1 IE Hang,Case Study 1 IE Hang,Case Study 1 IE Hang,So we have spent a significant portion of time in MSHTML.DLL!CHtPvPv:LookupI happened to
22、have a dump of the process as well.,Case Study 1 IE Hang,Summary: Searching around this mshtml code, this is being reviewed via a Windows 7 bug. Unfortunately, we didnt have a repro at the time. I provided this etl trace to the team (with dumps and error reporting data) and the investigation continu
23、es!A great example of the portability advantages of this tool!,Case Study 2 Everything slow,Xperf -on latency -stackwalk profile Repro slowness Xperf -d WUlatency.etl Note overlay of Disk Utilization in the middle.,Case Study 2 Everything slow,So we have some activity here in cbscore.dll!CCbsPublicP
24、ackage:EvaluateApplicability, etc.,Case Study 2 Everything slow,What is cbscore.dll? Component-Based Servicing Component-Based Servicing (CBS) is part of the servicing stack. The servicing stack is a set of files and resources that are required to service a Windows image or operating system. The ser
25、vicing stack is available on all Windows Vista and Windows Server 2008 installations, as well as in the Windows Automated Installation Kit AIK (Windows AIK) and the Windows OEM Preinstallation Kit (Windows OPK). CBS provides various APIs (which are not publicly available) to its client installers to
26、 service the operating system components. Client installers such as Windows Update or Windows Installer work with CBS to enumerate, install, update, and uninstall component packages on the destination operating system. CBS interacts with the Component Servicing Infrastructure to perform the necessar
27、y system changes. For more information about CBS, see http:/ theres even CBS logging as well if needed: http:/ Study 2 Everything slow,Summary: My machine is working hard scanning for updates, so this seems “expected“. However, it turns out there are actually performance improvements exactly here in
28、 cbscore for Windows 7 and Vista SP1 to mitigate this issue!,Case Study 3 Outlook hangs on startup,Xperf -on latency -stackwalk profile Start Outlook Xperf -d outlookhang.etl,Ive got a DPC issue on this machine to track down further with the NIC :).15% DPC time as a constant is not good for performa
29、nce!,Interesting to note that it matches the CPU Sampling by CPU.,Case Study 3 Outlook hangs on startup,However, we are 41% idle per the above,Case Study 3 Outlook hangs on startup,What is going on here? Half Idle, Half,Case Study 3 Outlook hangs on startup,What is going on here? Half Idle, Half,Cas
30、e Study 3 Outlook hangs on startup,Ah, my backup application at PID 496 is running doing its thing,Case Study 3 Outlook hangs on startup,But what is Outlook doing during all this?,Case Study 3 Outlook hangs on startup,Summary: So Ive found a taxing network DPC issue on my machine so Ill go follow up
31、 with my NIC configuration/driver. (turns out disable renable did the trick) Also, Ive seen that Ive got my backup application querying the file system here likely indexing by design. Also, Ive got more digging to do into Outlook using the Cswitch stackwalk event (running the trace before the hang).
32、Addendum: Found two separate hang bugs in Outlook actually where the root cause of the hangs(with the debugger, I cheated, sorry, I feel bad). The write stack matched one though!,Case Study 4 Run Scott Run,The situation is that on two different platforms (physical vs. virtual) of stated same build a
33、re displaying two different scheduling behaviors when viewing CPU time for RunScottRun.exe. The exe “doesnt stay on a processor”xperf -on PROFILE+DISPATCHER+PROC_THREAD+LOADER+HARD_FAULTS+INTERRUPT+DPC+CSWITCH -maxbuffers 1024Here is what the “good“ trace looks likeWe have our Process level view to
34、clone the selection.,Case Study 4 Run Scott Run,Now the CPU Scheduling section. At a high level the problem reported is that RunScottRun.exe is having trouble getting enough time on a single CPU and we see context switches.,Case Study 4 Run Scott Run,We can see this at a little bit lower level what
35、is getting to run,Case Study 4 Run Scott Run,At an even lower level RunScottRun.exe is lower priority, we see SomeService.exe sneak in here at Pri 9, then back to RunScottRun.exe after Delay Execution.,Case Study 4 Run Scott Run,If you are an Excel wiz :) the data can be exported for another view. H
36、eres the “good“ trace summary/pivot,“Good”,“Bad”,Case Study 5 A Random Stackwalk - CreateProcess,C:xperfxperf -on DiagEasy -stackwalk ProcessCreate+ProcessDelete C:xperfnotepad C:xperfxperf -d processcreate.etl Merged Etl: processcreate.etl (shows stacks on 7sdk Stack Counts by Type),Case Study 5 A
37、Random Stackwalk - CreateProcess,Case Study 5 A Random Stackwalk - CreateProcess,Case Study 5 A Random Stackwalk - CreateProcess,Case Study 5 A Random Stackwalk - CreateProcess,Case Study 6 A Random Stackwalk File I/O,C:xperfxperf -on FileIO+FILENAME -stackwalk file.txtC:xperfxperf -d file.etl Merge
38、d Etl: file.etl C:xperfxperf file.etl (shows stacks with 08 ver, in File IO Section),Case Study 6 A Random Stackwalk File I/O,Case Study 7 A Random Stackwalk Registry,C:xperfxperf -on DiagEasy+REGISTRY -stackwalk reg.txt C:xperfregedit C:xperfxperf -d reg.etl Merged Etl: reg.etl (shows stacks with 0
39、8 ver, in Registry section),Case Study 7 A Random Stackwalk Registry,Case Study 8 A Random Stackwalk Power,C:xperfxperf -on DiagEasy+POWER -stackwalk power.txt C:xperfxperf -d power.etl Merged Etl: power.etl C:xperfxperf power.etl C:xperfc:xperf7sdkxperf power.etl (shows stacks with 7sdk Stack Count
40、s by Type),Case Study 8 A Random Stackwalk Power,Case Study 9 SLOOOW I/O,Case Study 9 SLOOOW I/O,What didnt we cover?,Xbootmgr.exe! It can trace boot, hibernate, standby, shutdown, rebootCycle performance.,Heap and VirtualAlloc tracing in user mode.,Where do I get this?,The toolkit: http:/ forum: Th
41、e Windows Performance Toolkit ForumRemember to check the Windows 7 SDK for the cool Stack Counts by Type section and other improvements!,Top Questions,Can I use Performance Analyzer on Windows XP or Windows Server 2003?Unfortunately, the answer is no. While Windows XP and Windows Server 2003 do supp
42、ort collection of ETL traces, these OSes do not contain instrumentation for most of the events needed by Performance Analyzer (PA). You need Windows Vista or later OS to use PA. An example of crucial instrumentation added in Vista is stack walking. Performance analysis without stacks can be an extre
43、mely daunting task that only a true expert with access to source code can tackle. With the Windows Vista release, Microsoft has really taken the OS to next level in terms of system diagnosibility and a lot of the analysis based on this instrumentation isnt feasible on Windows XP/Server 2003 ETL trac
44、es.,Top Questions,How do I find out the time span of the trace my customer collected to correlate with other data? (-a tracestats),Top Questions,How to enable stackwalking on x64 systems? x64 ETW stackwalking is only supported on Windows Vista SP1, Windows Server 2008, and above, and requires settin
45、g a certain registry value (see below) to 1 and rebooting the machine ( so that Windows kernel picks it up).HKLMSystemCurrentControlSetControlSession ManagerMemory ManagementDisablePagingExecutive 1, 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product
46、 names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market c
47、onditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.,
