7 Intel Processor Trace (Intel PT) is an extension of Intel Architecture that
8 collects information about software execution such as control flow, execution
9 modes and timings and formats it into highly compressed binary packets.
10 Technical details are documented in the Intel 64 and IA-32 Architectures
11 Software Developer Manuals, Chapter 36 Intel Processor Trace.
13 Intel PT is first supported in Intel Core M and 5th generation Intel Core
14 processors that are based on the Intel micro-architecture code name Broadwell.
16 Trace data is collected by 'perf record' and stored within the perf.data file.
17 See below for options to 'perf record'.
19 Trace data must be 'decoded' which involves walking the object code and matching
20 the trace data packets. For example a TNT packet only tells whether a
21 conditional branch was taken or not taken, so to make use of that packet the
22 decoder must know precisely which instruction was being executed.
24 Decoding is done on-the-fly. The decoder outputs samples in the same format as
25 samples output by perf hardware events, for example as though the "instructions"
26 or "branches" events had been recorded. Presently 3 tools support this:
27 'perf script', 'perf report' and 'perf inject'. See below for more information
30 The main distinguishing feature of Intel PT is that the decoder can determine
31 the exact flow of software execution. Intel PT can be used to understand why
32 and how did software get to a certain point, or behave a certain way. The
33 software does not have to be recompiled, so Intel PT works with debug or release
34 builds, however the executed images are needed - which makes use in JIT-compiled
35 environments, or with self-modified code, a challenge. Also symbols need to be
36 provided to make sense of addresses.
38 A limitation of Intel PT is that it produces huge amounts of trace data
39 (hundreds of megabytes per second per core) which takes a long time to decode,
40 for example two or three orders of magnitude longer than it took to collect.
41 Another limitation is the performance impact of tracing, something that will
42 vary depending on the use-case and architecture.
48 It is important to start small. That is because it is easy to capture vastly
49 more data than can possibly be processed.
51 The simplest thing to do with Intel PT is userspace profiling of small programs.
52 Data is captured with 'perf record' e.g. to trace 'ls' userspace-only:
54 perf record -e intel_pt//u ls
56 And profiled with 'perf report' e.g.
60 To also trace kernel space presents a problem, namely kernel self-modifying
61 code. A fairly good kernel image is available in /proc/kcore but to get an
62 accurate image a copy of /proc/kcore needs to be made under the same conditions
63 as the data capture. A script perf-with-kcore can do that, but beware that the
64 script makes use of 'sudo' to copy /proc/kcore. If you have perf installed
65 locally from the source tree you can do:
67 ~/libexec/perf-core/perf-with-kcore record pt_ls -e intel_pt// -- ls
69 which will create a directory named 'pt_ls' and put the perf.data file and
70 copies of /proc/kcore, /proc/kallsyms and /proc/modules into it. Then to use
71 'perf report' becomes:
73 ~/libexec/perf-core/perf-with-kcore report pt_ls
75 Because samples are synthesized after-the-fact, the sampling period can be
76 selected for reporting. e.g. sample every microsecond
78 ~/libexec/perf-core/perf-with-kcore report pt_ls --itrace=i1usge
80 See the sections below for more information about the --itrace option.
82 Beware the smaller the period, the more samples that are produced, and the
83 longer it takes to process them.
85 Also note that the coarseness of Intel PT timing information will start to
86 distort the statistical value of the sampling as the sampling period becomes
89 To represent software control flow, "branches" samples are produced. By default
90 a branch sample is synthesized for every single branch. To get an idea what
91 data is available you can use the 'perf script' tool with no parameters, which
92 will list all the samples.
94 perf record -e intel_pt//u ls
97 An interesting field that is not printed by default is 'flags' which can be
100 perf script -Fcomm,tid,pid,time,cpu,event,trace,ip,sym,dso,addr,symoff,flags
102 The flags are "bcrosyiABEx" which stand for branch, call, return, conditional,
103 system, asynchronous, interrupt, transaction abort, trace begin, trace end, and
104 in transaction, respectively.
106 While it is possible to create scripts to analyze the data, an alternative
107 approach is available to export the data to a postgresql database. Refer to
108 script export-to-postgresql.py for more details, and to script
109 call-graph-from-postgresql.py for an example of using the database.
111 As mentioned above, it is easy to capture too much data. One way to limit the
112 data captured is to use 'snapshot' mode which is explained further below.
113 Refer to 'new snapshot option' and 'Intel PT modes of operation' further below.
115 Another problem that will be experienced is decoder errors. They can be caused
116 by inability to access the executed image, self-modified or JIT-ed code, or the
117 inability to match side-band information (such as context switches and mmaps)
118 which results in the decoder not knowing what code was executed.
120 There is also the problem of perf not being able to copy the data fast enough,
121 resulting in data lost because the buffer was full. See 'Buffer handling' below
131 The Intel PT kernel driver creates a new PMU for Intel PT. PMU events are
132 selected by providing the PMU name followed by the "config" separated by slashes.
133 An enhancement has been made to allow default "config" e.g. the option
137 will use a default config value. Currently that is the same as
139 -e intel_pt/tsc,noretcomp=0/
143 -e intel_pt/tsc=1,noretcomp=0/
145 Note there are now new config terms - see section 'config terms' further below.
147 The config terms are listed in /sys/devices/intel_pt/format. They are bit
148 fields within the config member of the struct perf_event_attr which is
149 passed to the kernel by the perf_event_open system call. They correspond to bit
150 fields in the IA32_RTIT_CTL MSR. Here is a list of them and their definitions:
152 $ grep -H . /sys/bus/event_source/devices/intel_pt/format/*
153 /sys/bus/event_source/devices/intel_pt/format/cyc:config:1
154 /sys/bus/event_source/devices/intel_pt/format/cyc_thresh:config:19-22
155 /sys/bus/event_source/devices/intel_pt/format/mtc:config:9
156 /sys/bus/event_source/devices/intel_pt/format/mtc_period:config:14-17
157 /sys/bus/event_source/devices/intel_pt/format/noretcomp:config:11
158 /sys/bus/event_source/devices/intel_pt/format/psb_period:config:24-27
159 /sys/bus/event_source/devices/intel_pt/format/tsc:config:10
161 Note that the default config must be overridden for each term i.e.
163 -e intel_pt/noretcomp=0/
167 -e intel_pt/tsc=1,noretcomp=0/
169 So, to disable TSC packets use:
173 It is also possible to specify the config value explicitly:
175 -e intel_pt/config=0x400/
177 Note that, as with all events, the event is suffixed with event modifiers:
186 'h', 'G' and 'H' are for virtualization which is not supported by Intel PT.
187 'p' is also not relevant to Intel PT. So only options 'u' and 'k' are
188 meaningful for Intel PT.
190 perf_event_attr is displayed if the -vv option is used e.g.
192 ------------------------------------------------------------
197 { sample_period, sample_freq } 1
198 sample_type IP|TID|TIME|CPU|IDENTIFIER
206 ------------------------------------------------------------
207 sys_perf_event_open: pid 31104 cpu 0 group_fd -1 flags 0x8
208 sys_perf_event_open: pid 31104 cpu 1 group_fd -1 flags 0x8
209 sys_perf_event_open: pid 31104 cpu 2 group_fd -1 flags 0x8
210 sys_perf_event_open: pid 31104 cpu 3 group_fd -1 flags 0x8
211 ------------------------------------------------------------
217 The June 2015 version of Intel 64 and IA-32 Architectures Software Developer
218 Manuals, Chapter 36 Intel Processor Trace, defined new Intel PT features.
219 Some of the features are reflect in new config terms. All the config terms are
222 tsc Always supported. Produces TSC timestamp packets to provide
223 timing information. In some cases it is possible to decode
224 without timing information, for example a per-thread context
225 that does not overlap executable memory maps.
227 The default config selects tsc (i.e. tsc=1).
229 noretcomp Always supported. Disables "return compression" so a TIP packet
230 is produced when a function returns. Causes more packets to be
231 produced but might make decoding more reliable.
233 The default config does not select noretcomp (i.e. noretcomp=0).
235 psb_period Allows the frequency of PSB packets to be specified.
237 The PSB packet is a synchronization packet that provides a
238 starting point for decoding or recovery from errors.
240 Support for psb_period is indicated by:
242 /sys/bus/event_source/devices/intel_pt/caps/psb_cyc
244 which contains "1" if the feature is supported and "0"
247 Valid values are given by:
249 /sys/bus/event_source/devices/intel_pt/caps/psb_periods
251 which contains a hexadecimal value, the bits of which represent
252 valid values e.g. bit 2 set means value 2 is valid.
254 The psb_period value is converted to the approximate number of
255 trace bytes between PSB packets as:
259 e.g. value 3 means 16KiB bytes between PSBs
261 If an invalid value is entered, the error message
262 will give a list of valid values e.g.
264 $ perf record -e intel_pt/psb_period=15/u uname
265 Invalid psb_period for intel_pt. Valid values are: 0-5
267 If MTC packets are selected, the default config selects a value
268 of 3 (i.e. psb_period=3) or the nearest lower value that is
269 supported (0 is always supported). Otherwise the default is 0.
271 If decoding is expected to be reliable and the buffer is large
272 then a large PSB period can be used.
274 Because a TSC packet is produced with PSB, the PSB period can
275 also affect the granularity to timing information in the absence
278 mtc Produces MTC timing packets.
280 MTC packets provide finer grain timestamp information than TSC
281 packets. MTC packets record time using the hardware crystal
282 clock (CTC) which is related to TSC packets using a TMA packet.
284 Support for this feature is indicated by:
286 /sys/bus/event_source/devices/intel_pt/caps/mtc
288 which contains "1" if the feature is supported and
291 The frequency of MTC packets can also be specified - see
294 mtc_period Specifies how frequently MTC packets are produced - see mtc
295 above for how to determine if MTC packets are supported.
297 Valid values are given by:
299 /sys/bus/event_source/devices/intel_pt/caps/mtc_periods
301 which contains a hexadecimal value, the bits of which represent
302 valid values e.g. bit 2 set means value 2 is valid.
304 The mtc_period value is converted to the MTC frequency as:
306 CTC-frequency / (2 ^ value)
308 e.g. value 3 means one eighth of CTC-frequency
310 Where CTC is the hardware crystal clock, the frequency of which
311 can be related to TSC via values provided in cpuid leaf 0x15.
313 If an invalid value is entered, the error message
314 will give a list of valid values e.g.
316 $ perf record -e intel_pt/mtc_period=15/u uname
317 Invalid mtc_period for intel_pt. Valid values are: 0,3,6,9
319 The default value is 3 or the nearest lower value
320 that is supported (0 is always supported).
322 cyc Produces CYC timing packets.
324 CYC packets provide even finer grain timestamp information than
325 MTC and TSC packets. A CYC packet contains the number of CPU
326 cycles since the last CYC packet. Unlike MTC and TSC packets,
327 CYC packets are only sent when another packet is also sent.
329 Support for this feature is indicated by:
331 /sys/bus/event_source/devices/intel_pt/caps/psb_cyc
333 which contains "1" if the feature is supported and
336 The number of CYC packets produced can be reduced by specifying
337 a threshold - see cyc_thresh below.
339 cyc_thresh Specifies how frequently CYC packets are produced - see cyc
340 above for how to determine if CYC packets are supported.
342 Valid cyc_thresh values are given by:
344 /sys/bus/event_source/devices/intel_pt/caps/cycle_thresholds
346 which contains a hexadecimal value, the bits of which represent
347 valid values e.g. bit 2 set means value 2 is valid.
349 The cyc_thresh value represents the minimum number of CPU cycles
350 that must have passed before a CYC packet can be sent. The
351 number of CPU cycles is:
355 e.g. value 4 means 8 CPU cycles must pass before a CYC packet
356 can be sent. Note a CYC packet is still only sent when another
357 packet is sent, not at, e.g. every 8 CPU cycles.
359 If an invalid value is entered, the error message
360 will give a list of valid values e.g.
362 $ perf record -e intel_pt/cyc,cyc_thresh=15/u uname
363 Invalid cyc_thresh for intel_pt. Valid values are: 0-12
365 CYC packets are not requested by default.
371 The difference between full trace and snapshot from the kernel's perspective is
372 that in full trace we don't overwrite trace data that the user hasn't collected
373 yet (and indicated that by advancing aux_tail), whereas in snapshot mode we let
374 the trace run and overwrite older data in the buffer so that whenever something
375 interesting happens, we can stop it and grab a snapshot of what was going on
376 around that interesting moment.
378 To select snapshot mode a new option has been added:
382 Optionally it can be followed by the snapshot size e.g.
386 The default snapshot size is the auxtrace mmap size. If neither auxtrace mmap size
387 nor snapshot size is specified, then the default is 4MiB for privileged users
388 (or if /proc/sys/kernel/perf_event_paranoid < 0), 128KiB for unprivileged users.
389 If an unprivileged user does not specify mmap pages, the mmap pages will be
390 reduced as described in the 'new auxtrace mmap size option' section below.
392 The snapshot size is displayed if the option -vv is used e.g.
394 Intel PT snapshot size: %zu
397 new auxtrace mmap size option
398 ---------------------------
400 Intel PT buffer size is specified by an addition to the -m option e.g.
404 selects a buffer size of 16 pages i.e. 64KiB.
406 Note that the existing functionality of -m is unchanged. The auxtrace mmap size
407 is specified by the optional addition of a comma and the value.
409 The default auxtrace mmap size for Intel PT is 4MiB/page_size for privileged users
410 (or if /proc/sys/kernel/perf_event_paranoid < 0), 128KiB for unprivileged users.
411 If an unprivileged user does not specify mmap pages, the mmap pages will be
412 reduced from the default 512KiB/page_size to 256KiB/page_size, otherwise the
413 user is likely to get an error as they exceed their mlock limit (Max locked
414 memory as shown in /proc/self/limits). Note that perf does not count the first
415 512KiB (actually /proc/sys/kernel/perf_event_mlock_kb minus 1 page) per cpu
416 against the mlock limit so an unprivileged user is allowed 512KiB per cpu plus
417 their mlock limit (which defaults to 64KiB but is not multiplied by the number
420 In full-trace mode, powers of two are allowed for buffer size, with a minimum
421 size of 2 pages. In snapshot mode, it is the same but the minimum size is
424 The mmap size and auxtrace mmap size are displayed if the -vv option is used e.g.
427 auxtrace mmap length 4198400
430 Intel PT modes of operation
431 ---------------------------
433 Intel PT can be used in 2 modes:
437 Full-trace mode traces continuously e.g.
439 perf record -e intel_pt//u uname
441 Snapshot mode captures the available data when a signal is sent e.g.
443 perf record -v -e intel_pt//u -S ./loopy 1000000000 &
446 Recording AUX area tracing snapshot
448 Note that the signal sent is SIGUSR2.
449 Note that "Recording AUX area tracing snapshot" is displayed because the -v
452 The 2 modes cannot be used together.
458 There may be buffer limitations (i.e. single ToPa entry) which means that actual
459 buffer sizes are limited to powers of 2 up to 4MiB (MAX_ORDER). In order to
460 provide other sizes, and in particular an arbitrarily large size, multiple
461 buffers are logically concatenated. However an interrupt must be used to switch
462 between buffers. That has two potential problems:
463 a) the interrupt may not be handled in time so that the current buffer
464 becomes full and some trace data is lost.
465 b) the interrupts may slow the system and affect the performance
468 If trace data is lost, the driver sets 'truncated' in the PERF_RECORD_AUX event
469 which the tools report as an error.
471 In full-trace mode, the driver waits for data to be copied out before allowing
472 the (logical) buffer to wrap-around. If data is not copied out quickly enough,
473 again 'truncated' is set in the PERF_RECORD_AUX event. If the driver has to
474 wait, the intel_pt event gets disabled. Because it is difficult to know when
475 that happens, perf tools always re-enable the intel_pt event after copying out
479 Intel PT and build ids
480 ----------------------
482 By default "perf record" post-processes the event stream to find all build ids
483 for executables for all addresses sampled. Deliberately, Intel PT is not
484 decoded for that purpose (it would take too long). Instead the build ids for
485 all executables encountered (due to mmap, comm or task events) are included
486 in the perf.data file.
488 To see buildids included in the perf.data file use the command:
492 If the perf.data file contains Intel PT data, that is the same as:
494 perf buildid-list --with-hits
497 Snapshot mode and event disabling
498 ---------------------------------
500 In order to make a snapshot, the intel_pt event is disabled using an IOCTL,
501 namely PERF_EVENT_IOC_DISABLE. However doing that can also disable the
502 collection of side-band information. In order to prevent that, a dummy
503 software event has been introduced that permits tracking events (like mmaps) to
504 continue to be recorded while intel_pt is disabled. That is important to ensure
505 there is complete side-band information to allow the decoding of subsequent
508 A test has been created for that. To find the test:
512 23: Test using a dummy software event to keep tracking
517 23: Test using a dummy software event to keep tracking : Ok
520 perf record modes (nothing new here)
521 ------------------------------------
523 perf record essentially operates in one of three modes:
528 "per thread" mode is selected by -t or by --per-thread (with -p or -u or just a
530 "per cpu" is selected by -C or -a.
531 "workload only" mode is selected by not using the other options but providing a
532 command to run (i.e. the workload).
534 In per-thread mode an exact list of threads is traced. There is no inheritance.
535 Each thread has its own event buffer.
537 In per-cpu mode all processes (or processes from the selected cgroup i.e. -G
538 option, or processes selected with -p or -u) are traced. Each cpu has its own
539 buffer. Inheritance is allowed.
541 In workload-only mode, the workload is traced but with per-cpu buffers.
542 Inheritance is allowed. Note that you can now trace a workload in per-thread
543 mode by using the --per-thread option.
546 Privileged vs non-privileged users
547 ----------------------------------
549 Unless /proc/sys/kernel/perf_event_paranoid is set to -1, unprivileged users
550 have memory limits imposed upon them. That affects what buffer sizes they can
551 have as outlined above.
553 Unless /proc/sys/kernel/perf_event_paranoid is set to -1, unprivileged users are
554 not permitted to use tracepoints which means there is insufficient side-band
555 information to decode Intel PT in per-cpu mode, and potentially workload-only
556 mode too if the workload creates new processes.
558 Note also, that to use tracepoints, read-access to debugfs is required. So if
559 debugfs is not mounted or the user does not have read-access, it will again not
560 be possible to decode Intel PT in per-cpu mode.
563 sched_switch tracepoint
564 -----------------------
566 The sched_switch tracepoint is used to provide side-band data for Intel PT
567 decoding. sched_switch events are automatically added. e.g. the second event
570 $ perf record -vv -e intel_pt//u uname
571 ------------------------------------------------------------
576 { sample_period, sample_freq } 1
577 sample_type IP|TID|TIME|CPU|IDENTIFIER
585 ------------------------------------------------------------
586 sys_perf_event_open: pid 31104 cpu 0 group_fd -1 flags 0x8
587 sys_perf_event_open: pid 31104 cpu 1 group_fd -1 flags 0x8
588 sys_perf_event_open: pid 31104 cpu 2 group_fd -1 flags 0x8
589 sys_perf_event_open: pid 31104 cpu 3 group_fd -1 flags 0x8
590 ------------------------------------------------------------
595 { sample_period, sample_freq } 1
596 sample_type IP|TID|TIME|CPU|PERIOD|RAW|IDENTIFIER
601 ------------------------------------------------------------
602 sys_perf_event_open: pid -1 cpu 0 group_fd -1 flags 0x8
603 sys_perf_event_open: pid -1 cpu 1 group_fd -1 flags 0x8
604 sys_perf_event_open: pid -1 cpu 2 group_fd -1 flags 0x8
605 sys_perf_event_open: pid -1 cpu 3 group_fd -1 flags 0x8
606 ------------------------------------------------------------
611 { sample_period, sample_freq } 1
612 sample_type IP|TID|TIME|IDENTIFIER
625 ------------------------------------------------------------
626 sys_perf_event_open: pid 31104 cpu 0 group_fd -1 flags 0x8
627 sys_perf_event_open: pid 31104 cpu 1 group_fd -1 flags 0x8
628 sys_perf_event_open: pid 31104 cpu 2 group_fd -1 flags 0x8
629 sys_perf_event_open: pid 31104 cpu 3 group_fd -1 flags 0x8
631 AUX area mmap length 4194304
632 perf event ring buffer mmapped per cpu
633 Synthesizing auxtrace information
635 [ perf record: Woken up 1 times to write data ]
636 [ perf record: Captured and wrote 0.042 MB perf.data ]
638 Note, the sched_switch event is only added if the user is permitted to use it
639 and only in per-cpu mode.
641 Note also, the sched_switch event is only added if TSC packets are requested.
642 That is because, in the absence of timing information, the sched_switch events
643 cannot be matched against the Intel PT trace.
649 By default, perf script will decode trace data found in the perf.data file.
650 This can be further controlled by new option --itrace.
656 Having no option is the same as
660 which, in turn, is the same as
666 i synthesize "instructions" events
667 b synthesize "branches" events
668 x synthesize "transactions" events
669 c synthesize branches events (calls only)
670 r synthesize branches events (returns only)
671 e synthesize tracing error events
673 g synthesize a call chain (use with i or x)
674 l synthesize last branch entries (use with i or x)
676 "Instructions" events look like they were recorded by "perf record -e
679 "Branches" events look like they were recorded by "perf record -e branches". "c"
680 and "r" can be combined to get calls and returns.
682 "Transactions" events correspond to the start or end of transactions. The
683 'flags' field can be used in perf script to determine whether the event is a
684 tranasaction start, commit or abort.
686 Error events are new. They show where the decoder lost the trace. Error events
687 are quite important. Users must know if what they are seeing is a complete
690 The "d" option will cause the creation of a file "intel_pt.log" containing all
691 decoded packets and instructions. Note that this option slows down the decoder
692 and that the resulting file may be very large.
694 In addition, the period of the "instructions" event can be specified. e.g.
698 sets the period to 10us i.e. one instruction sample is synthesized for each 10
699 microseconds of trace. Alternatives to "us" are "ms" (milliseconds),
700 "ns" (nanoseconds), "t" (TSC ticks) or "i" (instructions).
702 "ms", "us" and "ns" are converted to TSC ticks.
704 The timing information included with Intel PT does not give the time of every
705 instruction. Consequently, for the purpose of sampling, the decoder estimates
706 the time since the last timing packet based on 1 tick per instruction. The time
707 on the sample is *not* adjusted and reflects the last known value of TSC.
709 For Intel PT, the default period is 100us.
711 Setting it to a zero period means "as often as possible".
713 In the case of Intel PT that is the same as a period of 1 and a unit of
714 'instructions' (i.e. --itrace=i1i).
716 Also the call chain size (default 16, max. 1024) for instructions or
717 transactions events can be specified. e.g.
722 Also the number of last branch entries (default 64, max. 1024) for instructions or
723 transactions events can be specified. e.g.
728 Note that last branch entries are cleared for each sample, so there is no overlap
729 from one sample to the next.
731 To disable trace decoding entirely, use the option --no-itrace.
737 perf script has an option (-D) to "dump" the events i.e. display the binary
740 When -D is used, Intel PT packets are displayed. The packet decoder does not
741 pay attention to PSB packets, but just decodes the bytes - so the packets seen
742 by the actual decoder may not be identical in places where the data is corrupt.
743 One example of that would be when the buffer-switching interrupt has been too
744 slow, and the buffer has been filled completely. In that case, the last packet
745 in the buffer might be truncated and immediately followed by a PSB as the trace
746 continues in the next buffer.
748 To disable the display of Intel PT packets, combine the -D option with
755 By default, perf report will decode trace data found in the perf.data file.
756 This can be further controlled by new option --itrace exactly the same as
757 perf script, with the exception that the default is --itrace=igxe.
763 perf inject also accepts the --itrace option in which case tracing data is
764 removed and replaced with the synthesized events. e.g.
766 perf inject --itrace -i perf.data -o perf.data.new
768 Below is an example of using Intel PT with autofdo. It requires autofdo
769 (https://github.com/google/autofdo) and gcc version 5. The bubble
770 sort example is from the AutoFDO tutorial (https://gcc.gnu.org/wiki/AutoFDO/Tutorial)
771 amended to take the number of elements as a parameter.
773 $ gcc-5 -O3 sort.c -o sort_optimized
774 $ ./sort_optimized 30000
775 Bubble sorting array of 30000 elements
782 $ perf record -e intel_pt//u ./sort 3000
783 Bubble sorting array of 3000 elements
785 [ perf record: Woken up 2 times to write data ]
786 [ perf record: Captured and wrote 3.939 MB perf.data ]
787 $ perf inject -i perf.data -o inj --itrace=i100usle --strip
788 $ ./create_gcov --binary=./sort --profile=inj --gcov=sort.gcov -gcov_version=1
789 $ gcc-5 -O3 -fauto-profile=sort.gcov sort.c -o sort_autofdo
790 $ ./sort_autofdo 30000
791 Bubble sorting array of 30000 elements
794 Note there is currently no advantage to using Intel PT instead of LBR, but
795 that may change in the future if greater use is made of the data.