Commit e3ff42bd authored by Adrian Hunter's avatar Adrian Hunter Committed by Arnaldo Carvalho de Melo

perf intel-pt: Parse VM Time Correlation options and set up decoding

Add parsing and validation of VM Time Correlation options, and pass
parameters to the decoder. Also update the Intel PT documentation
accordingly.
Signed-off-by: default avatarAdrian Hunter <adrian.hunter@intel.com>
Reviewed-by: default avatarAndi Kleen <ak@linux.intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: https://lore.kernel.org/r/20210430070309.17624-13-adrian.hunter@intel.comSigned-off-by: default avatarArnaldo Carvalho de Melo <acme@redhat.com>
parent fa8f949d
...@@ -1155,8 +1155,9 @@ include::build-xed.txt[] ...@@ -1155,8 +1155,9 @@ include::build-xed.txt[]
Tracing Virtual Machines Tracing Virtual Machines
------------------------ ------------------------
Currently, only kernel tracing is supported and only with "timeless" decoding Currently, only kernel tracing is supported and only with either "timeless" decoding
i.e. no TSC timestamps (i.e. no TSC timestamps) or VM Time Correlation. VM Time Correlation is an extra step
using 'perf inject' and requires unchanging VMX TSC Offset and no VMX TSC Scaling.
Other limitations and caveats Other limitations and caveats
...@@ -1167,7 +1168,7 @@ Other limitations and caveats ...@@ -1167,7 +1168,7 @@ Other limitations and caveats
Guest VCPU is unknown but may be able to be inferred from the host thread Guest VCPU is unknown but may be able to be inferred from the host thread
Callchains are not supported Callchains are not supported
Example Example using "timeless" decoding
Start VM Start VM
...@@ -1231,6 +1232,107 @@ perf script can be used to provide an instruction trace ...@@ -1231,6 +1232,107 @@ perf script can be used to provide an instruction trace
:1440 1440 ffffffffbb74603c clockevents_program_event+0x4c ([guest.kernel.kallsyms]) popq %rbx :1440 1440 ffffffffbb74603c clockevents_program_event+0x4c ([guest.kernel.kallsyms]) popq %rbx
:1440 1440 ffffffffbb74603d clockevents_program_event+0x4d ([guest.kernel.kallsyms]) popq %r12 :1440 1440 ffffffffbb74603d clockevents_program_event+0x4d ([guest.kernel.kallsyms]) popq %r12
Example using VM Time Correlation
Start VM
$ sudo virsh start kubuntu20.04
Domain kubuntu20.04 started
Mount the guest file system. Note sshfs needs -o direct_io to enable reading of proc files. root access is needed to read /proc/kcore.
$ mkdir -p vm0
$ sshfs -o direct_io root@vm0:/ vm0
Copy the guest /proc/kallsyms, /proc/modules and /proc/kcore
$ perf buildid-cache -v --kcore vm0/proc/kcore
same kcore found in /home/user/.debug/[kernel.kcore]/cc9c55a98c5e4ec0aeda69302554aabed5cd6491/2021021312450777
$ KALLSYMS=/home/user/.debug/\[kernel.kcore\]/cc9c55a98c5e4ec0aeda69302554aabed5cd6491/2021021312450777/kallsyms
Find the VM process
$ ps -eLl | grep 'KVM\|PID'
F S UID PID PPID LWP C PRI NI ADDR SZ WCHAN TTY TIME CMD
3 S 64055 16998 1 17005 13 80 0 - 1818189 - ? 00:00:16 CPU 0/KVM
3 S 64055 16998 1 17006 4 80 0 - 1818189 - ? 00:00:05 CPU 1/KVM
3 S 64055 16998 1 17007 3 80 0 - 1818189 - ? 00:00:04 CPU 2/KVM
3 S 64055 16998 1 17008 4 80 0 - 1818189 - ? 00:00:05 CPU 3/KVM
Start an open-ended perf record, tracing the VM process, do something on the VM, and then ctrl-C to stop.
IPC can be determined, hence cyc=1 can be added.
Only kernel decoding is supported, so 'k' must be specified.
Intel PT traces both the host and the guest so --guest and --host need to be specified.
$ sudo perf kvm --guest --host --guestkallsyms $KALLSYMS record --kcore -e intel_pt/cyc=1/k -p 16998
^C[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 9.041 MB perf.data.kvm ]
Now 'perf inject' can be used to determine the VMX TCS Offset. Note, Intel PT TSC packets are
only 7-bytes, so the TSC Offset might differ from the actual value in the 8th byte. That will
have no effect i.e. the resulting timestamps will be correct anyway.
$ perf inject -i perf.data.kvm --vm-time-correlation=dry-run
ERROR: Unknown TSC Offset for VMCS 0x1bff6a
VMCS: 0x1bff6a TSC Offset 0xffffe42722c64c41
ERROR: Unknown TSC Offset for VMCS 0x1cbc08
VMCS: 0x1cbc08 TSC Offset 0xffffe42722c64c41
ERROR: Unknown TSC Offset for VMCS 0x1c3ce8
VMCS: 0x1c3ce8 TSC Offset 0xffffe42722c64c41
ERROR: Unknown TSC Offset for VMCS 0x1cbce9
VMCS: 0x1cbce9 TSC Offset 0xffffe42722c64c41
Each virtual CPU has a different Virtual Machine Control Structure (VMCS)
shown above with the calculated TSC Offset. For an unchanging TSC Offset
they should all be the same for the same virtual machine.
Now that the TSC Offset is known, it can be provided to 'perf inject'
$ perf inject -i perf.data.kvm --vm-time-correlation="dry-run 0xffffe42722c64c41"
Note the options for 'perf inject' --vm-time-correlation are:
[ dry-run ] [ <TSC Offset> [ : <VMCS> [ , <VMCS> ]... ] ]...
So it is possible to specify different TSC Offsets for different VMCS.
The option "dry-run" will cause the file to be processed but without updating it.
Note it is also possible to get a intel_pt.log file by adding option --itrace=d
There were no errors so, do it for real
$ perf inject -i perf.data.kvm --vm-time-correlation=0xffffe42722c64c41 --force
'perf script' can be used to see if there are any decoder errors
$ perf script -i perf.data.kvm --guestkallsyms $KALLSYMS --itrace=e-o
There were none.
'perf script' can be used to provide an instruction trace showing timestamps
$ perf script -i perf.data.kvm --guestkallsyms $KALLSYMS --insn-trace --xed -F+ipc | grep -C10 vmresume | head -21
CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133cdd __vmx_vcpu_run+0x3d ([kernel.kallsyms]) movq 0x48(%rax), %r9
CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133ce1 __vmx_vcpu_run+0x41 ([kernel.kallsyms]) movq 0x50(%rax), %r10
CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133ce5 __vmx_vcpu_run+0x45 ([kernel.kallsyms]) movq 0x58(%rax), %r11
CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133ce9 __vmx_vcpu_run+0x49 ([kernel.kallsyms]) movq 0x60(%rax), %r12
CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133ced __vmx_vcpu_run+0x4d ([kernel.kallsyms]) movq 0x68(%rax), %r13
CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133cf1 __vmx_vcpu_run+0x51 ([kernel.kallsyms]) movq 0x70(%rax), %r14
CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133cf5 __vmx_vcpu_run+0x55 ([kernel.kallsyms]) movq 0x78(%rax), %r15
CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133cf9 __vmx_vcpu_run+0x59 ([kernel.kallsyms]) movq (%rax), %rax
CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133cfc __vmx_vcpu_run+0x5c ([kernel.kallsyms]) callq 0xffffffff82133c40
CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133c40 vmx_vmenter+0x0 ([kernel.kallsyms]) jz 0xffffffff82133c46
CPU 1/KVM 17006 [001] 11500.262866075: ffffffff82133c42 vmx_vmenter+0x2 ([kernel.kallsyms]) vmresume IPC: 0.05 (40/769)
:17006 17006 [001] 11500.262869216: ffffffff82200cb0 asm_sysvec_apic_timer_interrupt+0x0 ([guest.kernel.kallsyms]) clac
:17006 17006 [001] 11500.262869216: ffffffff82200cb3 asm_sysvec_apic_timer_interrupt+0x3 ([guest.kernel.kallsyms]) pushq $0xffffffffffffffff
:17006 17006 [001] 11500.262869216: ffffffff82200cb5 asm_sysvec_apic_timer_interrupt+0x5 ([guest.kernel.kallsyms]) callq 0xffffffff82201160
:17006 17006 [001] 11500.262869216: ffffffff82201160 error_entry+0x0 ([guest.kernel.kallsyms]) cld
:17006 17006 [001] 11500.262869216: ffffffff82201161 error_entry+0x1 ([guest.kernel.kallsyms]) pushq %rsi
:17006 17006 [001] 11500.262869216: ffffffff82201162 error_entry+0x2 ([guest.kernel.kallsyms]) movq 0x8(%rsp), %rsi
:17006 17006 [001] 11500.262869216: ffffffff82201167 error_entry+0x7 ([guest.kernel.kallsyms]) movq %rdi, 0x8(%rsp)
:17006 17006 [001] 11500.262869216: ffffffff8220116c error_entry+0xc ([guest.kernel.kallsyms]) pushq %rdx
:17006 17006 [001] 11500.262869216: ffffffff8220116d error_entry+0xd ([guest.kernel.kallsyms]) pushq %rcx
:17006 17006 [001] 11500.262869216: ffffffff8220116e error_entry+0xe ([guest.kernel.kallsyms]) pushq %rax
SEE ALSO SEE ALSO
......
...@@ -1018,6 +1018,19 @@ static bool intel_pt_have_tsc(struct intel_pt *pt) ...@@ -1018,6 +1018,19 @@ static bool intel_pt_have_tsc(struct intel_pt *pt)
return have_tsc; return have_tsc;
} }
static bool intel_pt_have_mtc(struct intel_pt *pt)
{
struct evsel *evsel;
u64 config;
evlist__for_each_entry(pt->session->evlist, evsel) {
if (intel_pt_get_config(pt, &evsel->core.attr, &config) &&
(config & pt->mtc_bit))
return true;
}
return false;
}
static bool intel_pt_sampling_mode(struct intel_pt *pt) static bool intel_pt_sampling_mode(struct intel_pt *pt)
{ {
struct evsel *evsel; struct evsel *evsel;
...@@ -1182,6 +1195,8 @@ static struct intel_pt_queue *intel_pt_alloc_queue(struct intel_pt *pt, ...@@ -1182,6 +1195,8 @@ static struct intel_pt_queue *intel_pt_alloc_queue(struct intel_pt *pt,
params.tsc_ctc_ratio_n = pt->tsc_ctc_ratio_n; params.tsc_ctc_ratio_n = pt->tsc_ctc_ratio_n;
params.tsc_ctc_ratio_d = pt->tsc_ctc_ratio_d; params.tsc_ctc_ratio_d = pt->tsc_ctc_ratio_d;
params.quick = pt->synth_opts.quick; params.quick = pt->synth_opts.quick;
params.vm_time_correlation = pt->synth_opts.vm_time_correlation;
params.vm_tm_corr_dry_run = pt->synth_opts.vm_tm_corr_dry_run;
params.first_timestamp = pt->first_timestamp; params.first_timestamp = pt->first_timestamp;
if (pt->filts.cnt > 0) if (pt->filts.cnt > 0)
...@@ -2465,7 +2480,7 @@ static int intel_pt_run_decoder(struct intel_pt_queue *ptq, u64 *timestamp) ...@@ -2465,7 +2480,7 @@ static int intel_pt_run_decoder(struct intel_pt_queue *ptq, u64 *timestamp)
if (pt->per_cpu_mmaps && if (pt->per_cpu_mmaps &&
(pt->have_sched_switch == 1 || pt->have_sched_switch == 3) && (pt->have_sched_switch == 1 || pt->have_sched_switch == 3) &&
!pt->timeless_decoding && intel_pt_tracing_kernel(pt) && !pt->timeless_decoding && intel_pt_tracing_kernel(pt) &&
!pt->sampling_mode) { !pt->sampling_mode && !pt->synth_opts.vm_time_correlation) {
pt->switch_ip = intel_pt_switch_ip(pt, &pt->ptss_ip); pt->switch_ip = intel_pt_switch_ip(pt, &pt->ptss_ip);
if (pt->switch_ip) { if (pt->switch_ip) {
intel_pt_log("switch_ip: %"PRIx64" ptss_ip: %"PRIx64"\n", intel_pt_log("switch_ip: %"PRIx64" ptss_ip: %"PRIx64"\n",
...@@ -3496,6 +3511,65 @@ static int intel_pt_setup_time_ranges(struct intel_pt *pt, ...@@ -3496,6 +3511,65 @@ static int intel_pt_setup_time_ranges(struct intel_pt *pt,
return 0; return 0;
} }
static int intel_pt_parse_vm_tm_corr_arg(struct intel_pt *pt, char **args)
{
struct intel_pt_vmcs_info *vmcs_info;
u64 tsc_offset, vmcs;
char *p = *args;
errno = 0;
p = skip_spaces(p);
if (!*p)
return 1;
tsc_offset = strtoull(p, &p, 0);
if (errno)
return -errno;
p = skip_spaces(p);
if (*p != ':') {
pt->dflt_tsc_offset = tsc_offset;
*args = p;
return 0;
}
while (1) {
vmcs = strtoull(p, &p, 0);
if (errno)
return -errno;
if (!vmcs)
return -EINVAL;
vmcs_info = intel_pt_findnew_vmcs(&pt->vmcs_info, vmcs, tsc_offset);
if (!vmcs_info)
return -ENOMEM;
p = skip_spaces(p);
if (*p != ',')
break;
p += 1;
}
*args = p;
return 0;
}
static int intel_pt_parse_vm_tm_corr_args(struct intel_pt *pt)
{
char *args = pt->synth_opts.vm_tm_corr_args;
int ret;
if (!args)
return 0;
do {
ret = intel_pt_parse_vm_tm_corr_arg(pt, &args);
} while (!ret);
if (ret < 0) {
pr_err("Failed to parse VM Time Correlation options\n");
return ret;
}
return 0;
}
static const char * const intel_pt_info_fmts[] = { static const char * const intel_pt_info_fmts[] = {
[INTEL_PT_PMU_TYPE] = " PMU Type %"PRId64"\n", [INTEL_PT_PMU_TYPE] = " PMU Type %"PRId64"\n",
[INTEL_PT_TIME_SHIFT] = " Time Shift %"PRIu64"\n", [INTEL_PT_TIME_SHIFT] = " Time Shift %"PRIu64"\n",
...@@ -3667,6 +3741,28 @@ int intel_pt_process_auxtrace_info(union perf_event *event, ...@@ -3667,6 +3741,28 @@ int intel_pt_process_auxtrace_info(union perf_event *event,
pt->sampling_mode = intel_pt_sampling_mode(pt); pt->sampling_mode = intel_pt_sampling_mode(pt);
pt->est_tsc = !pt->timeless_decoding; pt->est_tsc = !pt->timeless_decoding;
if (pt->synth_opts.vm_time_correlation) {
if (pt->timeless_decoding) {
pr_err("Intel PT has no time information for VM Time Correlation\n");
err = -EINVAL;
goto err_free_queues;
}
if (session->itrace_synth_opts->ptime_range) {
pr_err("Time ranges cannot be specified with VM Time Correlation\n");
err = -EINVAL;
goto err_free_queues;
}
/* Currently TSC Offset is calculated using MTC packets */
if (!intel_pt_have_mtc(pt)) {
pr_err("MTC packets must have been enabled for VM Time Correlation\n");
err = -EINVAL;
goto err_free_queues;
}
err = intel_pt_parse_vm_tm_corr_args(pt);
if (err)
goto err_free_queues;
}
pt->unknown_thread = thread__new(999999999, 999999999); pt->unknown_thread = thread__new(999999999, 999999999);
if (!pt->unknown_thread) { if (!pt->unknown_thread) {
err = -ENOMEM; err = -ENOMEM;
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment