Replaying trace takes a lot longer than recording

[d-netto]

Summary

We're trying to use rr to debug a crash in Julia's garbage collector. However, replaying the trace takes too long, and makes debugging infeasible.

Commands

Recording the program with:

rr record --chaos julia -g2 mwe.jl

takes around 5-6min.

I've left rr replaying the program with:

rr replay

and it takes more than 15min.

Not sure if it matters, but I've also run this command in GDB:

handle SIGSEGV nostop noprint

to ignore segmentation faults than happen when Julia stops threads for GC.

Microarchitecture

Architecture:                x86_64
  CPU op-mode(s):            32-bit, 64-bit
  Address sizes:             46 bits physical, 48 bits virtual
  Byte Order:                Little Endian
CPU(s):                      36
  On-line CPU(s) list:       0-35
Vendor ID:                   GenuineIntel
  Model name:                Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
    CPU family:              6
    Model:                   85
    Thread(s) per core:      2
    Core(s) per socket:      18
    Socket(s):               1
    Stepping:                4
    BogoMIPS:                5999.99

Reproducer

Unfortunately I can't share mwe.jl here due to licensing, but @Keno might have access to it.

[rocallahan]

You're using chaos mode so you would probably benefit from the bpf replay acceleration work that @khuey did. Try building rr with cmake -Dbpf=TRUE to cmake. You will probably need to run the replay as root. Kyle might have additional instructions.

[kpamnany]

Used cmake -DCMAKE_BUILD_TYPE=Release -Dbpf=TRUE ../rr. make -j8 (tried on both 5.9.0 and on master):

/home/ubuntu/rr/src/PerfCounters.cc: In static member function ‘static std::shared_ptr<rr::BpfAccelerator> rr::BpfAccelerator::get_or_create()’:
/home/ubuntu/rr/src/PerfCounters.cc:1270:32: error: ‘bpf_object__next_program’ was not declared in this scope; did you mean ‘bpf_object__unpin_programs’?
 1270 |     struct bpf_program* prog = bpf_object__next_program(obj, nullptr);
      |                                ^~~~~~~~~~~~~~~~~~~~~~~~
      |                                bpf_object__unpin_programs

Machine is an EC2 c5.9xlarge: Linux ip-XXX-XX-XX-XX 6.8.0-1035-aws #37~22.04.1-Ubuntu SMP.

libbpf-dev/jammy-updates,jammy-security,now 1:0.5.0-1ubuntu22.04.1 amd64 [installed]
  eBPF helper library (development files)

libbpf0/jammy-updates,jammy-security,now 1:0.5.0-1ubuntu22.04.1 amd64 [installed,automatic]
  eBPF helper library (shared library)

Looks like bpf_object__next_program is somewhat new? More instructions would be great.

[khuey]

What's your libbpf version?

[kpamnany]

Posted above. The latest available packaged version on Ubuntu 22.04 is 0.5.0. Which seems pretty old?

I reverted the change described here and rr seems to have built properly. I'll try to replay with this version shortly... but any other tips?

Since the time this issue was opened, I have gathered a couple of other rr traces with which replay seems to completely freeze -- even Ctrl+C does nothing.

[kpamnany]

It seems like building with libbpf hasn't really helped -- still stuck at Continuing. I'd expect that without the trace, it is hard to help? So we need Keno to look at what's going on. :-/

Will it help any to record with a libbpf-enabled rr or does it only help replay?

[kpamnany]

Another Q: if we'd like to help figure out why the replay is so slow, is there some profiling we can do that will be of use?

[rocallahan]

Sure. Any kind of profile would probably help.

[kpamnany]

Looking to run perf on rr, I find that there are multiple processes:

• rr replay
• gdb
• [rr:julia-debug] * 68

But the only process that is consuming any CPU is gdb. Attempting to perf record one of the [rr:julia-debug] processes:

$ sudo perf record -p 340622 -o replay-rr_julia-debug1.data -- sleep 120
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.083 MB replay-rr_julia-debug1.data (172 samples) ]

Any guidance? What would you like to see?

[khuey]

If you rr replay -a (i.e. without gdb) is it still slow?

[kpamnany]

Yes, it is still slow with -a, but it actually completed the replay. It took a little over 2 hours for a ~15 minute recording.

Without gdb, the rr process was using CPU, so I grabbed a couple of profiles, about 10 minutes apart. The second one completed right before the replay finished. Hopefully these are useful. Taken with: sudo perf record -p 341523 -o replay-rrX.data -g -- sleep 120.

The first and the second.

[kpamnany]

The perf profiles in the previous comment are of a build of rr master (commit) with the older version of libbpf on Ubuntu 22.04 (as I described here).

[rocallahan]

Sorry, the profile data by itself won't work since we would need rr symbols. Maybe after gathering the profile data run perf report and redirect the output to a file?

[Keno]

I've been provided access to this trace. One thing I'm observing is that the workload exhibits quite different behavior across its execution time, so not sure how useful a global profile will be. One issue I observed were a bunch of SCHED's into tight loops, but I expect the bpf acceleration will help there, so running that now. Will report back what else I find. We may want to build some additional tooling to graph execution speed vs real time to zero in on the periods where replay is slow - it's definitely not the whole trace.

[Keno]

bpf does not appear to be helping those SCHED's too much. Example log:

[ReplaySession] [event 844782] 165310: replaying SCHED; state  (none)
[ReplaySession]     (syscllbufsz:0, abrtcmt:0, locked:0)
[ReplaySession] advancing 1850435 ticks to reach 7375634543/0x4697800ab746
[ReplaySession]   programming interrupt for 1850309 ticks
[Task] resuming execution of 158485 with PTRACE_SYSEMU tick_period 1850309 wait 2
[PerfCounters] Resetting counters with period 1850309
[Task] going into blocking wait for 158485 ...
[Task]   Task 158485 changed status to 0x107f (STOP-SIGSTKFLT)
[Task]   (refreshing register cache)
[Task] Requesting registers from tracee 158485
[ReplaySession]   not same execution point: 109 ticks left (@0x4697800ab746) replay vs rec: rax 0xfe != (nil), rcx 0xffff != 0xe81c5b165b8b867c, rdx 0x40c43236c0ff != 0x7ee380000900, rsi 0x20 != (nil), r8 0x18 != 0xc, r13 0xffff != 0xff01, r14 0xb != 0x13, rip 0x4697800abaf4 != 0x4697800ab746
[ReplaySession]     breaking on target $ip
[Task] resuming execution of 158485 with PTRACE_SYSEMU tick_period -1 wait 2
[PerfCounters] Resetting counters with period 1152921504606846976
[Task] going into blocking wait for 158485 ...
[Task]   Task 158485 changed status to 0x57f (STOP-SIGTRAP)
[Task]   (refreshing register cache)
[Task] Requesting registers from tracee 158485
[ReplaySession]   not same execution point: 104 ticks left (@0x4697800ab746) replay vs rec: rcx 0x998fba6a7487810c != 0xe81c5b165b8b867c, r8 0xf != 0xc, r14 0xb != 0x13
[ReplaySession]   not same execution point: 104 ticks left (@0x4697800ab746) replay vs rec: rcx 0x998fba6a7487810c != 0xe81c5b165b8b867c, r8 0xf != 0xc, r14 0xb != 0x13
[ReplaySession]     fast-forwarded through 1 breakpoint hits with bpf
[ReplaySession]     trap was for target $ip
[ReplaySession]   not same execution point: 104 ticks left (@0x4697800ab746) replay vs rec: rcx 0x998fba6a7487810c != 0xe81c5b165b8b867c, r8 0xf != 0xc, r14 0xb != 0x13
[ReplaySession]     (fast-forwarding over target $ip)
[Task] resuming execution of 158485 with PTRACE_SINGLESTEP tick_period -1 wait 2
[PerfCounters] Resetting counters with period 1152921504606846976
[Task] going into blocking wait for 158485 ...
[Task]   Task 158485 changed status to 0x57f (STOP-SIGTRAP)
[Task]   (refreshing register cache)
[Task] Requesting registers from tracee 158485
[ReplaySession]   not same execution point: 104 ticks left (@0x4697800ab746) replay vs rec: rcx 0x998fba6a74878100 != 0xe81c5b165b8b867c, r8 0xf != 0xc, r14 0xb != 0x13, rip 0x4697800ab749 != 0x4697800ab746
[ReplaySession]   not same execution point: 104 ticks left (@0x4697800ab746) replay vs rec: rcx 0x998fba6a74878100 != 0xe81c5b165b8b867c, r8 0xf != 0xc, r14 0xb != 0x13, rip 0x4697800ab749 != 0x4697800ab746
[ReplaySession]     (SIGTRAP; stepi'd target $ip)
[ReplaySession]     breaking on target $ip
[Task] resuming execution of 158485 with PTRACE_SYSEMU tick_period -1 wait 2
[PerfCounters] Resetting counters with period 1152921504606846976
[Task] going into blocking wait for 158485 ...
[Task]   Task 158485 changed status to 0x57f (STOP-SIGTRAP)
[Task]   (refreshing register cache)
[Task] Requesting registers from tracee 158485
[ReplaySession]   not same execution point: 91 ticks left (@0x4697800ab746) replay vs rec: rcx 0x5d04e869f6c7472 != 0xe81c5b165b8b867c, r8 0x14 != 0xc, r14 0xc != 0x13
[ReplaySession]   not same execution point: 91 ticks left (@0x4697800ab746) replay vs rec: rcx 0x5d04e869f6c7472 != 0xe81c5b165b8b867c, r8 0x14 != 0xc, r14 0xc != 0x13
[ReplaySession]     fast-forwarded through 1 breakpoint hits with bpf
[ReplaySession]     trap was for target $ip
[ReplaySession]   not same execution point: 91 ticks left (@0x4697800ab746) replay vs rec: rcx 0x5d04e869f6c7472 != 0xe81c5b165b8b867c, r8 0x14 != 0xc, r14 0xc != 0x13
[ReplaySession]     (fast-forwarding over target $ip)
[Task] resuming execution of 158485 with PTRACE_SINGLESTEP tick_period -1 wait 2
[PerfCounters] Resetting counters with period 1152921504606846976
[Task] going into blocking wait for 158485 ...
[Task]   Task 158485 changed status to 0x57f (STOP-SIGTRAP)
[Task]   (refreshing register cache)
[Task] Requesting registers from tracee 158485
[ReplaySession]   not same execution point: 91 ticks left (@0x4697800ab746) replay vs rec: rcx 0x5d04e869f6c7400 != 0xe81c5b165b8b867c, r8 0x14 != 0xc, r14 0xc != 0x13, rip 0x4697800ab749 != 0x4697800ab746
[ReplaySession]   not same execution point: 91 ticks left (@0x4697800ab746) replay vs rec: rcx 0x5d04e869f6c7400 != 0xe81c5b165b8b867c, r8 0x14 != 0xc, r14 0xc != 0x13, rip 0x4697800ab749 != 0x4697800ab746
[ReplaySession]     (SIGTRAP; stepi'd target $ip)
[ReplaySession]     breaking on target $ip
[Task] resuming execution of 158485 with PTRACE_SYSEMU tick_period -1 wait 2
[PerfCounters] Resetting counters with period 1152921504606846976
[Task] going into blocking wait for 158485 ...
[Task]   Task 158485 changed status to 0x57f (STOP-SIGTRAP)
[Task]   (refreshing register cache)
[Task] Requesting registers from tracee 158485
[ReplaySession]   not same execution point: 78 ticks left (@0x4697800ab746) replay vs rec: rcx 0x49b1bf87d95873c8 != 0xe81c5b165b8b867c, r8 0x1c != 0xc, r14 0xd != 0x13
[ReplaySession]   not same execution point: 78 ticks left (@0x4697800ab746) replay vs rec: rcx 0x49b1bf87d95873c8 != 0xe81c5b165b8b867c, r8 0x1c != 0xc, r14 0xd != 0x13
[ReplaySession]     fast-forwarded through 1 breakpoint hits with bpf
[ReplaySession]     trap was for target $ip
[ReplaySession]   not same execution point: 78 ticks left (@0x4697800ab746) replay vs rec: rcx 0x49b1bf87d95873c8 != 0xe81c5b165b8b867c, r8 0x1c != 0xc, r14 0xd != 0x13
[ReplaySession]     (fast-forwarding over target $ip)
[Task] resuming execution of 158485 with PTRACE_SINGLESTEP tick_period -1 wait 2
[PerfCounters] Resetting counters with period 1152921504606846976
[Task] going into blocking wait for 158485 ...
[Task]   Task 158485 changed status to 0x57f (STOP-SIGTRAP)
[Task]   (refreshing register cache)
[Task] Requesting registers from tracee 158485
[ReplaySession]   not same execution point: 78 ticks left (@0x4697800ab746) replay vs rec: rcx 0x49b1bf87d9587300 != 0xe81c5b165b8b867c, r8 0x1c != 0xc, r14 0xd != 0x13, rip 0x4697800ab749 != 0x4697800ab746
[ReplaySession]   not same execution point: 78 ticks left (@0x4697800ab746) replay vs rec: rcx 0x49b1bf87d9587300 != 0xe81c5b165b8b867c, r8 0x1c != 0xc, r14 0xd != 0x13, rip 0x4697800ab749 != 0x4697800ab746
[ReplaySession]     (SIGTRAP; stepi'd target $ip)
[ReplaySession]     breaking on target $ip
[Task] resuming execution of 158485 with PTRACE_SYSEMU tick_period -1 wait 2
[PerfCounters] Resetting counters with period 1152921504606846976
[Task] going into blocking wait for 158485 ...
[Task]   Task 158485 changed status to 0x57f (STOP-SIGTRAP)
[Task]   (refreshing register cache)
[Task] Requesting registers from tracee 158485
[ReplaySession]   not same execution point: 65 ticks left (@0x4697800ab746) replay vs rec: rcx 0x479121d01565fc8e != 0xe81c5b165b8b867c, r8 0x3 != 0xc, r14 0xe != 0x13
[ReplaySession]   not same execution point: 65 ticks left (@0x4697800ab746) replay vs rec: rcx 0x479121d01565fc8e != 0xe81c5b165b8b867c, r8 0x3 != 0xc, r14 0xe != 0x13
[ReplaySession]     fast-forwarded through 1 breakpoint hits with bpf
[ReplaySession]     trap was for target $ip
[ReplaySession]   not same execution point: 65 ticks left (@0x4697800ab746) replay vs rec: rcx 0x479121d01565fc8e != 0xe81c5b165b8b867c, r8 0x3 != 0xc, r14 0xe != 0x13
[ReplaySession]     (fast-forwarding over target $ip)
[Task] resuming execution of 158485 with PTRACE_SINGLESTEP tick_period -1 wait 2
[PerfCounters] Resetting counters with period 1152921504606846976
[Task] going into blocking wait for 158485 ...
[Task]   Task 158485 changed status to 0x57f (STOP-SIGTRAP)
[Task]   (refreshing register cache)
[Task] Requesting registers from tracee 158485
[ReplaySession]   not same execution point: 65 ticks left (@0x4697800ab746) replay vs rec: rcx 0x479121d01565fc00 != 0xe81c5b165b8b867c, r8 0x3 != 0xc, r14 0xe != 0x13, rip 0x4697800ab749 != 0x4697800ab746
[ReplaySession]   not same execution point: 65 ticks left (@0x4697800ab746) replay vs rec: rcx 0x479121d01565fc00 != 0xe81c5b165b8b867c, r8 0x3 != 0xc, r14 0xe != 0x13, rip 0x4697800ab749 != 0x4697800ab746
[ReplaySession]     (SIGTRAP; stepi'd target $ip)
[ReplaySession]     breaking on target $ip
[Task] resuming execution of 158485 with PTRACE_SYSEMU tick_period -1 wait 2
[PerfCounters] Resetting counters with period 1152921504606846976
[Task] going into blocking wait for 158485 ...
[Task]   Task 158485 changed status to 0x57f (STOP-SIGTRAP)
[Task]   (refreshing register cache)
[Task] Requesting registers from tracee 158485
[ReplaySession]   not same execution point: 52 ticks left (@0x4697800ab746) replay vs rec: rcx 0xd318d3729cac2844 != 0xe81c5b165b8b867c, r8 0x14 != 0xc, r14 0xf != 0x13
[ReplaySession]   not same execution point: 52 ticks left (@0x4697800ab746) replay vs rec: rcx 0xd318d3729cac2844 != 0xe81c5b165b8b867c, r8 0x14 != 0xc, r14 0xf != 0x13
[ReplaySession]     fast-forwarded through 1 breakpoint hits with bpf
[ReplaySession]     trap was for target $ip
[ReplaySession]   not same execution point: 52 ticks left (@0x4697800ab746) replay vs rec: rcx 0xd318d3729cac2844 != 0xe81c5b165b8b867c, r8 0x14 != 0xc, r14 0xf != 0x13
[ReplaySession]     (fast-forwarding over target $ip)
[Task] resuming execution of 158485 with PTRACE_SINGLESTEP tick_period -1 wait 2
[PerfCounters] Resetting counters with period 1152921504606846976
[Task] going into blocking wait for 158485 ...
[Task]   Task 158485 changed status to 0x57f (STOP-SIGTRAP)
[Task]   (refreshing register cache)
[Task] Requesting registers from tracee 158485
[ReplaySession]   not same execution point: 52 ticks left (@0x4697800ab746) replay vs rec: rcx 0xd318d3729cac2800 != 0xe81c5b165b8b867c, r8 0x14 != 0xc, r14 0xf != 0x13, rip 0x4697800ab749 != 0x4697800ab746
[ReplaySession]   not same execution point: 52 ticks left (@0x4697800ab746) replay vs rec: rcx 0xd318d3729cac2800 != 0xe81c5b165b8b867c, r8 0x14 != 0xc, r14 0xf != 0x13, rip 0x4697800ab749 != 0x4697800ab746
[ReplaySession]     (SIGTRAP; stepi'd target $ip)
[ReplaySession]     breaking on target $ip
[Task] resuming execution of 158485 with PTRACE_SYSEMU tick_period -1 wait 2
[PerfCounters] Resetting counters with period 1152921504606846976
[Task] going into blocking wait for 158485 ...
[Task]   Task 158485 changed status to 0x57f (STOP-SIGTRAP)
[Task]   (refreshing register cache)
[Task] Requesting registers from tracee 158485
[ReplaySession]   not same execution point: 39 ticks left (@0x4697800ab746) replay vs rec: rcx 0x15a589718d9f0c6a != 0xe81c5b165b8b867c, r8 0x12 != 0xc, r14 0x10 != 0x13
[ReplaySession]   not same execution point: 39 ticks left (@0x4697800ab746) replay vs rec: rcx 0x15a589718d9f0c6a != 0xe81c5b165b8b867c, r8 0x12 != 0xc, r14 0x10 != 0x13
[ReplaySession]     fast-forwarded through 1 breakpoint hits with bpf
[ReplaySession]     trap was for target $ip
[ReplaySession]   not same execution point: 39 ticks left (@0x4697800ab746) replay vs rec: rcx 0x15a589718d9f0c6a != 0xe81c5b165b8b867c, r8 0x12 != 0xc, r14 0x10 != 0x13
[ReplaySession]     (fast-forwarding over target $ip)
[Task] resuming execution of 158485 with PTRACE_SINGLESTEP tick_period -1 wait 2
[PerfCounters] Resetting counters with period 1152921504606846976
[Task] going into blocking wait for 158485 ...
[Task]   Task 158485 changed status to 0x57f (STOP-SIGTRAP)
[Task]   (refreshing register cache)
[Task] Requesting registers from tracee 158485
[ReplaySession]   not same execution point: 39 ticks left (@0x4697800ab746) replay vs rec: rcx 0x15a589718d9f0c00 != 0xe81c5b165b8b867c, r8 0x12 != 0xc, r14 0x10 != 0x13, rip 0x4697800ab749 != 0x4697800ab746
[ReplaySession]   not same execution point: 39 ticks left (@0x4697800ab746) replay vs rec: rcx 0x15a589718d9f0c00 != 0xe81c5b165b8b867c, r8 0x12 != 0xc, r14 0x10 != 0x13, rip 0x4697800ab749 != 0x4697800ab746
[ReplaySession]     (SIGTRAP; stepi'd target $ip)
[ReplaySession]     breaking on target $ip
[Task] resuming execution of 158485 with PTRACE_SYSEMU tick_period -1 wait 2
[PerfCounters] Resetting counters with period 1152921504606846976
[Task] going into blocking wait for 158485 ...
[Task]   Task 158485 changed status to 0x57f (STOP-SIGTRAP)
[Task]   (refreshing register cache)
[Task] Requesting registers from tracee 158485
[ReplaySession]   not same execution point: 26 ticks left (@0x4697800ab746) replay vs rec: rcx 0xfd027aeeaebf1a80 != 0xe81c5b165b8b867c, r8 0x16 != 0xc, r14 0x11 != 0x13
[ReplaySession]   not same execution point: 26 ticks left (@0x4697800ab746) replay vs rec: rcx 0xfd027aeeaebf1a80 != 0xe81c5b165b8b867c, r8 0x16 != 0xc, r14 0x11 != 0x13
[ReplaySession]     fast-forwarded through 1 breakpoint hits with bpf
[ReplaySession]     trap was for target $ip
[ReplaySession]   not same execution point: 26 ticks left (@0x4697800ab746) replay vs rec: rcx 0xfd027aeeaebf1a80 != 0xe81c5b165b8b867c, r8 0x16 != 0xc, r14 0x11 != 0x13
[ReplaySession]     (fast-forwarding over target $ip)
[Task] resuming execution of 158485 with PTRACE_SINGLESTEP tick_period -1 wait 2
[PerfCounters] Resetting counters with period 1152921504606846976
[Task] going into blocking wait for 158485 ...
[Task]   Task 158485 changed status to 0x57f (STOP-SIGTRAP)
[Task]   (refreshing register cache)
[Task] Requesting registers from tracee 158485
[ReplaySession]   not same execution point: 26 ticks left (@0x4697800ab746) replay vs rec: rcx 0xfd027aeeaebf1a00 != 0xe81c5b165b8b867c, r8 0x16 != 0xc, r14 0x11 != 0x13, rip 0x4697800ab749 != 0x4697800ab746
[ReplaySession]   not same execution point: 26 ticks left (@0x4697800ab746) replay vs rec: rcx 0xfd027aeeaebf1a00 != 0xe81c5b165b8b867c, r8 0x16 != 0xc, r14 0x11 != 0x13, rip 0x4697800ab749 != 0x4697800ab746
[ReplaySession]     (SIGTRAP; stepi'd target $ip)
[ReplaySession]     breaking on target $ip
[Task] resuming execution of 158485 with PTRACE_SYSEMU tick_period -1 wait 2
[PerfCounters] Resetting counters with period 1152921504606846976
[Task] going into blocking wait for 158485 ...
[Task]   Task 158485 changed status to 0x57f (STOP-SIGTRAP)
[Task]   (refreshing register cache)
[Task] Requesting registers from tracee 158485
[ReplaySession]   not same execution point: 13 ticks left (@0x4697800ab746) replay vs rec: rcx 0xb5f7c7e302ef8006 != 0xe81c5b165b8b867c, r8 0x1 != 0xc, r14 0x12 != 0x13
[ReplaySession]   not same execution point: 13 ticks left (@0x4697800ab746) replay vs rec: rcx 0xb5f7c7e302ef8006 != 0xe81c5b165b8b867c, r8 0x1 != 0xc, r14 0x12 != 0x13
[ReplaySession]     fast-forwarded through 1 breakpoint hits with bpf
[ReplaySession]     trap was for target $ip
[ReplaySession]   not same execution point: 13 ticks left (@0x4697800ab746) replay vs rec: rcx 0xb5f7c7e302ef8006 != 0xe81c5b165b8b867c, r8 0x1 != 0xc, r14 0x12 != 0x13
[ReplaySession]     (fast-forwarding over target $ip)
[Task] resuming execution of 158485 with PTRACE_SINGLESTEP tick_period -1 wait 2
[PerfCounters] Resetting counters with period 1152921504606846976
[Task] going into blocking wait for 158485 ...
[Task]   Task 158485 changed status to 0x57f (STOP-SIGTRAP)
[Task]   (refreshing register cache)
[Task] Requesting registers from tracee 158485
[ReplaySession]   not same execution point: 13 ticks left (@0x4697800ab746) replay vs rec: rcx 0xb5f7c7e302ef8000 != 0xe81c5b165b8b867c, r8 0x1 != 0xc, r14 0x12 != 0x13, rip 0x4697800ab749 != 0x4697800ab746
[ReplaySession]   not same execution point: 13 ticks left (@0x4697800ab746) replay vs rec: rcx 0xb5f7c7e302ef8000 != 0xe81c5b165b8b867c, r8 0x1 != 0xc, r14 0x12 != 0x13, rip 0x4697800ab749 != 0x4697800ab746
[ReplaySession]     (SIGTRAP; stepi'd target $ip)
[ReplaySession]     breaking on target $ip
[Task] resuming execution of 158485 with PTRACE_SYSEMU tick_period -1 wait 2
[PerfCounters] Resetting counters with period 1152921504606846976
[Task] going into blocking wait for 158485 ...
[Task]   Task 158485 changed status to 0x57f (STOP-SIGTRAP)
[Task]   (refreshing register cache)
[Task] Requesting registers from tracee 158485
[Task]   (refreshing extra-register cache using XSAVE)
[ReplaySession]   same execution point
[ReplaySession]   same execution point
[ReplaySession]     fast-forwarded through 0 breakpoint hits with bpf
[ReplaySession]     trap was for target $ip
[ReplaySession]   same execution point