bpftrace recipes: 5 real problems solved
Clarendon Auditorium | Wed 15 Mar 3:45 p.m.–4:30 p.m.
Presented by
-
Trent Lloyd
@lathiat
https://lathiat.net
Trent Lloyd first presented at the age of 15 to an audience at linux.conf.au 2003. A long time passionate speaker and member of the Linux & Open Source community he worked in the Global MySQL Support Team from 2007-2016 before switching to his present role as a Sustaining Software Engineer in the Ubuntu Support Team at Canonical specialisting in OpenStack, Ceph and Networking for the last 7 years.
Trent Lloyd
@lathiat
https://lathiat.net
Abstract
As an SRE, Systems Administrator or other kind of Support Engineer we often do not have the luxury of modifying the software or system under test, additionally, many difficult to diagnose problems manifest as outlier cases requiring us to statistically measure and correlate requests.
Historically this can be challenging as sampling those outliers requires specific debug/analysis code to be added and systems to be restarted. Instead, dynamic runtime tracing combined with BPF allows us to load very small and fast programs into the kernel that run in the hotpath and summarise or analyse exactly the events we need, transmitting only a very small amount of data out of the kernel to be analysed in userspace. This instrumentation is installed at runtime with no changes to the system or a substantial impact on system latency or performance.
While such programs can be written as more complex and clunky C+Python scripts, 'bpftrace' allows us to write these in a nice Domain Specific Language (DSL) that combines the in-kernel data collection and userspace analysis components into a single coherent script that is understandable even by those who may not be software engineers or kernel experts.
Let's walk through how we can identify and solve 5 real problems using bpftrace to kickstart you on the way to using this technology to solve your next problem.
YouTube: https://www.youtube.com/watch?v=ZDTfcrp9pJI
LA Archive: http://mirror.linux.org.au/pub/everythingopen/2023/clarendon_auditorium/Wednesday/bpftrace_recipes_5_real_problems_solved.webm
As an SRE, Systems Administrator or other kind of Support Engineer we often do not have the luxury of modifying the software or system under test, additionally, many difficult to diagnose problems manifest as outlier cases requiring us to statistically measure and correlate requests. Historically this can be challenging as sampling those outliers requires specific debug/analysis code to be added and systems to be restarted. Instead, dynamic runtime tracing combined with BPF allows us to load very small and fast programs into the kernel that run in the hotpath and summarise or analyse exactly the events we need, transmitting only a very small amount of data out of the kernel to be analysed in userspace. This instrumentation is installed at runtime with no changes to the system or a substantial impact on system latency or performance. While such programs can be written as more complex and clunky C+Python scripts, 'bpftrace' allows us to write these in a nice Domain Specific Language (DSL) that combines the in-kernel data collection and userspace analysis components into a single coherent script that is understandable even by those who may not be software engineers or kernel experts. Let's walk through how we can identify and solve 5 real problems using bpftrace to kickstart you on the way to using this technology to solve your next problem. YouTube: https://www.youtube.com/watch?v=ZDTfcrp9pJI LA Archive: http://mirror.linux.org.au/pub/everythingopen/2023/clarendon_auditorium/Wednesday/bpftrace_recipes_5_real_problems_solved.webm