feat(layers/dtrace): Support User Statically-Defined Tracing(aka USDT) on Linux by Zheaoli · Pull Request #4053 · apache/opendal

Zheaoli · 2024-01-22T18:14:51Z

This feature is an experimental feature

TL;DR;

The PR will make the debug and tracing the opendal process easier in the Linux platform. The people can use so many modern tools like bpftrace to trace the internal state in the process

USDT is a way to allow people to pre-define some static hooks. Those hooks could be used to fetch some internal information when the process runs.

I will use this PR to describe the detailed behavior of the DTrace in the OpenDAL.

When people enabled layers-dtrace feature and added the layer to their code like the following code:

use anyhow::Result;
use opendal::services::Fs;
use opendal::Operator;
use opendal::layers::DTraceLayer;

#[tokio::main]
async fn main() -> Result<()> {
    let mut builder = Fs::default();

    builder.root("/tmp");

    // `Accessor` provides the low level APIs, we will use `Operator` normally.
    let op: Operator = Operator::new(builder)?.layer(DTraceLayer{}).finish();
    
    let path="/tmp/test.txt";
    for _ in 1..100000{
        let bs = vec![0; 64 * 1024 * 1024];
        op.write(path, bs).await?;
        op.read(path).await?;
    }
    Ok(())
}

Then we compile the code, and use readelf -n ${target_binary} to get the binary detail, we will find some information like following below

Displaying notes found in: .note.stapsdt
  Owner                Data size        Description
  stapsdt              0x00000039       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: create_dir_start
    Location: 0x00000000000f8f05, Base: 0x0000000000000000, Semaphore: 0x00000000003649f8
    Arguments: -8@%rax
  stapsdt              0x00000037       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: create_dir_end
    Location: 0x00000000000f9284, Base: 0x0000000000000000, Semaphore: 0x00000000003649fa
    Arguments: -8@%rax
  stapsdt              0x0000003c       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: blocking_list_start
    Location: 0x00000000000f9487, Base: 0x0000000000000000, Semaphore: 0x0000000000364a28
    Arguments: -8@%rax
  stapsdt              0x0000003a       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: blocking_list_end
    Location: 0x00000000000f9546, Base: 0x0000000000000000, Semaphore: 0x0000000000364a2a
    Arguments: -8@%rax
  stapsdt              0x0000003c       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: blocking_read_start
    Location: 0x00000000000f9740, Base: 0x0000000000000000, Semaphore: 0x0000000000364a18
    Arguments: -8@%rax
  stapsdt              0x0000003a       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: blocking_read_end
    Location: 0x00000000000f97fd, Base: 0x0000000000000000, Semaphore: 0x0000000000364a1a
    Arguments: -8@%rax
  stapsdt              0x0000003c       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: blocking_stat_start
    Location: 0x00000000000f99f3, Base: 0x0000000000000000, Semaphore: 0x0000000000364a20
    Arguments: -8@%rax
  stapsdt              0x0000003a       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: blocking_stat_end
    Location: 0x00000000000f9ab9, Base: 0x0000000000000000, Semaphore: 0x0000000000364a22
    Arguments: -8@%rax
  stapsdt              0x0000003d       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: blocking_write_start
    Location: 0x00000000000f9cec, Base: 0x0000000000000000, Semaphore: 0x0000000000364a1c
    Arguments: -8@%rax
  stapsdt              0x0000003b       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: blocking_write_end
    Location: 0x00000000000f9df2, Base: 0x0000000000000000, Semaphore: 0x0000000000364a1e
    Arguments: -8@%rax
  stapsdt              0x0000003e       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: blocking_delete_start
    Location: 0x00000000000f9fdc, Base: 0x0000000000000000, Semaphore: 0x0000000000364a24
    Arguments: -8@%rax
  stapsdt              0x0000003c       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: blocking_delete_end
    Location: 0x00000000000fa0e6, Base: 0x0000000000000000, Semaphore: 0x0000000000364a26
    Arguments: -8@%rax
  stapsdt              0x00000042       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: blocking_create_dir_start
    Location: 0x00000000000fa250, Base: 0x0000000000000000, Semaphore: 0x0000000000364a14
    Arguments: -8@%rax
  stapsdt              0x00000040       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: blocking_create_dir_end
    Location: 0x00000000000fa30c, Base: 0x0000000000000000, Semaphore: 0x0000000000364a16
    Arguments: -8@%rax
  stapsdt              0x00000033       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: list_start
    Location: 0x00000000000fa869, Base: 0x0000000000000000, Semaphore: 0x0000000000364a0c
    Arguments: -8@%rax
  stapsdt              0x00000031       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: list_end
    Location: 0x00000000000faac7, Base: 0x0000000000000000, Semaphore: 0x0000000000364a0e
    Arguments: -8@%rax
  stapsdt              0x00000033       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: read_start
    Location: 0x00000000000fb0a4, Base: 0x0000000000000000, Semaphore: 0x00000000003649fc
    Arguments: -8@%rax
  stapsdt              0x00000031       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: read_end
    Location: 0x00000000000fb321, Base: 0x0000000000000000, Semaphore: 0x00000000003649fe
    Arguments: -8@%rax
  stapsdt              0x00000033       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: stat_start
    Location: 0x00000000000fb8eb, Base: 0x0000000000000000, Semaphore: 0x0000000000364a04
    Arguments: -8@%rax
  stapsdt              0x00000031       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: stat_end
    Location: 0x00000000000fbb66, Base: 0x0000000000000000, Semaphore: 0x0000000000364a06
    Arguments: -8@%rax
  stapsdt              0x00000034       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: write_start
    Location: 0x00000000000fc6e5, Base: 0x0000000000000000, Semaphore: 0x0000000000364a00
    Arguments: -8@%rax
  stapsdt              0x00000032       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: write_end
    Location: 0x00000000000fc954, Base: 0x0000000000000000, Semaphore: 0x0000000000364a02
    Arguments: -8@%rax
  stapsdt              0x00000035       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: delete_start
    Location: 0x00000000000fcf30, Base: 0x0000000000000000, Semaphore: 0x0000000000364a08
    Arguments: -8@%rax
  stapsdt              0x00000033       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: delete_end
    Location: 0x00000000000fd2aa, Base: 0x0000000000000000, Semaphore: 0x0000000000364a0a
    Arguments: -8@%rax
  stapsdt              0x00000036       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: presign_start
    Location: 0x00000000000fd87b, Base: 0x0000000000000000, Semaphore: 0x0000000000364a10
    Arguments: -8@%rax
  stapsdt              0x00000034       NT_STAPSDT (SystemTap probe descriptors)
    Provider: opendal
    Name: presign_end
    Location: 0x00000000000fdaf6, Base: 0x0000000000000000, Semaphore: 0x0000000000364a12
    Arguments: -8@%rax

We describe the offset, the argument of hook, and so many information else in each section of the text. For example, the write_start hook, we can find the offset is 0x00000000000fc6e5 and we can use gdb to verify it.

We will figure out the asm in 0x5555556506e5 is NOP (the process virtual address started at 0x555555554000 on my machine, and 0x555555554000+0x00000000000fc6e5=0x5555556506e5). The NOP instruction code will be replaced by the interrupt code when the hook is attached (In x86_64, the interrupt code is INT3)

We can also set a breakpoint at this address.

We will find the code paused on line 146 in dtrace.rs. This is exactly what we want

The USDT can help us debug it more easily and make it easier to trace it.

╭─   manjusaka@manjusaka-garuda   ~/Documents/projects/rust-demo                                                               02:14:06  
╰─ sudo bpftrace -e ' 
usdt:/home/manjusaka/Documents/projects/rust-demo/target/debug/demo:opendal:write_start { 
    @start[tid,str(arg0)]=nsecs;
}

usdt:/home/manjusaka/Documents/projects/rust-demo/target/debug/demo:opendal:write_end { 
    printf("Write %s use %d nsecs\n", str(arg0), nsecs - @start[tid,str(arg0)]);
    delete(@start[tid,str(arg0)]);
}' -p 385697      
Attaching 2 probes...
Write tmp/test.txt use 59609 nsecs
Write tmp/test.txt use 56546 nsecs
Write tmp/test.txt use 53352 nsecs
Write tmp/test.txt use 57003 nsecs
Write tmp/test.txt use 58479 nsecs
Write tmp/test.txt use 52876 nsecs
Write tmp/test.txt use 56553 nsecs
Write tmp/test.txt use 59414 nsecs
Write tmp/test.txt use 101736 nsecs
Write tmp/test.txt use 54204 nsecs
Write tmp/test.txt use 63272 nsecs
Write tmp/test.txt use 64367 nsecs
Write tmp/test.txt use 56620 nsecs
Write tmp/test.txt use 56536 nsecs
Write tmp/test.txt use 55383 nsecs
Write tmp/test.txt use 52547 nsecs
Write tmp/test.txt use 50337 nsecs
Write tmp/test.txt use 52908 nsecs
Write tmp/test.txt use 59654 nsecs
Write tmp/test.txt use 64653 nsecs
Write tmp/test.txt use 66440 nsecs

We can use many other tools to trace it.

core/src/layers/dtrace.rs

core/src/layers/mod.rs

core/src/layers/dtrace.rs

…) on Linux Signed-off-by: Manjusaka <me@manjusaka.me>

Signed-off-by: Manjusaka <me@manjusaka.me>

Xuanwo

All our probe names should follow the same rule. I raised some of them but not all. I feel like it's better to follow the same rules across the whole project.

core/src/layers/dtrace.rs

Xuanwo · 2024-01-24T13:00:30Z

core/src/layers/dtrace.rs

+                n
+            })
+            .map_err(|e| {
+                probe_lazy!(opendal, blocking_read_complete_error, c_path.as_ptr());


blocking_reader_read_error

core/src/layers/dtrace.rs

Signed-off-by: Manjusaka <me@manjusaka.me>

Xuanwo

Thanks a lot!

Zheaoli requested review from PsiACE and Xuanwo as code owners January 22, 2024 18:14

github-actions bot requested review from ClSlaid and oowl January 22, 2024 18:15

github-actions bot added the releases-note/feat The PR implements a new feature or has a title that begins with "feat" label Jan 22, 2024

Zheaoli force-pushed the manjusaka/support-dtrace-layers branch from c48eab7 to c2f0679 Compare January 22, 2024 18:15

Xuanwo reviewed Jan 23, 2024

View reviewed changes

oowl reviewed Jan 23, 2024

View reviewed changes

core/src/layers/dtrace.rs Show resolved Hide resolved

Zheaoli force-pushed the manjusaka/support-dtrace-layers branch from 6716317 to 653d889 Compare January 23, 2024 15:05

Zheaoli requested review from Xuanwo and oowl January 23, 2024 15:05

Xuanwo reviewed Jan 23, 2024

View reviewed changes

core/src/layers/dtrace.rs Outdated Show resolved Hide resolved

core/src/layers/dtrace.rs Outdated Show resolved Hide resolved

core/src/layers/dtrace.rs Outdated Show resolved Hide resolved

Zheaoli added 6 commits January 24, 2024 20:41

feat(layers/dtrace): Support User Statically-Defined Tracing(aka USDT…

13ee121

…) on Linux Signed-off-by: Manjusaka <me@manjusaka.me>

fix typo

2237e10

Signed-off-by: Manjusaka <me@manjusaka.me>

fix code style

60f9f62

Signed-off-by: Manjusaka <me@manjusaka.me>

fix review idea

3bfcefa

Signed-off-by: Manjusaka <me@manjusaka.me>

fix review idea

0ee4d24

Signed-off-by: Manjusaka <me@manjusaka.me>

update code

ffe0794

Signed-off-by: Manjusaka <me@manjusaka.me>

Zheaoli force-pushed the manjusaka/support-dtrace-layers branch from 0964c56 to ffe0794 Compare January 24, 2024 12:41

Zheaoli requested a review from Xuanwo January 24, 2024 12:42

format code

a5d5fdf

Signed-off-by: Manjusaka <me@manjusaka.me>

Xuanwo reviewed Jan 24, 2024

View reviewed changes

Zheaoli added 2 commits January 24, 2024 22:19

fix review idea

9609561

Signed-off-by: Manjusaka <me@manjusaka.me>

fix review idea

32b34b0

Signed-off-by: Manjusaka <me@manjusaka.me>

Zheaoli requested a review from Xuanwo January 24, 2024 14:31

Xuanwo approved these changes Jan 24, 2024

View reviewed changes

Xuanwo merged commit 1783dd7 into main Jan 24, 2024

Xuanwo deleted the manjusaka/support-dtrace-layers branch January 24, 2024 14:56

morristai mentioned this pull request Jan 29, 2024

chore: Bump to version 0.45.0 to start release process #4088

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(layers/dtrace): Support User Statically-Defined Tracing(aka USDT) on Linux#4053

feat(layers/dtrace): Support User Statically-Defined Tracing(aka USDT) on Linux#4053
Xuanwo merged 9 commits intomainfrom
manjusaka/support-dtrace-layers

Zheaoli commented Jan 22, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Xuanwo left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Xuanwo Jan 24, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Xuanwo left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Zheaoli commented Jan 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Xuanwo left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Xuanwo Jan 24, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Xuanwo left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Zheaoli commented Jan 22, 2024 •

edited

Loading

Xuanwo left a comment •

edited

Loading