Testing Address Sanitizers

This comment acts as a "what-we-already-tried" benchmark to work from, which can guide our next steps in resolving this issue. During the investigation of #974 I had a look into the following 3 ASAN types:

ASan

(Android Docs)
This first Address Sanitizer gave me actual results when trying to build an Android app with native C memory bug detection. For future reference, this might make the most sense to add as a first "basic crashing test".

To run ASan on an emulated Pixel 9 Pro (running Android 14.0 Default Android System Image, so we have Root access) I added the following to the sample app (found here):
In /sample/cmakelists.txt

target_compile_options(ndk-sample PRIVATE -fsanitize=address -fno-omit-frame-pointer)
target_link_options(ndk-sample PRIVATE -fsanitize=address)

set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fsanitize=address -fno-omit-frame-pointer")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fsanitize=address -fno-omit-frame-pointer")
set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -fsanitize=address")

And these to /sample/build.gradle.kts and sentry-native/ndk/lib/build.gradle.kts

externalNativeBuild {
    cmake {
        arguments.add(0, "-DANDROID_STL=c++_shared")
        arguments.add(0, "-DSENTRY_NATIVE_SRC=$sentryNativeSrc")
        arguments.add("-DCMAKE_VERBOSE_MAKEFILE=ON")
        abiFilters.addAll(listOf("x86_64", "arm64-v8a"))
        abiFilters.forEach { abi ->
            println("ABI=$abi")
            if (abi == "arm64-v8a" || abi == "x86_64") {
                arguments.add(0, "-DANDROID_SANITIZE=address")
            }
        }
    }
}

After which I needed to copy [your ndk folder]\toolchains\llvm\prebuilt\windows-x86_64\lib64\clang\9.0.9\lib\linux\libclang_rt.asan-aarch64-android.so to [your app]/src/main/jniLibs/arm64-v8a/ (as instructed here)

When running the following snippet, we get ASan output:

char *ptr_f = (char *)malloc(10);
free(ptr_f);
*ptr_f = 'A';

ASan output

2024-10-22 13:56:47.897  3284-3284  libc                    io.sentry.ndk.sample                 A  Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 3284 (ntry.ndk.sample), pid 3284 (ntry.ndk.sample)
2024-10-22 13:56:48.020  3315-3315  DEBUG                   crash_dump64                         A  Cmdline: io.sentry.ndk.sample
2024-10-22 13:56:48.020  3315-3315  DEBUG                   crash_dump64                         A  pid: 3284, tid: 3284, name: ntry.ndk.sample  >>> io.sentry.ndk.sample <<<
2024-10-22 13:56:48.020  3315-3315  DEBUG                   crash_dump64                         A  Abort message: '=================================================================
                                                                                                    ==3284==ERROR: AddressSanitizer: heap-use-after-free on address 0x003a8ad50010 at pc 0x00715b1c8524 bp 0x007fc1c688d0 sp 0x007fc1c688c8
                                                                                                    WRITE of size 1 at 0x003a8ad50010 thread T0
                                                                                                        #0 0x715b1c8520  (/data/app/~~aKZshbc8F5M5JpsqBdf-bA==/io.sentry.ndk.sample-tq5NmovDEYAkK0IQ4HblkQ==/lib/arm64/libndk-sample.so+0x3520) (BuildId: 8e7e1f15d92eb5af50726b904e27f3ba40ca8f38)
                                                                                                        #1 0x71d6546c30  (/apex/com.android.art/lib64/libart.so+0x346c30) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #2 0x71d6530480  (/apex/com.android.art/lib64/libart.so+0x330480) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #3 0x71d671a244  (/apex/com.android.art/lib64/libart.so+0x51a244) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #4 0x71d66a3adc  (/apex/com.android.art/lib64/libart.so+0x4a3adc) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #5 0x71d65493d8  (/apex/com.android.art/lib64/libart.so+0x3493d8) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #6 0x71d6565700  (/apex/com.android.art/lib64/libart.so+0x365700) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #7 0x71d671ad8c  (/apex/com.android.art/lib64/libart.so+0x51ad8c) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #8 0x71d66a3adc  (/apex/com.android.art/lib64/libart.so+0x4a3adc) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #9 0x71d65493d8  (/apex/com.android.art/lib64/libart.so+0x3493d8) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #10 0x71d6565700  (/apex/com.android.art/lib64/libart.so+0x365700) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #11 0x71d671ad8c  (/apex/com.android.art/lib64/libart.so+0x51ad8c) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #12 0x71d66a4618  (/apex/com.android.art/lib64/libart.so+0x4a4618) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #13 0x71d65493d8  (/apex/com.android.art/lib64/libart.so+0x3493d8) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #14 0x71d6565700  (/apex/com.android.art/lib64/libart.so+0x365700) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #15 0x71d6564ff8  (/apex/com.android.art/lib64/libart.so+0x364ff8) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #16 0x71d6546d68  (/apex/com.android.art/lib64/libart.so+0x346d68) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #17 0x71d67d0e30  (/apex/com.android.art/lib64/libart.so+0x5d0e30) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #18 0x71d67d0dd4  (/apex/com.android.art/lib64/libart.so+0x5d0dd4) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #19 0x71d6530480  (/apex/com.android.art/lib64/libart.so+0x330480) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #20 0x71d671a244  (/apex/com.android.art/lib64/libart.so+0x51a244) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #21 0x71d66a3adc  (/apex/com.android.art/lib64/libart.so+0x4a3adc) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #22 0x71d65493d8  (/apex/com.android.art/lib64/libart.so+0x3493d8) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #23 0x71d6565700  (/apex/com.android.art/lib64/libart.so+0x365700) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #24 0x71d671ad8c  (/apex/com.android.art/lib64/libart.so+0x51ad8c) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #25 0x71d66a4618  (/apex/com.android.art/lib64/libart.so+0x4a4618) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #26 0x71d65493d8  (/apex/com.android.art/lib64/libart.so+0x3493d8) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #27 0x71d6565700  (/apex/com.android.art/lib64/libart.so+0x365700) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #28 0x71d671ad8c  (/apex/com.android.art/lib64/libart.so+0x51ad8c) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #29 0x71d66a3adc  (/apex/com.android.art/lib64/libart.so+0x4a3adc) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #30 0x71d65493d8  (/apex/com.android.art/lib64/libart.so+0x3493d8) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #31 0x71d6565700  (/apex/com.android.art/lib64/libart.so+0x365700) (BuildId: d22b3b69a6db691fdd84720465c7a214)
2024-10-22 13:56:48.020  3315-3315  DEBUG                   crash_dump64                         A      #32 0x71d671ad8c  (/apex/com.android.art/lib64/libart.so+0x51ad8c) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #33 0x71d66a376c  (/apex/com.android.art/lib64/libart.so+0x4a376c) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #34 0x71d65493d8  (/apex/com.android.art/lib64/libart.so+0x3493d8) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #35 0x71d6565700  (/apex/com.android.art/lib64/libart.so+0x365700) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #36 0x71d671ad8c  (/apex/com.android.art/lib64/libart.so+0x51ad8c) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #37 0x71d66a3adc  (/apex/com.android.art/lib64/libart.so+0x4a3adc) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #38 0x71d65493d8  (/apex/com.android.art/lib64/libart.so+0x3493d8) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #39 0x71d6565700  (/apex/com.android.art/lib64/libart.so+0x365700) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #40 0x71d671ad8c  (/apex/com.android.art/lib64/libart.so+0x51ad8c) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #41 0x71d66a3adc  (/apex/com.android.art/lib64/libart.so+0x4a3adc) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #42 0x71d65493d8  (/apex/com.android.art/lib64/libart.so+0x3493d8) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #43 0x71d6565700  (/apex/com.android.art/lib64/libart.so+0x365700) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #44 0x71d6564ff8  (/apex/com.android.art/lib64/libart.so+0x364ff8) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #45 0x71d6546d68  (/apex/com.android.art/lib64/libart.so+0x346d68) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #46 0x71d6530480  (/apex/com.android.art/lib64/libart.so+0x330480) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #47 0x71d658ae2c  (/apex/com.android.art/lib64/libart.so+0x38ae2c) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #48 0x71d658a7e8  (/apex/com.android.art/lib64/libart.so+0x38a7e8) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #49 0x71d6546c30  (/apex/com.android.art/lib64/libart.so+0x346c30) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #50 0x71d65301a4  (/apex/com.android.art/lib64/libart.so+0x3301a4) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #51 0x71d671a0c0  (/apex/com.android.art/lib64/libart.so+0x51a0c0) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #52 0x71d66a376c  (/apex/com.android.art/lib64/libart.so+0x4a376c) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #53 0x71d65493d8  (/apex/com.android.art/lib64/libart.so+0x3493d8) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #54 0x71d6565700  (/apex/com.android.art/lib64/libart.so+0x365700) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #55 0x71d6564ff8  (/apex/com.android.art/lib64/libart.so+0x364ff8) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #56 0x71d6546d68  (/apex/com.android.art/lib64/libart.so+0x346d68) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                    
                                                                                                    0x003a8ad50010 is located 0 bytes inside of 10-byte region [0x003a8ad50010,0x003a8ad5001a)
                                                                                                    freed by thread T0 here:
                                                                                                        #0 0x714f16d54c  (/data/app/~~aKZshbc8F5M5JpsqBdf-bA==/io.sentry.ndk.sample-tq5NmovDEYAkK0IQ4HblkQ==/lib/arm64/libclang_rt.asan-aarch64-android.so+0xe454c) (BuildId: 2c287b347827d5aec5b5ef36619b30b1df0e20ac)
                                                                                                        #1 0x715b1c84cc  (/data/app/~~aKZshbc8F5M5JpsqBdf-bA==/io.sentry.ndk.sample-tq5NmovDEYAkK0IQ4HblkQ==/lib/arm64/libndk-sample.so+0x34cc) (BuildId: 8e7e1f15d92eb5af50726b904e27f3ba40ca8f38)
                                                                                                        #2 0x71d6546c30  (/apex/com.android.art/lib64/libart.so+0x346c30) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #3 0x74a07bfe78  (<unknown module>)
                                                                                                        #4 0x7fc1c68f1c  ([stack]+0x7f7f1c)
                                                                                                    
                                                                                                    previously allocated by thread T0 here:
                                                                                                        #0 0x714f16d668  (/data/app/~~aKZshbc8F5M5JpsqBdf-bA==/io.sentry.ndk.sample-tq5NmovDEYAkK0IQ4HblkQ==/lib/arm64/libclang_rt.asan-aarch64-android.so+0xe4668) (BuildId: 2c287b347827d5aec5b5ef36619b30b1df0e20ac)
2024-10-22 13:56:48.020  3315-3315  DEBUG                   crash_dump64                         A      #1 0x715b1c84c0  (/data/app/~~aKZshbc8F5M5JpsqBdf-bA==/io.sentry.ndk.sample-tq5NmovDEYAkK0IQ4HblkQ==/lib/arm64/libndk-sample.so+0x34c0) (BuildId: 8e7e1f15d92eb5af50726b904e27f3ba40ca8f38)
                                                                                                        #2 0x71d6546c30  (/apex/com.android.art/lib64/libart.so+0x346c30) (BuildId: d22b3b69a6db691fdd84720465c7a214)
                                                                                                        #3 0x74a07bfe78  (<unknown module>)
                                                                                                        #4 0x7fc1c68f1c  ([stack]+0x7f7f1c)
                                                                                                    
                                                                                                    SUMMARY: AddressSanitizer: heap-use-after-free (/data/app/~~aKZshbc8F5M5JpsqBdf-bA==/io.sentry.ndk.sample-tq5NmovDEYAkK0IQ4HblkQ==/lib/arm64/libndk-sample.so+0x3520) (BuildId: 8e7e1f15d92eb5af50726b904e27f3ba40ca8f38) 
                                                                                                    Shadow bytes around the buggy address:
                                                                                                      0x003a8ad4fd80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
                                                                                                      0x003a8ad4fe00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
                                                                                                      0x003a8ad4fe80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
                                                                                                      0x003a8ad4ff00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
                                                                                                      0x003a8ad4ff80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
                                                                                                    =>0x003a8ad50000: fa fa[fd]fd fa fa fa fa fa fa fa fa fa fa fa fa
                                                                                                      0x003a8ad50080: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
                                                                                                      0x003a8ad50100: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
                                                                                                      0x003a8ad50180: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
                                                                                                      0x003a8ad50200: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
                                                                                                      0x003a8ad50280: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
                                                                                                    Shadow byte legend (one shadow byte represents 8 application bytes):
                                                                                                      Addressable:           00
                                                                                                      Partially addressable: 01 02 03 04 05 06 07 
                                                                                                      Heap left redzone:       fa
                                                                                                      Freed heap region:       fd
                                                                                                      Stack left redzone:      f1
                                                                                                      Stack mid redzone:       f2
                                                                                                      Stack right redzone:     f3
                                                                                                      Stack after return:      f5
                                                                                                      Stack use after scope:   f8
                                                                                                      Global redzone:          f9
                                                                                                      Global init order:       f6
                                                                                                      Poisoned by user:        f7
                                                                                                      Container overflow:      fc
                                                                                                      Array cookie:            ac
                                                                                                      Intra object redzone:    bb
                                                                                                      ASan internal:           fe
                                                                                                      Left alloca redzone:     ca
                                                                                                      Right alloca redzone:    cb'
2024-10-22 13:56:48.020  3315-3315  DEBUG                   crash_dump64                         A        #01 pc 000000000006c5b4  /data/app/~~aKZshbc8F5M5JpsqBdf-bA==/io.sentry.ndk.sample-tq5NmovDEYAkK0IQ4HblkQ==/lib/arm64/libclang_rt.asan-aarch64-android.so (BuildId: 2c287b347827d5aec5b5ef36619b30b1df0e20ac)
2024-10-22 13:56:48.020  3315-3315  DEBUG                   crash_dump64                         A        #02 pc 000000000006b034  /data/app/~~aKZshbc8F5M5JpsqBdf-bA==/io.sentry.ndk.sample-tq5NmovDEYAkK0IQ4HblkQ==/lib/arm64/libclang_rt.asan-aarch64-android.so (BuildId: 2c287b347827d5aec5b5ef36619b30b1df0e20ac)
2024-10-22 13:56:48.020  3315-3315  DEBUG                   crash_dump64                         A        #03 pc 00000000000e88fc  /data/app/~~aKZshbc8F5M5JpsqBdf-bA==/io.sentry.ndk.sample-tq5NmovDEYAkK0IQ4HblkQ==/lib/arm64/libclang_rt.asan-aarch64-android.so (BuildId: 2c287b347827d5aec5b5ef36619b30b1df0e20ac)
2024-10-22 13:56:48.020  3315-3315  DEBUG                   crash_dump64                         A        #04 pc 00000000000eb7fc  /data/app/~~aKZshbc8F5M5JpsqBdf-bA==/io.sentry.ndk.sample-tq5NmovDEYAkK0IQ4HblkQ==/lib/arm64/libclang_rt.asan-aarch64-android.so (BuildId: 2c287b347827d5aec5b5ef36619b30b1df0e20ac)
2024-10-22 13:56:48.020  3315-3315  DEBUG                   crash_dump64                         A        #05 pc 00000000000ec81c  /data/app/~~aKZshbc8F5M5JpsqBdf-bA==/io.sentry.ndk.sample-tq5NmovDEYAkK0IQ4HblkQ==/lib/arm64/libclang_rt.asan-aarch64-android.so (__asan_report_store1+52) (BuildId: 2c287b347827d5aec5b5ef36619b30b1df0e20ac)
2024-10-22 13:56:48.020  3315-3315  DEBUG                   crash_dump64                         A        #06 pc 0000000000003520  /data/app/~~aKZshbc8F5M5JpsqBdf-bA==/io.sentry.ndk.sample-tq5NmovDEYAkK0IQ4HblkQ==/lib/arm64/libndk-sample.so (Java_io_sentry_ndk_sample_NdkSample_spawnBreadcrumbProcess+468) (BuildId: 8e7e1f15d92eb5af50726b904e27f3ba40ca8f38)
2024-10-22 13:56:48.020  3315-3315  DEBUG                   crash_dump64                         A        #12 pc 00000000000008a8  <anonymous:74a07bf000> (io.sentry.ndk.sample.MainActivity.lambda$onCreate$3+0)
2024-10-22 13:56:48.020  3315-3315  DEBUG                   crash_dump64                         A        #17 pc 00000000000006d8  <anonymous:74a07bf000> (io.sentry.ndk.sample.MainActivity$$ExternalSyntheticLambda3.onClick+0)

GWP-ASan

(Android Docs)
This method relies on intercepting randomly chosen subsets of heap allocations to catch memory corruption bugs. This method isn't reliable enough for testing, since you'd need a large enough number of runs to find statistically significant results. It is meant more as a lightweight in-production ASan that, given enough users, will find heap memory safety bugs.

Enabling this for an app can be done by adding this to the AndroidManifest.xml file:

<application android:gwpAsanMode="always">
  ...
</application>

HW-ASan

(Android Docs)
Hardware-Assisted ASan can be useful, but needs device support. In my testing (only using emulated devices) this did not work (probably because it needs the actual hardware). In theory it catches a superset of ASan (the same bugs + stack use after return, plus it has lower RAM overhead)

Uh oh!

[#inc-517] Add basic crashing test with asan (HW Asan, GWP-ASan) #962

Description

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions