Sorry for not having a MCVE for this issue. I'm still constructing it. For now I have to refer to a large codebase to reproduce the issue.

Basically, after we upgrade from nightly-2022-07-29 to nightly-2022-10-16, we found two things:

• Capturing stack backtrace is much slower than before. A single capture can take as long as 300 ms. Previously the Backtrace::capture() at the same code location only takes 1ms. Most of the time are spent on libunwind's CFI_Parser. performance regression (for debug & single binary) of backtrace capturing after bumping toolchain on macOS risingwavelabs/risingwave#6131
• Sometimes capturing a stack backtrace will segfault. bug(debug profile): segfault/EXC_BAD_ACCESS during backtrace capture risingwavelabs/risingwave#6205 (comment)

This issue is stably reproducible on some specific commits of our project with some specific way of compiling, so I guess it's probably not related to incremental compile. I guess it would be more likely to be a problem with the LLVM 15 upgrade in August or the std::backtrace::Backtrace stabilization.

Reproduce 1

On this commit: risingwavelabs/risingwave@227e9e5

RUST_BACKTRACE=1 cargo run --bin risingwave -- playground

In another terminal, use psql (Postgres's client) to connect to the program:

psql -h localhost -p 4566 -d dev -U root
CREATE TABLE t(a int, b int);
CREATE VIEW v AS SELECT * FROM t;
DROP TABLE t;

The program will immediately segfault in Backtrace::capture.

Interestingly, if we use cargo build -p risingwave_cmd_all && ./target/debug/risingwave playground, it works. The commit following the buggy commit risingwavelabs/risingwave@604a0a5 also magically resolves the issue with some random code change.

Reproduce 2

On this commit: risingwavelabs/risingwave@484b9ab

cargo build -p risingwave_cmd
RUST_BACKTRACE=1 ./target/debug/meta-node # in terminal 1
RUST_BACKTRACE=1 ./target/debug/compute-node # in terminal 2
RUST_BACKTRACE=1 ./target/debug/frontend # in terminal 3

In another terminal:

psql -h localhost -p 4566 -d dev -U root
CREATE TABLE BOOLTBL2 (f1 bool); INSERT INTO BOOLTBL2 (f1) VALUES (bool 'XXX');

compute-node will also immediately segfault when capturing a backtrace.

Thanks for investigating into this!

Meta

rustc --version --verbose:

rustc 1.66.0-nightly (b8c35ca26 2022-10-15)
binary: rustc
commit-hash: b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1
commit-date: 2022-10-15
host: aarch64-apple-darwin
release: 1.66.0-nightly
LLVM version: 15.0.2

<backtrace>