Update: I had to enable the cmake-build feature flag for rdkafka because it does not appear to build on windows without it.

Sadly this introduces a dependency on cmake for the build environment.

Does rdkafka even build in musl? The original rationale for using kafka was to be able to statically compile the whole thing.

Yes, but not with the cmake-build flag. (at least without some modifications to the build image).

It appears to look for musl-g++ which at least the clux/muslrust image lacks

Got it working even with the cmake-build flag using the following minimum effort dockerfile:

Update: demonstrated the result ist statically linked by running it in from scratch image

FROM clux/muslrust as build

RUN apt-get update && apt-get install -y cmake protobuf-compiler && ln -s /usr/bin/g++ /usr/bin/musl-g++

WORKDIR /build
COPY ./erased-serde-json ./erased-serde-json
COPY ./src ./src
COPY ./stream-delimit ./stream-delimit
COPY ./tests ./tests
COPY ./utils ./utils

COPY ./Cargo.toml ./Cargo.lock ./

RUN cargo test
RUN cargo build --release

FROM scratch as test
COPY --from=build /build/target/x86_64-unknown-linux-musl/release/pq /
RUN ["/pq", "--help"]

OK - so I'm pretty surprised that the kafka crate is just broken - not that I don't believe you, but do you have more details on that? Is it a known issue? https://github.com/spicavigo/kafka-rust

I have not investigated too deeply as there was no error message, and switching the underlying kafka library seemed to gave me results instantly.
There does not seem to be such a related issue on the rust-kafka project.

If you'd like I can look into it some more over the weekend (and mabye add an integation test that tries to talk to a kafka server)

Well, that's up to you - I don't want to volunteer you for extra work (and I'm not affiliated with the other project).

Just that a mysterious error isn't my favorite reason for making a significant change to pq's dependencies.

Figured it out.
Our kafka server gets message sets that are greater than 32KB, the default value for the rust kafka client.
by increasing this value, the issue is also fixed

Thanks for that - however this branch represents a lot of work, it would be a shame to lose it. Do you think there's value in allowing for both kafka backends with some cargo switches?

I merged the other PR - will cut a release soon.

Thanks for that - however this branch represents a lot of work, it would be a shame to lose it. Do you think there's value in allowing for both kafka backends with some cargo switches?

I think there is value in that (especially given that the rdkafka based client supports features the pure rust client does not, like zstd compression)

That's up to you though. I would also be fine with just closing the pr, given thet my main motivating issue at the moment is fixed now.

Alright then - if you ever find yourself wishing for librdkafka again, we can revisit. Thanks.