Description

As mentioned in 562, kafka-python is no longer maintained and while there is a new fork, it is unclear how well that will be maintained either. This switches the Kafka stream shims to confluent-kafka.

Fixes

• Fixes Decide what to do about Kafka dependency #562

Type of Change

• Breaking Change (fix or enhancement which changes existing semantics of the public interface)
• Enhancement (new features or improvements to existing functionality)
• Bug (fixes for a bug or issue)
• Internal (refactoring, style changes, testing, optimizations)
• Documentation update (changes to documentation or examples)
• Package (dependencies, versions, package metadata)
• Development (CI workflows, pre-commit, linters, templates)
• Security (security related changes)

Testing

Updated unit tests and validated the shim with testing/scripts/kafka_pubsub.py.

Pull Request Checklist

• I have read the Contributing and PR submission guides.

Please confirm the PR meets the following requirements.

• Tags added to PR (e.g., breaking, bug, enhancement, internal, documentation, package, development, security).
• Code changes pass pre-commit (e.g., mypy, ruff, etc.).
• Tests have been added to show the fix is effective or that the new feature works.
• New and existing unit tests pass locally with the changes.
• Docs have been updated and reviewed if relevant.