Added support for NATS JetStream by dmitry-sles-novikov · Pull Request #84799 · ClickHouse/ClickHouse

dmitry-sles-novikov · 2025-07-31T14:00:08Z

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Users can now use NATS JetStream to consume messages by specifying the new settings of nats_stream and nats_consumer for the NATS engine.

Documentation entry for user-facing changes

Documentation is written (mandatory for new features)

#39459

clickhouse-gh · 2025-07-31T14:04:46Z

Workflow [PR], commit [bdae3a3]

Summary: ❌

job_name	test_name	status
Upgrade check (amd_tsan)		failure
	Killed by signal (in clickhouse-server.log)	FAIL
	Fatal message in clickhouse-server.log (see fatal_messages.txt)	FAIL
	Killed by signal (output files)	FAIL
	Found signal in gdb.log	FAIL

antaljanosbenjamin

First of all, sorry for being late.

Second, it is looking great! There a few things and I haven't checked the tests yet, will check them tomorrow probably.

src/Storages/NATS/NATSCoreConsumer.cpp

src/Storages/NATS/INATSConsumer.h

src/Storages/NATS/NATSCoreConsumer.h

src/Storages/NATS/NATSCoreProducer.h

src/Storages/NATS/StorageNATS.cpp

Co-authored-by: János Benjamin Antal <antaljanosbenjamin@users.noreply.github.com>

dmitry-sles-novikov · 2025-08-14T12:44:48Z

@antaljanosbenjamin, Sorry, I accidentally requested a review.

antaljanosbenjamin · 2025-08-26T10:41:50Z

I wasn't working until today, will have a look now.

tests/integration/compose/docker_compose_nats.yml

tests/integration/test_storage_nats/common.py

tests/integration/test_storage_nats/test_nats_jet_stream.py

Co-authored-by: János Benjamin Antal <antaljanosbenjamin@users.noreply.github.com>

dmitry-sles-novikov · 2025-08-28T06:24:56Z

@antaljanosbenjamin I have finished the fixes and am ready to continue the review

antaljanosbenjamin

I found one typo, after that I am fine to merge the PR. Sorry for asking to fix the typo, but it is in the name of a function and I just feel bad merging it as is. In a comment it would be perfectly fine, but for the function name I feel like I am not comfortable with it.

And I think you did a really great job here again, good PR, very patient with the reviews, so thanks for everything!

tests/integration/test_storage_nats/test_nats_core.py

Co-authored-by: János Benjamin Antal <antaljanosbenjamin@users.noreply.github.com>

dmitry-sles-novikov · 2025-08-29T10:29:22Z

I think so too, typos definitely need to be fixed.
All comments were on point, and I completely agree with them, code of the test has become much better.

Thanks to you too =)))

antaljanosbenjamin · 2025-09-08T13:11:30Z

Integration tests (amd_asan, old analyzer, 3/6) - test_restore_db_replica/test.py::test_restore_db_replica_on_cluster failed so far only in this PR
Integration tests (amd_tsan, 3/6) - test_distributed_ddl/test.py::test_on_server_fail[configs] also failed so far only in this PR
Stress test (amd_debug) - Stress test fails with Disk does not support WriteMode::Append #84669
Stress test (amd_msan) - Failed assert assertHasValidVersionMetadata() #83988
Upgrade check (amd_asan) - Logical error: Expected JOIN table expression to be table, table function, query or union node #81144
Upgrade check (amd_tsan) - is not connected I think, but I have to check it

Let me try to restart the the two integrations tests just to make sure they are not related in any way.

Edit: I couldn't restart only the two integration tests, I had to restart all failed jobs 🤷

antaljanosbenjamin · 2025-09-09T09:11:52Z

Upgrade check (amd_tsan) didn't fail recently apart from this PR. I have to check it out in details, what caused the issue and why a memory limit exceeded exception caused a crash in CH. @dmitry-sles-novikov if yyou have any idea, please feel free to share it.

dmitry-sles-novikov · 2025-09-09T17:58:19Z

Is Upgrade check (amd_tsan) the same problem as #81144?

Edit: In this PR test failed too

antaljanosbenjamin · 2025-09-10T07:58:24Z

You are right, I messed up something.

antaljanosbenjamin · 2025-09-10T08:05:28Z

Thanks again, great job!

dmitry-sles-novikov added 11 commits July 31, 2025 14:21

rename NATSConsumer to INATSConsumer

114c55f

added NATSCoreConsumer

4f1d735

added jet stream push consumer

8e837aa

renamed NATSJetStreamPushConsumer to NATSJetStreamAsyncConsumer

6026478

added async pull consumer instead of async push consumer

b685265

added NATS jet stream settings validation

9586d4f

ranamed NATSProducer to INATSProducer

75e7234

added NATSCoreProducer

a135d17

added NATS jets stream producer

d5ca667

remoded jet stream consumer mode

43c4b3e

fixed nats_no_connection tests

dd249b5

antaljanosbenjamin added the can be tested Allows running workflows for external contributors label Jul 31, 2025

antaljanosbenjamin self-assigned this Jul 31, 2025

clickhouse-gh bot added the pr-feature Pull request with new product feature label Jul 31, 2025

dmitry-sles-novikov added 3 commits July 31, 2025 17:30

fixed codestyle

fa5d588

fix spelling

a1ac7df

Merge branch 'master' into nats_jet_stream_support

e3dfe7d

dmitry-sles-novikov changed the title ~~Added support for NATS JetStream~~ [WIP] Added support for NATS JetStream Aug 4, 2025

fixed jet stream flaky tests

353ddd8

dmitry-sles-novikov changed the title ~~[WIP] Added support for NATS JetStream~~ Added support for NATS JetStream Aug 4, 2025

Merge branch 'master' into nats_jet_stream_support

d1ccbc3

antaljanosbenjamin reviewed Aug 6, 2025

View reviewed changes

dmitry-sles-novikov and others added 3 commits August 6, 2025 15:40

Apply suggestions from code review

41d8f11

Co-authored-by: János Benjamin Antal <antaljanosbenjamin@users.noreply.github.com>

added using of inheriting constructors

184f463

Merge branch 'master' into nats_jet_stream_support

b2d1f84

dmitry-sles-novikov requested a review from antaljanosbenjamin August 14, 2025 08:32

dmitry-sles-novikov added 2 commits August 15, 2025 12:13

fix test no_connection_at_startup_1

a6645d6

fix conflicts

a51dac7

antaljanosbenjamin reviewed Aug 26, 2025

View reviewed changes

dmitry-sles-novikov and others added 10 commits August 26, 2025 15:22

Update tests/integration/test_storage_nats/test_nats_jet_stream.py

1f4be93

Co-authored-by: János Benjamin Antal <antaljanosbenjamin@users.noreply.github.com>

Update tests/integration/test_storage_nats/test_nats_jet_stream.py

ef2e444

Co-authored-by: János Benjamin Antal <antaljanosbenjamin@users.noreply.github.com>

moved NATS credentials to config_cluster.py

52f527d

refactor of check_query_result fuction

70193b1

removed test test_nats_many_materialized_views

f0a67c7

refactored test test_nats_protobuf

800d326

refactored function wait_query_result

14e4d07

removed infinite loops from tests

9726bfa

added error text checking

8809ee0

Merge branch 'master' into nats_jet_stream_support

94069ee

antaljanosbenjamin reviewed Aug 29, 2025

View reviewed changes

tests/integration/test_storage_nats/test_nats_core.py Outdated Show resolved Hide resolved

dmitry-sles-novikov and others added 2 commits August 29, 2025 12:36

Update tests/integration/test_storage_nats/test_nats_core.py

2e92f8b

Co-authored-by: János Benjamin Antal <antaljanosbenjamin@users.noreply.github.com>

fixed typo

bdae3a3

dmitry-sles-novikov requested a review from antaljanosbenjamin September 4, 2025 07:31

antaljanosbenjamin added this pull request to the merge queue Sep 10, 2025

Merged via the queue into ClickHouse:master with commit 7760f14 Sep 10, 2025
232 of 240 checks passed

robot-clickhouse-ci-1 added the pr-synced-to-cloud The PR is synced to the cloud repo label Sep 10, 2025

dmitry-sles-novikov deleted the nats_jet_stream_support branch September 10, 2025 08:57

Nps-rf mentioned this pull request Oct 1, 2025

Add JetStream persistence support for NATS #39459

Closed

rienath mentioned this pull request Oct 18, 2025

Flaky test_storage_nats/test_nats_jet_stream.py::test_nats_overloaded_insert #88775

Closed

Conversation

dmitry-sles-novikov commented Jul 31, 2025 • edited by antaljanosbenjamin Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Documentation entry for user-facing changes

Uh oh!

clickhouse-gh bot commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

antaljanosbenjamin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dmitry-sles-novikov commented Aug 14, 2025

Uh oh!

antaljanosbenjamin commented Aug 26, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dmitry-sles-novikov commented Aug 28, 2025

Uh oh!

antaljanosbenjamin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dmitry-sles-novikov commented Aug 29, 2025

Uh oh!

antaljanosbenjamin commented Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

antaljanosbenjamin commented Sep 9, 2025

Uh oh!

dmitry-sles-novikov commented Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

antaljanosbenjamin commented Sep 10, 2025

Uh oh!

antaljanosbenjamin commented Sep 10, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dmitry-sles-novikov commented Jul 31, 2025 •

edited by antaljanosbenjamin

Loading

clickhouse-gh bot commented Jul 31, 2025 •

edited

Loading

antaljanosbenjamin commented Sep 8, 2025 •

edited

Loading

dmitry-sles-novikov commented Sep 9, 2025 •

edited

Loading