Add shards to data type `Map`#47045

Closed

CurtizJ wants to merge 34 commits intoClickHouse:masterfrom

CurtizJ:sharded-map

Member

CurtizJ commented Mar 1, 2023

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

TBD

Documentation entry for user-facing changes

Documentation is written (mandatory for new features)

CurtizJ added 2 commits

February 28, 2023 02:49


          add shards to data type Map

944ed18


          optimize scatter in Map

835325b

robot-clickhouse-ci-2 added the pr-feature label

CurtizJ added 4 commits

March 1, 2023 15:56


          optimize scatter of string columns

7aaac1d


          optimize reading of Map columns


          fix build

6b15256


          fix build

ab90640

Contributor

den-crane commented Mar 3, 2023

Sorry, but I think "sharding" kinda is overused. I would name it buckets or baskets.

Member Author

CurtizJ commented Mar 3, 2023

Ok, we can rename shards to buckets in that case.

CurtizJ added 3 commits

March 6, 2023 18:37


          optimize merges of type Map

5d6d0e8


          fix performance of gathering Map column

f8b7ca2


          fix type Map with LowCardinality

969320b

CurtizJ force-pushed the sharded-map branch from 68d922d to 969320b Compare

March 7, 2023 16:22

CurtizJ added 17 commits

March 9, 2023 23:12


          fix merges with column ttl

15ac072


          Merge remote-tracking branch 'upstream/master' into HEAD

a7637d2


          support reading of subcolumns in Map serialization

f3d665b


          fix style check

066ff22


          support shards of type Map with old analyzer

9e7f206


          better reading of subcolumns in type Map

416c3b8


          fix fuzzer with type Map

004ece6


          Merge remote-tracking branch 'upstream/master' into HEAD

58fd627


          fix reading of subcolumns from type Map

ef51b56


          do not use substream cache if it is not needed

8b3cb93


          fix merging Map column with compact parts

c624639


          temporary enable shard in type Map by default

a538bbc


          fix scatter of string columns

368bff7


          fix default number of shards in Map

961f41b


          fix skip indexes with type Map

fe70a73


          Merge remote-tracking branch 'upstream/master' into HEAD

1310e8b


          fix build

2786f44

alexey-milovidov mentioned this pull request

Optimize system.asynchronous_metric_log #56635

Closed

UnamedRus mentioned this pull request

Add new TimeSeries table engine to handle Prometheus protocols #64183

Merged

20 tasks

CurtizJ added 2 commits

June 8, 2024 23:18


          Merge remote-tracking branch 'upstream/master' into HEAD

3e10476


          fix build

860ba90

Contributor

robot-clickhouse-ci-1 commented Jun 9, 2024 •

edited by robot-clickhouse

Loading

This is an automated comment for commit af55627 with description of existing statuses. It's updated for the latest CI running

❌ Click here to open a full report in a separate page

Check name	Description	Status
A Sync	If it fails, ask a maintainer for help	⏳ pending
CI running	A meta-check that indicates the running CI. Normally, it's in success or pending state. The failed status indicates some problems with the PR	❌ failure
ClickHouse build check	Builds ClickHouse in various configurations for use in further steps. You have to fix the builds that fail. Build logs often has enough information to fix the error, but you might have to reproduce the failure locally. The cmake options can be found in the build log, grepping for cmake. Use these options and follow the general build process	⏳ pending
Fast test	Normally this is the first check that is ran for a PR. It builds ClickHouse and runs most of stateless functional tests, omitting some. If it fails, further checks are not started until it is fixed. Look at the report to see which tests fail, then reproduce the failure locally as described here	❌ failure
Mergeable Check	Checks if all other necessary checks are successful	❌ failure
PR Check	Checks correctness of the PR's body	❌ failure

Successful checks

Check name	Description	Status
Docs check	Builds and tests the documentation	✅ success
Style check	Runs a set of checks to keep the code style clean. If some of tests failed, see the related log from the report	✅ success

CurtizJ added 6 commits

June 10, 2024 14:40


          type map support dynamic number of shards

1e7d4e3


          remote for old analyzer

e94f109


          type map support dynamic number of shards

c30b6e2


          ColumnMap support shards (wip)

aef64c6


          Merge remote-tracking branch 'upstream/master' into HEAD

03557b7


          rework functions for type Map

af55627

Contributor

clickhouse-gh bot commented Aug 20, 2024

Dear @CurtizJ, this PR hasn't been updated for a while. Will you continue working on it? If not, please close it. Otherwise, ignore this message.

2 similar comments

Contributor

clickhouse-gh bot commented Oct 22, 2024

Dear @CurtizJ, this PR hasn't been updated for a while. Will you continue working on it? If not, please close it. Otherwise, ignore this message.

Contributor

clickhouse-gh bot commented Dec 24, 2024

Dear @CurtizJ, this PR hasn't been updated for a while. Will you continue working on it? If not, please close it. Otherwise, ignore this message.

alexey-milovidov mentioned this pull request

Roadmap 2025 #74046

Closed

76 tasks

Contributor

clickhouse-gh bot commented Feb 25, 2025

Dear @CurtizJ, this PR hasn't been updated for a while. Will you continue working on it? If not, please close it. Otherwise, ignore this message.

den-crane mentioned this pull request

Subcolumn storage for Map #78373

Open

Contributor

clickhouse-gh bot commented Apr 29, 2025

Dear @CurtizJ, this PR hasn't been updated for a while. Will you continue working on it? If not, please close it. Otherwise, ignore this message.

Contributor

EmeraldShift commented Apr 29, 2025

Is the plan for the number of shards/buckets to be a constant, or dynamic based on the number of keys? If so, maybe #78373 could be implemented completely by this PR instead of requiring anything new?

Contributor

clickhouse-gh bot commented Jul 1, 2025

Dear @CurtizJ, this PR hasn't been updated for a while. Will you continue working on it? If not, please close it. Otherwise, ignore this message.

2 similar comments

Contributor

clickhouse-gh bot commented Sep 2, 2025

Dear @CurtizJ, this PR hasn't been updated for a while. Will you continue working on it? If not, please close it. Otherwise, ignore this message.

Contributor

clickhouse-gh bot commented Nov 4, 2025

Dear @CurtizJ, this PR hasn't been updated for a while. Will you continue working on it? If not, please close it. Otherwise, ignore this message.

Member Author

CurtizJ commented Dec 30, 2025

Maybe it will be implemented on top of shared data of JSON data type later, or maybe not.

CurtizJ closed this

Contributor

EmeraldShift commented Dec 30, 2025 •

edited

Loading

Implementing a map via JSON storage (dynamic sybcolumns; shared-data, either sharded or Advanced), but without the front-end overhead and restrictions of JSON (e.g. UTF-8, #75017; Variant/Dynamic typing) would be ideal for lots of Observability use cases I have. It's a bit unfortunate that the innovative storage and the JSON front-end are so tightly coupled, or at least that the former isn't reusable. (tracked in #78373)

Member Author

CurtizJ commented Dec 30, 2025

Implementing a map via JSON storage, but without the front-end overhead and restrictions of JSON

Yes, the idea is exactly about that.

alexey-milovidov mentioned this pull request

ClickHouse roadmap 2026 #93288

Open

72 tasks

Member

Avogar commented Mar 10, 2026

For those who subscubed to this PR, there is a new implementation for automatic buckets in Map type: #99200

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels