So if we want the best possible safety guarantees and are willing to pay for it with ingestion performance - every incoming data chunk can be persisted to disk.

Cool! We are seemly on the same line. 😄 For your HSE, I will have a look.

This is not a classic LSM. The merge happens mainly in storage part, but it gives excellent compression rate and query performance.

Yes. And you said "the merge happens mainly in storage" is understandable. This also makes the I said OLTP ingestion(one-row-by-one-row) prohibitive for CH. The logic as I said belongs to OLAP. But CH's delayed merging behavior worsened the already heavy data movement on external storage. In the heavy bigdata ingestions (but this should be common for bigdata), I have observed severe performance degradation.

Can you point me to TB part that implements storage?

thanks for interesting. I should roll out more readings for community. But the time is a bit tight. I hope make it coming soon. Not sure if the people in community has some interests to help:)

Let's do a quick note before more detailed explanation:

TB uses a data structure that we call "Partition Tree" (now we use a sled as a implementation the api is in the PartStore here). The storage lay is thin now and embedded in the runtime::write. The data is directly written to the partition file. While maintaining the append only write performance, it avoids the subsequent compact overhead of the LSM structure.

Write Ahead Log #49

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions