Pulsar <> Cortx

Inspiration

Message brokers like Kafka and Rabbit MQ are optimized for latency and throughput and have minimal retention options. In any case, retention on these brokers depends on comparatively costly storage. This project is an attempt to combine the power of fast messaging from Pulsar with near-unlimited storage offered by Seagate Cortx.

What it does

Our integration allows Apache Pulsar to use Cortx as a storage backend to offload older messages on a topic.

This very old data from Cortx can be streamed from any given message id later.

Theoretically, based on scaling promises made by Pulsar and Cortx(2^120 objects), this integration can be used to store every event ever observed in our universe, and then replay them in order from any point, hundreds of years later with consistency guarantees while preserving processing semantics(like exactly-once).

This stream replay is suitable for oracles, recommendation engines, tracing backends, backtesting algos on trading floors and heavily instrumented operations like aircrafts(to be replayed for flight simulations, for instance).

How we built it

1. We setup Cortx on a VM.
2. We use aws s3 cli to check the bucket data on Cortx S3.
3. We setup Pulsar to use Cortx as tiered storage backend.
4. We make some hacky adjustments to make sure Cortx accepts the payload sizes sent by Pulsar. See issues.
5. We publish messages on a test topic.
6. We offload the topic messages(older ones anyway) to Cortx s3.
7. We prove that the messages are offloaded based on internal stats of the topic via pulsar-admin.
8. We cross check in Cortx s3, by checking that the S3 bucket has message ledgers with index data.
9. We replay the data from the earliest message in the topic(which would be on Cortx S3 by now)

Challenges we ran into

1. Cortx's motr has a configurable layout which decides what unit_size of storage it uses for object storage. Cortx's s3 server cannot save multi part uploads whose first part has size which is not a multiple of this unit_size. There is probably a performance benefit in doing this but it prevents all multi part uploads with size smaller than unit_size(which is 1MB for default layout). We had to configure for a layout with smaller unit size to get this integration work.
2. Pulsar has misleading error messages in its tiered storage configuration module. Some documentation is outdated as well with previously supported features removed/abruptly renamed. We had to dig into code of both repos to understand the issues involved.

Accomplishments that we're proud of

After all the configuration and tweaks, it was satisfying to see 100s of 1000s of messages stream without issues and onto Cortx. I learned about two new projects in a very short time and integrated them.

This is the first time I could see that data from a stream is archived(in such significant amounts) and replayed so effortlessly and seamlessly. All the other times, I had to involve Spark jobs or Flink as intermediary between Kafka and storage. That needed additional effort for deployment and maintenance.

What we learned

Stream replay from slow storage is feasible and can be easy. I learned about how s3 server wrappers around storage work.

What's next for Pulsar Cortx

Based on the schema of Pulsar topic, we may be able to efficiently run ETL queries on topic data over the entire history of the topic using an engine like presto, or even better using something like a plugin from within Cortx. For instance, storing topic data in Cortx as parquet allows efficient SQL queries.

Being able to run ETL and stream replay from same storage, that too, from the cheapest store possible is a valuable capability.

Built With

• python
• s3