CORTX-Fluentd Integration

Inspiration

I am very interested in IoT. I truly believe it is the future of computing, especially with edge computing on the rise. As a result, we are going to have tons of data for processing and post-analysis. Fluentd can act as a log collection point and output it to the highly scalable, CORTX.

What it does

This is an integration of which Fluentd runs as a daemon on a machine. It looks for data sources given the input you choose and outputs it to CORTX via S3-type matching in Fluentd.

How I built it

I essentially used the VM provided by Seagate on Cloudshare and hooked up the endpoints after creating the configuration file for Fluentd. I tested the writes and reads to CORTX via the AWS CLI.

Challenges I ran into

It is very easy to misconfigure something. I spent nearly 3 days trying to write a custom plugin for Fluentd to support CORTX. After I came across MinIO (another S3 compatible storage) being used with Fluentd, I knew for sure I had messed up my configuration and that is indeed what had happened. The other issue I had is that the Fluentd documentation is very young and needs more care. I noticed things that seemed off and different in some places. Not sure if those are errors, but I will definitely be submitting PRs for those by creating examples in the documentation for others to use.

Accomplishments that I'm proud of

I successfully integrated a popular CNCF tool with CORTX and made my first pull request in a large project like CORTX.

What I learned

Alot of things can go wrong. Just keep trying it out :)

What's next for CORTX-Fluentd Integration

I'm going to launch Fluentd as part of the EFK stack and output my results to CORTX.

Built With

• cortx
• fluentd