Getting Started

Prerequisites¶

To verify that Apache Kafka is running, run the following command:
```
docker ps | grep kafka
```
If it is not running, then set up Kafka. For instructions, see Kafka Setup.

For Existing Applications¶

Add Karafka to your Gemfile:

# Make sure to install Karafka 2.5 as Karafka 1.4 is no longer maintained
bundle add karafka --version ">= 2.5.0"

To install Karafka for both Rails and standalone applications, run the following command:
```
bundle exec karafka install
```
Result: All necessary files and directories are generated:

karafka.rb — the main file where you configure Karafka and define which consumers should consume what topics
app/consumers/example_consumer.rb — an example consumer
app/consumers/application_consumer.rb — the base consumer from which all consumers should inherit

To produce test messages, open the development console and enter:

# Works from any place in your code and is thread-safe
# You usually want to produce async but here it may raise exception if Kafka is not available, etc
Karafka.producer.produce_sync(topic: 'example', payload: { 'ping' => 'pong' }.to_json)

To start consuming messages, run the Karafka server:

bundle exec karafka server

# example outcome
[7616dc24-505a-417f-b87b-6bf8fc2d98c5] Polled 2 messages in 1000ms
[dcf3a8d8-0bd9-433a-8f63-b70a0cdb0732] Consume job for ExampleConsumer on example started
{"ping"=>"pong"}
{"ping"=>"pong"}
[dcf3a8d8-0bd9-433a-8f63-b70a0cdb0732] Consume job for ExampleConsumer on example finished in 0ms

Below is the demo of the installation process:

(Optional) To install and configure the Web UI, follow this documentation section.

For New Applications (Starting From Scratch)¶

Create a Gemfile:

# Gemfile
source "https://rubygems.org"

gem "karafka", ">= 2.5.0"

Run: bundle install
To install Karafka for both Rails and standalone applications, run the following command:
```
bundle exec karafka install
```
Result: All necessary files and directories are generated:
- karafka.rb — the main file where you configure Karafka and where you define which consumers should consume what topics.
- app/consumers/example_consumer.rb — an example consumer.
- app/consumers/application_consumer.rb — the base consumer from which all consumers should inherit.

Run a development console to produce messages to this example topic:

# Works from any place in your code and is thread-safe
# You usually want to produce async but here it may raise exception if Kafka is not available, etc
Karafka.producer.produce_sync(topic: 'example', payload: { 'ping' => 'pong' }.to_json)

To start consuming messages, run the karafka server:

bundle exec karafka server

# example outcome
[7616dc24-505a-417f-b87b-6bf8fc2d98c5] Polled 2 messages in 1000ms
[dcf3a8d8-0bd9-433a-8f63-b70a0cdb0732] Consume job for ExampleConsumer on example started
{"ping"=>"pong"}
{"ping"=>"pong"}
[dcf3a8d8-0bd9-433a-8f63-b70a0cdb0732] Consume job for ExampleConsumer on example finished in 0ms

(Optional) To install and configure the Web UI, see Getting Started with the Web UI.

Example applications¶

If you have any problems setting up Karafka or need a ready application to play with, clone our examples repository:

git clone https://github.com/karafka/example-apps ./example_apps

For instructions, see Karafka Example Applications Wiki.

Use cases, edge cases, and usage examples¶

Karafka ships with a full integration test suite that illustrates various use cases and edge cases of working with Karafka and Kafka. For comprehensive understanding of our framework, visit Integrations directory of the Karafka repository.