Real-Time Data Processing Pipeline

A comprehensive real-time data processing pipeline built with Python, Apache Kafka, and Redis.

• Kafka Producer: Generates and ingests real-time data
• Kafka Consumer: Processes and transforms incoming data
• Data Aggregation: Real-time aggregation and analytics
• Redis Caching: Fast caching of processed results
• Web Dashboard: Real-time visualization of data flow and metrics
• Monitoring: System health and performance monitoring

├── requirements.txt          # Python dependencies
├── config/
│   ├── kafka_config.py      # Kafka configuration
│   └── redis_config.py      # Redis configuration
├── models/
│   ├── data_models.py       # Pydantic data models
│   └── metrics.py           # Metrics and monitoring models
├── producers/
│   ├── data_producer.py     # Kafka producer for data ingestion
│   └── sensor_producer.py   # Simulated sensor data producer
├── consumers/
│   ├── data_consumer.py     # Kafka consumer for data processing
│   └── aggregator.py        # Data aggregation consumer
├── processors/
│   ├── data_processor.py    # Data transformation logic
│   └── aggregator.py        # Aggregation logic
├── cache/
│   └── redis_client.py      # Redis caching layer
├── dashboard/
│   ├── app.py              # Dash web application
│   ├── components.py       # Dashboard components
│   └── static/            # Static assets
├── monitoring/
│   ├── metrics.py          # Prometheus metrics
│   └── health_check.py     # Health monitoring
├── utils/
│   ├── logger.py           # Logging configuration
│   └── helpers.py          # Utility functions
└── scripts/
    ├── start_kafka.py      # Kafka setup script
    ├── start_redis.py      # Redis setup script
    └── run_pipeline.py     # Main pipeline runner

1. Install Dependencies:

   pip install -r requirements.txt

2. Start Kafka and Redis:

   python scripts/start_kafka.py
   python scripts/start_redis.py

3. Run the Pipeline:

   python scripts/run_pipeline.py

4. Access Dashboard: Open http://localhost:8050 in your browser

• Generates simulated sensor data (temperature, humidity, pressure)
• Publishes to Kafka topics with configurable frequency
• Supports multiple data types and formats

• Consumes data from Kafka topics
• Performs real-time data transformation
• Calculates aggregations (min, max, avg, count)
• Stores results in Redis cache

• Transforms raw data into structured format
• Applies business logic and validation
• Handles data quality checks

• Stores processed results and aggregations
• Provides fast access to historical data
• Supports TTL for data expiration

• Real-time data visualization
• System metrics and health monitoring
• Interactive charts and graphs
• Responsive design with Bootstrap

Edit the configuration files in config/ to customize:

• Kafka broker settings
• Redis connection parameters
• Data processing rules
• Dashboard appearance

The system includes comprehensive monitoring:

• Prometheus metrics for system health
• Real-time performance indicators
• Error tracking and alerting
• Resource usage monitoring

• Python 3.8+: Core programming language
• Apache Kafka: Message streaming platform
• Redis: In-memory data store and cache
• FastAPI: Web framework for APIs
• Dash: Interactive web dashboard
• asyncio: Asynchronous programming
• Pydantic: Data validation and serialization