Overview

Relevant source files

• README.md
• docs/en/ContributionGuide/ContributingCode/Code-Structure.md
• docs/images/wechat.jpg
• docs/zh/ContributionGuide/ContributingCode/Code-Structure.md
• makefile
• src/common/metadata-struct/src/mqtt/topic_rewrite_rule.rs
• src/common/metadata-struct/src/schema.rs
• src/meta-service/src/raft/route/mqtt.rs
• src/schema-register/src/avro.rs
• src/schema-register/src/schema.rs

Purpose and Scope

This document provides a high-level introduction to RobustMQ, a next-generation message queue system built in Rust. It covers the project's design philosophy, core features, system architecture, and major components. This overview serves as the entry point for understanding the entire RobustMQ codebase.

For detailed information about specific subsystems:

• System architecture and design principles → See System Architecture
• MQTT broker implementation → See MQTT Broker
• Bridge and connector system → See Bridge and Connector System
• Storage layer details → See Storage Layer
• Meta service and cluster coordination → See Meta Service and Cluster Coordination
• Management and operations → See Management and Operations

---

What is RobustMQ?

RobustMQ is a high-performance, multi-protocol message queue system designed for modern cloud-native and AI-native applications. Built entirely in Rust, it provides microsecond-level latency with zero garbage collection pauses while supporting multiple messaging protocols including MQTT 3.x/5.0, Kafka, and AMQP through a unified platform.

The system employs compute-storage separation architecture, enabling stateless broker nodes that can scale elastically in seconds. This design allows RobustMQ to support various deployment scenarios from edge devices to large-scale cloud clusters.

Current Development Status: RobustMQ is in Phase 1 (Foundation building) and is not production-ready. The project is actively developing its core architecture and MQTT broker implementation.

Sources: README.md38-60

---

Design Philosophy

RobustMQ is built on two core principles:

Cloud-Native and AI-Native

The system addresses critical challenges in the AI era:

• Protocol fragmentation - Eliminates the need for multiple message queue systems by supporting MQTT, Kafka, and AMQP in a single platform
• Latency jitter - Provides microsecond-level latency suitable for real-time AI workloads
• Compute-storage coupling - Enables Serverless deployment through stateless brokers
• Cost optimization - Supports elastic scaling to reduce infrastructure costs

Simplicity and "Slow is Fast"

As a community-driven project (not commercial), RobustMQ prioritizes:

• Building a rock-solid foundation over rushing features
• Thoughtful design and architecture to avoid future refactoring
• Quality over speed, depth over breadth
• Long-term sustainability over short-term gains

The project invests significant time in Phase 1 to create a scalable technical architecture with streamlined, abstraction-friendly code that supports multi-protocol adaptation, pluggable storage, and extensibility.

Sources: README.md41-60

---

Core Features

Feature	Description	Implementation Status
High Performance	Rust-powered zero-cost abstractions with microsecond latency and zero GC pauses	✅ Active
Compute-Storage Separation	Stateless broker nodes enable second-level elastic scaling and Serverless deployment	✅ Active
Multi-Protocol Unified	Native support for MQTT 3.1/3.1.1/5.0, Kafka, and AMQP with protocol isolation	🔄 MQTT Active, Kafka/AMQP Planned
Intelligent Storage Tiering	Memory/SSD/Object storage with Topic-level engine selection for cost optimization	✅ Active
Cloud-Native Ready	Kubernetes-native with auto-scaling, service discovery, and container-friendly deployment	✅ Active
Zero-Operations	Single binary deployment with built-in Raft consensus and comprehensive management tools	✅ Active
Enterprise Security	Multi-auth support, fine-grained permissions, and end-to-end encryption	✅ Active
Full Observability	Prometheus metrics, OpenTelemetry tracing, and Grafana dashboards	✅ Active

Sources: README.md75-85

---

High-Level Architecture

RobustMQ employs a four-layer distributed architecture that separates concerns and enables independent scaling of each layer:

Sources: README.md87-101 docs/zh/ContributionGuide/ContributingCode/Code-Structure.md8-16

---

System Components Overview

Application Layer: Broker Server

The BrokerServer (src/broker-server/) serves as the central orchestrator that coordinates all protocol brokers and provides a unified service entry point.

Key Responsibilities:

• Start and manage protocol-specific broker processes (MQTT, Kafka, AMQP)
• Coordinate cluster services and node management
• Provide gRPC interfaces for inter-service communication
• Collect and expose metrics for monitoring

Core Files:

• src/broker-server/src/cluster_service.rs - Cluster service management
• src/broker-server/src/grpc.rs - gRPC service interfaces
• src/broker-server/src/metrics.rs - Performance metrics collection

The broker can run in three modes controlled by configuration flags:

• is_broker_node=true - Runs protocol brokers
• is_meta_node=true - Runs meta service
• is_engine_node=true - Runs storage engine

Sources: README.md93-94 docs/zh/ContributionGuide/ContributingCode/Code-Structure.md39-46

---

Core Services: Meta Service

The Meta Service (src/meta-service/) implements a Raft-based distributed metadata management system that coordinates the entire cluster.

Key Responsibilities:

• Store and manage cluster metadata (topics, sessions, users, ACLs, subscriptions, connectors)
• Provide distributed consensus via Raft algorithm
• Handle node registration and health monitoring
• Synchronize metadata to broker caches
• Manage automatic failover

Core Components:

Metadata Types Managed (13+ types):

• MQTT: Users, Topics, Sessions, Subscriptions, ACLs, Blacklists, Connectors, Topic Rewrite Rules, Auto-Subscribe Rules, Last Will Messages, Retain Messages
• Storage: Shards, Segments, Consumer Offsets
• Cluster: Node Info, Configuration

Sources: README.md94 docs/zh/ContributionGuide/ContributingCode/Code-Structure.md47-56 src/meta-service/src/raft/route/mqtt.rs53-67

---

Protocol Layer: MQTT Broker

The MQTT Broker (src/mqtt-broker/) implements MQTT protocol versions 3.x, 4.x, and 5.0 with full feature support.

Key Components:

Port Configuration:

• TCP: :1883
• TLS: :8883
• WebSocket: :8083
• QUIC: Configurable

Sources: README.md93 docs/zh/ContributionGuide/ContributingCode/Code-Structure.md66-74

---

Integration Layer: Bridge and Connector System (HIGHEST IMPORTANCE)

The Bridge and Connector System is RobustMQ's most important feature (importance score: 314.63), enabling MQTT messages to be forwarded to 10 different external systems.

Supported External Sinks:

1. Kafka - Apache Kafka message broker
2. Pulsar - Apache Pulsar pub-sub messaging
3. MongoDB - NoSQL document database
4. PostgreSQL - Relational database
5. MySQL - Relational database
6. RabbitMQ - AMQP message broker
7. Elasticsearch - Search and analytics engine
8. Redis - In-memory data store
9. GreptimeDB - Time-series database
10. Local File - File system storage

Architecture:

Connector Lifecycle:

• Connectors are dynamically started/stopped based on metadata from meta service
• Each connector runs in its own thread with run_connector_loop
• Reads messages from MQTT topics via MessageStorage
• Writes batches to external sinks
• Supports three failure handling strategies: Discard, DiscardAfterRetry, DeadMessageQueue

Sources: README.md96 docs/zh/ContributionGuide/ContributingCode/Code-Structure.md66-74

---

Storage Layer: Three-Tier Architecture

The Storage Layer implements a three-tier architecture providing unified storage abstraction with pluggable backends:

Storage Operations:

• create_shard, list_shard, delete_shard - Shard management
• batch_write, read_by_offset, read_by_key, read_by_tag - Data operations
• commit_offset, get_offset_by_group - Consumer group management

Performance Tiers:

• Memory: Microsecond latency for hot data
• RocksDB: Millisecond latency for warm data
• Segments: Second-level latency for cold data
• External DB: Configurable latency

Sources: README.md96-97 docs/zh/ContributionGuide/ContributingCode/Code-Structure.md132-142

---

Management Layer: Admin Server and CLI

The Admin Server (src/admin-server/) provides HTTP REST APIs for cluster management, while robust-ctl (src/cli-command/) offers command-line administration.

Admin HTTP API Endpoints:

• MQTT management: /mqtt/topic/create, /mqtt/user/create, /mqtt/session/list
• Connector management: /connector/create, /connector/delete, /connector/detail
• Cluster management: /cluster/nodes, /cluster/config
• Monitoring: Prometheus metrics on :9000, $SYS topics

CLI Command Structure:

Sources: README.md98 docs/zh/ContributionGuide/ContributingCode/Code-Structure.md87-103

---

Deployment Architecture

RobustMQ supports flexible deployment modes:

Single Binary Deployment

All services (broker, meta, storage) run in a single process:

Distributed Deployment

Services run on separate nodes with role-based configuration:

• Meta nodes: is_meta_node=true - Run Raft consensus and metadata management
• Broker nodes: is_broker_node=true - Run protocol brokers (MQTT/Kafka/AMQP)
• Engine nodes: is_engine_node=true - Run storage engine services

Configuration Example:

Sources: README.md97-98

---

Build and Development

RobustMQ uses a Makefile for common development tasks:

Command	Purpose
make run	Run broker-server in development mode
make codecheck	Run format, check, clippy, license, docs
make build	Build current platform package (basic)
make build-full	Build complete package with frontend
make test	Run unit tests with cargo nextest
make ig-test	Run integration tests
make release	Create GitHub release and upload package
make release-docker	Build and push Docker image to GHCR

Build Artifacts:

• robust-server - Main server binary (broker + meta + storage)
• robust-ctl - Command-line management tool
• robust-bench - Performance benchmarking tool

Sources: makefile1-99

---

Technology Stack

Category	Technologies
Language	Rust 1.70+
Consensus	Raft (openraft crate)
Storage	RocksDB, Memory, File Segments, MySQL, PostgreSQL
Protocol	MQTT 3.x/5.0, Kafka (planned), AMQP (planned)
Transport	TCP, TLS, WebSocket, QUIC
Serialization	Protobuf, JSON, Avro
Observability	Prometheus, OpenTelemetry, Grafana
Container	Docker, Kubernetes
Testing	cargo nextest, integration tests

Sources: README.md18 src/schema-register/src/schema.rs54-69

---

Development Roadmap

Phase 1: Foundation (Current)

Building scalable technical architecture with solid, streamlined code. Establishing foundation for multi-protocol adaptation, pluggable storage, extensibility, and elasticity. Significant time investment to avoid future refactoring.

Phase 2: MQTT Broker

Deliver stable, high-performance MQTT Broker with complete protocol support (MQTT 3.x/5.0/SN). Optimize for edge deployment with package size under 20MB.

Phase 3: Kafka Protocol

Start Kafka protocol adaptation work after achieving Phase 1 and 2 goals.

Long-term Vision: Become the next-generation cloud-native and AI-native messaging infrastructure, aspiring to Apache top-level project status.

Sources: README.md61-73

---

Schema Validation Support

RobustMQ includes Schema Registry functionality for message validation:

Supported Schema Types:

• JSON Schema - JSON structure validation using jsonschema crate
• Avro - Apache Avro binary format validation
• Protobuf - Protocol Buffers validation (planned)

Architecture:

Usage:

• Topics can be bound to schemas for automatic message validation
• Validation occurs before message publishing
• Invalid messages are rejected based on schema rules

Sources: src/schema-register/src/schema.rs23-145 src/common/metadata-struct/src/schema.rs20-69

---

Topic Rewrite Rules

RobustMQ supports regex-based topic rewriting for flexible message routing:

Data Structure:

Use Cases:

• Transform topic hierarchies during subscribe/publish
• Route messages to different topic spaces
• Apply consistent naming conventions
• Support legacy topic structures

Sources: src/common/metadata-struct/src/mqtt/topic_rewrite_rule.rs18-35 src/meta-service/src/raft/route/mqtt.rs202-219

---

Getting Started

To begin working with RobustMQ:

1. Understanding the Architecture → Read System Architecture for the four-layer design and component interactions
2. MQTT Development → See MQTT Broker for protocol implementation details
3. Connector Development → See Bridge and Connector System for adding new external sinks
4. Storage Development → See Storage Layer for storage backend implementation
5. Operations → See Management and Operations for deployment and monitoring

Quick Links:

• Official Documentation: https://robustmq.com/
• GitHub Repository: https://github.com/robustmq/robustmq
• Discord Community: https://discord.gg/sygeGRh5

Sources: README.md152-158

---

Summary

RobustMQ is an ambitious project aiming to unify multiple messaging protocols in a single, high-performance platform. Its compute-storage separation architecture, multi-protocol support, and extensive integration capabilities (10 external sinks) make it suitable for diverse use cases from edge computing to cloud-scale deployments.

The project is community-driven and prioritizes long-term architectural soundness over rapid feature delivery. While currently in early development (Phase 1), the foundation being built supports future scalability and extensibility.

Key Differentiators:

• Single platform for MQTT, Kafka, and AMQP
• Microsecond latency with Rust zero-cost abstractions
• Stateless brokers enabling Serverless deployment
• Pluggable storage with intelligent tiering
• Extensive bridge system for data integration

Sources: README.md41-60 README.md75-101