TiFlow Overview

Relevant source files

• .codecov.yml
• .github/pull_request_template.md
• .golangci.yml
• CODE_OF_CONDUCT.md
• CONTRIBUTING.md
• Makefile
• README.md
• README_DM.md
• README_TiCDC.md
• SECURITY.md
• errors.toml
• go.mod
• go.sum
• pkg/errors/cdc_errors.go
• pkg/sink/codec/avro/confluent_schema_registry.go
• pkg/sink/codec/avro/confluent_schema_registry_test.go
• pkg/sink/codec/avro/glue_schema_registry.go
• tools/check/go.mod
• tools/check/go.sum
• tools/check/tools.go

Purpose and Scope

TiFlow is a comprehensive data replication and migration platform for TiDB/TiKV clusters. It provides two primary subsystems:

• TiCDC (Change Data Capture): Real-time, continuous replication of incremental changes from TiDB/TiKV to various downstream systems including MySQL, TiDB, Kafka, Pulsar, and cloud storage.
• DM (Data Migration): Full and incremental data migration from MySQL/MariaDB sources to TiDB, supporting both one-time migration and continuous synchronization.

This document provides a high-level overview of the TiFlow platform architecture, its major components, and design philosophy. For detailed information about specific subsystems:

• TiCDC internals: See TiCDC (Change Data Capture)
• DM internals: See DM (Data Migration)
• Build system details: See Build System and Dependencies
• Error handling: See Error Handling Framework

---

Platform Architecture

The following diagram illustrates the overall TiFlow platform architecture, showing how TiCDC and DM integrate with external systems:

Sources: go.mod1-128 Makefile1-130 cdc/model/owner.go1-24 dm/master/server.go14-66 dm/worker/server.go14-43

---

TiCDC: Change Data Capture

TiCDC is a distributed system for capturing and replicating change data from TiDB/TiKV clusters in real-time. It operates as a cluster of Capture nodes where one is elected as Owner for orchestration, while others act as Processors handling table replication.

Core Components

Component	Code Location	Purpose
CDC Server	cmd/cdc/main.go	Entry point for TiCDC service
Capture	cdc/capture/	Node in TiCDC cluster, runs Owner or Processor
Owner	cdc/owner/	Leader that schedules tables and manages changefeeds
Processor	cdc/processor/	Worker that replicates assigned tables
KV Client	cdc/kv/client.go	Pulls change events from TiKV
Mounter	cdc/entry/mounter.go	Decodes KV events into row changes
Sink	cdc/sink/	Writes data to downstream systems

Data Flow Pipeline

TiCDC Architecture Details: TiCDC uses a distributed architecture where the Owner (elected via etcd) manages changefeed lifecycle and schedules table replication across Processor nodes. Each processor pulls KV events from TiKV using gRPC streaming, sorts them by commit timestamp, decodes them using schema information, and writes to configured downstream sinks with at-least-once semantics.

Sources: cdc/model/owner.go14-144 dm/master/server.go102-139 pkg/errors/cdc_errors.go1-333

---

DM: Data Migration

DM (Data Migration) uses a master-worker architecture for migrating data from MySQL/MariaDB to TiDB. It supports full data migration (dump + load) and incremental synchronization via binlog replication.

Core Components

Component	Code Location	Purpose
DM-Master	cmd/dm-master/, dm/master/server.go	Cluster coordinator, task scheduler
DM-Worker	cmd/dm-worker/, dm/worker/server.go	Executes migration tasks
dmctl	cmd/dm-ctl/, dm/ctl/	Command-line control tool
Scheduler	dm/master/scheduler/	Assigns tasks to workers
Syncer	dm/syncer/syncer.go	Binlog replication unit
Dumper	dm/dumpling/	Full data dump unit
Loader	dm/loader/	Data load unit

DM Master-Worker Architecture

DM Architecture Details: DM-Master nodes form a cluster with leader election. The leader master schedules tasks to workers via gRPC. Each worker processes one MySQL source through a pipeline of units: Dumper exports data, Loader imports to TiDB, and Syncer performs continuous binlog replication. The system supports shard DDL coordination for merging multiple MySQL schemas.

Sources: dm/master/server.go68-155 dm/worker/server.go44-95 dm/worker/subtask.go42-125 dm/ctl/ctl.go1-34

---

Shared Infrastructure

TiFlow provides common infrastructure shared by both TiCDC and DM subsystems.

Build System

The project uses a sophisticated Makefile-based build system with Go modules for dependency management:

Key Build Configurations:

• Go Version: go 1.25.5 (go.mod3)
• CGO: Conditionally enabled based on OS and FIPS requirements (Makefile48-70)
• LDFLAGS: Injects version, build timestamp, git hash, and git branch (Makefile120-125)
• Build Targets:
  • TiCDC: make cdc → bin/cdc (Makefile160-161)
  • DM: make dm → bin/dm-master, bin/dm-worker, bin/dmctl (Makefile401-418)
  • Consumers: kafka_consumer, storage_consumer, pulsar_consumer (Makefile163-171)

Sources: Makefile1-130 go.mod1-128

Error Handling Framework

TiFlow uses a centralized RFC-style error framework with structured error codes:

Error Framework Structure:

• Error Catalog: errors.toml defines ~1000+ error codes with descriptions (errors.toml1-1000)
• Error Normalization: Uses errors.Normalize() for consistent error creation (pkg/errors/cdc_errors.go23-333)
• Error Categories:
  • CDC Errors: KV operations, sink operations, encoding/decoding (pkg/errors/cdc_errors.go24-333)
  • DM Errors: Task management, worker operations, replication errors
• RFC Codes: Format Component:ErrName (e.g., CDC:ErrChangeFeedNotExists) (errors.toml25-28)

Sources: errors.toml1-130 pkg/errors/cdc_errors.go14-333

Configuration Management

Configuration Type	Code Location	Description
ServerConfig	pkg/config/	Global TiCDC server settings
ReplicaConfig	cdc/model/	Per-changefeed configuration
SinkConfig	cdc/sink/	Downstream sink parameters
TaskConfig	dm/config/	DM task configuration
SubTaskConfig	dm/config/	DM subtask configuration

Configuration Loading Priority: CLI args → Config files (TOML) → etcd runtime config → Defaults

Sources: dm/master/server.go102-106 dm/worker/server.go86-87

Observability Stack

TiFlow provides comprehensive monitoring through Prometheus metrics and Grafana dashboards:

Component	Location	Purpose
Metrics Code	Various *_metrics.go files	Prometheus metric definitions
Grafana Dashboards	metrics/grafana/ticdc.json	Pre-built visualization panels
Alert Rules	metrics/ticdc.rules.yml	Alertmanager configurations

Key Metrics Categories:

• TiCDC: Checkpoint lag, resolved TS lag, table count, sink throughput
• DM: Binlog position, replication lag, DDL coordination status
• System: Go runtime metrics, gRPC latency, etcd operations

Sources: dm/master/server.go42 Makefile197-198

---

External Dependencies and Integration

TiFlow integrates with multiple external systems for coordination, data source, and data targets.

Source Systems

Coordination Infrastructure

Both TiCDC and DM rely on etcd for distributed coordination:

etcd Usage:

• Leader/Owner Election: TTL-based leases with automatic failover (dm/master/server.go68-87)
• State Storage: Changefeed info, task configs, worker registration (dm/master/server.go111)
• Coordination: DDL synchronization, table scheduling

Sources: dm/master/server.go68-139 dm/worker/server.go70-71

Downstream Targets

TiFlow supports multiple downstream systems:

Target Type	TiCDC Support	DM Support	Code Location
MySQL	✓	✓	cdc/sink/mysql/, dm/syncer/
TiDB	✓	✓	Same as MySQL
Kafka	✓	✗	cdc/sink/kafka/
Pulsar	✓	✗	cdc/sink/pulsar/
Cloud Storage	✓	✗	cdc/sink/cloudstorage/

Protocol Support (TiCDC):

• Avro: With schema registry (Confluent/AWS Glue)
• Canal-JSON: MySQL binlog event format
• Open Protocol: TiCDC native JSON format
• Maxwell: Maxwell-compatible JSON format
• Debezium: Debezium-compatible JSON format
• CSV: Comma-separated values

Sources: go.mod6-127 pkg/errors/cdc_errors.go172-333

---

Design Philosophy

TiFlow's architecture reflects several key design principles:

1. Distributed and Scalable

Both TiCDC and DM are designed as distributed systems that scale horizontally:

• TiCDC: Multiple Capture nodes process tables in parallel, with automatic rebalancing
• DM: Multiple workers handle different MySQL sources concurrently
• Leader Election: Uses etcd for fault-tolerant coordination (dm/master/server.go68-87)

2. Pluggable Sink Architecture

TiCDC provides a unified sink interface for extensibility:

• Common Sink interface abstracts downstream systems
• Easy to add new downstream targets by implementing the interface
• Supports multiple protocols and encoding formats

3. Exactly-Once Semantics (Target)

Both systems aim for data consistency:

• TiCDC: Transactional sinks ensure atomic writes (cdc/model/owner.go79-98)
• DM: Checkpoint mechanism enables idempotent replay
• DDL Coordination: Shard DDL locks ensure schema changes are applied correctly

4. Comprehensive Error Handling

Centralized error framework enables:

• Structured Errors: RFC-style error codes for easy diagnosis (errors.toml1-130)
• Retry Logic: Configurable backoff strategies
• Error Reporting: Errors propagated to API and stored in etcd

5. Observable and Debuggable

Rich instrumentation throughout the codebase:

• Metrics: Prometheus metrics for all critical paths
• Dashboards: Pre-built Grafana dashboards
• Logging: Structured logging with context
• Debug APIs: HTTP endpoints for runtime inspection

Sources: dm/master/server.go68-155 cdc/model/owner.go1-182 errors.toml1-130 Makefile197-198

---

Project Structure

High-level directory organization of the TiFlow repository:

Directory	Purpose	Key Entry Points
cdc/	TiCDC subsystem	capture/, owner/, processor/, sink/
dm/	DM subsystem	master/, worker/, syncer/, loader/
cmd/	Binary entry points	cdc/, dm-master/, dm-worker/, dm-ctl/
pkg/	Shared utilities	config/, errors/, etcdutil/, log/
engine/	Dataflow engine (new)	Experimental distributed execution engine
metrics/	Observability	grafana/, alert rules
tests/	Integration tests	integration_tests/
deployments/	Deployment configs	Docker Compose, Kubernetes manifests

Binary Outputs (from make build):

• bin/cdc - TiCDC server
• bin/dm-master - DM master server
• bin/dm-worker - DM worker server
• bin/dmctl - DM control CLI

Sources: Makefile141-142 go.mod1-3

---

Getting Started

To build TiFlow from source:

Build Requirements:

• Go 1.25.5 or later (go.mod3)
• CGO enabled (on macOS or when FIPS mode is enabled) (Makefile48-70)
• ~2.5GB disk space for dependencies

Sources: Makefile1-142 go.mod1-5