A high-performance log aggregation pipeline built with Rust and Go.

[Rust Agent] → gRPC/protobuf → [Go Server] → SQLite → [Web Dashboard]

• Rust agent tails files in real-time with low memory footprint
• Batched gRPC transport with automatic retry and backoff
• JSON and plaintext log parsing with level detection
• SQLite persistence with indexed queries
• REST API with filtering by level, host, and keyword
• Real-time web dashboard with live polling

Run locally:

# Terminal 1 - Start server
cd server && go run cmd/server/main.go

# Terminal 2 - Start agent
cd agent && cargo run

# Terminal 3 - Send test logs
echo "ERROR something failed" >> /tmp/logflow-test.log
echo '{"level":"info","message":"user logged in"}' >> /tmp/logflow-test.log

Open http://localhost:8080 for the dashboard.

Run with Docker:

docker-compose up --build

• Rust: tokio, tonic, prost, serde
• Go: net/grpc, modernc/sqlite, net/http
• Transport: gRPC + Protocol Buffers
• Storage: SQLite