Project Deep Dive
RelayOps
A CDC and replay prototype with logical replication, Redis Streams, Kafka archival, Protobuf, and gRPC replay control.
Overview
RelayOps is a prototype change data capture and replay system for PostgreSQL changes. It captures WAL events through logical replication, serializes them into a shared Protobuf contract, buffers recent events in Redis Streams for replay, archives them to Kafka, and replays them through worker-controlled scheduling.
Key Outcomes
Built an at-least-once CDC pipeline with WAL capture, Redis replay buffering, and Kafka archival.
Added replay workers with consumer-group recovery and configurable speed factors for time-travel style event playback.
Separated ingestion, replay execution, and control-plane orchestration instead of treating CDC as a single-process script.
Architecture Diagram
WAL changes move through logical replication, Redis replay buffering, Kafka archival, worker scheduling, and gRPC replay control.
Architecture
An ingestion layer captures PostgreSQL WAL changes through wal2json and replication slots, then converts supported inserts, updates, and deletes into Protobuf ChangeRecord events.
A dual transport pipeline writes captured events into Redis Streams for low-latency replay and Kafka for durable archival and downstream consumption.
Replay workers consume Redis events with consumer groups, recover pending messages, calculate timing from timestamps and speed factors, and acknowledge only after replay succeeds.
A Django control plane stores ReplayJob metadata and triggers gRPC replay requests against the replayer service.
Stack
- Python
- Django
- PostgreSQL logical replication
- Redis Streams
- Kafka
- Protobuf
- gRPC
- Docker Compose