Redis Streams — The Complete Guide
lightweight message bus connecting 13 agents without Kafka, RabbitMQ, or SQS
Redis Streams is a Redis feature (since version 5.0, 2018) that turns it into a lightweight message broker — async communication between services, without the complexity of Kafka or RabbitMQ. Redis itself is an in-memory key-value store running on hundreds of thousands of VPSes around the world — extremely fast (microsecond operations), easy to set up, and minimal resource usage. Streams added to it the ability to hold persistent message queues with consumer groups (groups of consumers that share work), acknowledgments (confirming a message was handled), and replay (the ability to go back to old messages). For me (Elad), Redis Streams is the 'central nervous system' of my 13-agent network on Hetzner: when a WhatsApp message hits Kami, it doesn't process it alone — it pushes a message to a stream, and various consumers (Box for nutrition, Adopter for content, Hermes for scheduling) read and react. If one agent goes down, messages wait in the stream until it returns. If we want a new agent listening to those events — we add it to a consumer group in 30 seconds. Since moving to Redis Streams (two years ago, Q2 2024), my system has been much more stable: each agent works independently, and the 'who listens to what' logic is managed in Redis instead of through direct API calls.
What this guide covers
What is Redis Streams: append-only log
Like Kafka, but in Redis without Zookeeper
A Redis Stream is an 'append-only log' data structure — a list of messages you can only append to the end, never edit in the middle. Each message gets a unique ID (millisecond timestamp + sequence), you can read a range of messages, and you can 'follow' (XREAD) new appends in real time. Conceptually it resembles a Kafka topic, but it is much simpler to operate: Redis itself is a single process, no Zookeeper, no partitions, no broker cluster. You just run Redis and you have Streams.
XADD and XREAD: the two basic operations
All you need for simple pub/sub
If you only want simple pub/sub — agent A pushes, agent B listens — you only need XADD and XREAD. No consumer groups, no acknowledgments. The Python code below shows both sides.
Consumer Groups: when multiple consumers share work
Each message is handled by exactly one consumer in the group
Consumer groups are the feature that makes Redis Streams a serious message broker. Instead of every consumer seeing every message (like pub/sub), consumers in the same group share the work — each message reaches exactly one consumer in the group. For redundancy, you spin up multiple consumers; if one falls over, the rest keep going. Plus there's the acknowledgment mechanism: a consumer says 'I handled message X', and Redis remembers. If the consumer dies before acking, the message is re-claimed by another consumer.
Patterns: fan-out, work-queue, event-sourcing
The common Streams use cases
Once you know the basics, there are several common patterns that help architect things.
Production: memory, persistence, monitoring
What to know before release
Redis in production needs three things: persistence (so you don't lose data when Redis crashes), size management (so the stream doesn't eat all memory), and monitoring.
Alternatives: Kafka, RabbitMQ, NATS
When to pick which
Redis Streams is lightweight, but not for every case. Here's a comparison to other solutions.
