Skip to content
Architecture Lab

Architecture Lab

Event-driven microservices with a message broker

Synchronous APIs for requests, a message broker for side effects, and the patterns that keep it reliable: idempotent consumers, retries and dead-letter queues.

  • Microservices
  • RabbitMQ
  • Django
  • REST APIs

Illustrative reference design built from general engineering practice. It is not the architecture of any employer's or client's production system.

Why it matters

Order flows touch billing, notifications and third-party systems that can be slow or unavailable. Doing all of that inside one request makes every dependency a point of failure for the user. I built Django microservices with RabbitMQ messaging for a telecom BSS platform; this page is a general version of that pattern, not that platform's design.

Diagram

Event-driven microservices: request path and asynchronous consumersOrder channels call an API gateway that routes to customer, order and catalog services, each with its own database. The order service publishes events to a message broker. A billing integration consumer calls an external payment system, a notification consumer sends messages, and messages that keep failing move to a dead-letter queue.Services (each owns its data)Order channelsWeb, CRM, partnersAPI gatewayAuth, routing, limitsCustomer serviceOwn databaseOrder serviceOwn database + outboxCatalog serviceOwn databaseMessage brokerRabbitMQ exchangeBilling consumerIdempotent handlerPayment systemThird partyNotification consumerEmail, SMSDead-letter queueInspect and replayretries used
  • Request / call
  • Asynchronous
  • Failure path
Illustrative
Diagram as text

Components

  • Order channels (input or output): Web, CRM, partners
  • API gateway (control point): Auth, routing, limits
  • Customer service (service): Own database
  • Order service (service): Own database + outbox
  • Catalog service (service): Own database
  • Message broker (queue or worker): RabbitMQ exchange
  • Billing consumer (service): Idempotent handler
  • Payment system (external system): Third party
  • Notification consumer (service): Email, SMS
  • Dead-letter queue (data store): Inspect and replay

Connections

  • Order channels → API gateway
  • API gateway → Customer service
  • API gateway → Order service
  • API gateway → Catalog service
  • Order service → Message broker (asynchronous)
  • Message broker → Billing consumer (asynchronous)
  • Message broker → Notification consumer (asynchronous)
  • Billing consumer → Payment system
  • Message broker → Dead-letter queue (failure path, retries used)

How it flows

  1. Accept the request

    The gateway authenticates the caller and routes the request to the service that owns the data.

  2. Commit locally

    The order service writes the order and an outbox record in the same database transaction.

  3. Publish

    A relay publishes outbox records to the broker, so an event is never lost if the service crashes after committing.

  4. Respond

    The user gets a response as soon as the order is stored; billing and notifications happen asynchronously.

  5. Consume

    Consumers process events and acknowledge them only after the work succeeds.

  6. Retry or park

    Failed messages are retried with backoff; after the retry limit they move to a dead-letter queue for inspection and replay.

Boundaries

Data ownership

Each service owns its database. Other services use its API or its events, never its tables.

Synchronous vs asynchronous

Only work the user must wait for is on the request path; everything else is an event.

Third-party boundary

External systems are reached only through dedicated consumers with timeouts, so their failures stay contained.

Failure handling

Consumer crashes mid-message
The message wasn't acknowledged, so the broker redelivers it; handlers are idempotent, keyed by event ID.
Poison message
After a fixed number of attempts it goes to the dead-letter queue instead of blocking the queue.
Slow or failing third party
Calls have timeouts and retries with backoff, and a circuit breaker stops calls while the system is down.
Service crashes after commit, before publish
The outbox relay publishes the event when the service recovers.

Trade-offs

Events for side effects

Gain
Users aren't blocked by billing or notification delays, and new consumers can be added without changing the order service.
Cost
Eventual consistency: other services see the order a moment later.

Transactional outbox

Gain
No lost or phantom events.
Cost
An extra table and a relay process to run.

At-least-once delivery

Gain
Simple and reliable with standard broker features.
Cost
Every consumer must handle duplicates.

Next design

Chaos engineering experiment platform