Unbreakable Microservices: Leveraging RabbitMQ, .NET, and the Outbox Pattern
Improve the resilience of your microservices using the Outbox pattern
Resilient microservices architecture
In this post, we will demonstrate how to build a resilient microservices system using RabbitMQ and the Outbox Pattern. We will create an ordering service that saves order events in an outbox table and a notification service that processes these events. Even if RabbitMQ goes down temporarily, our system will ensure no data is lost, and all orders will be processed once the queue is back online, maintaining seamless operation and customer satisfaction.
Microservices are great, but…
Microservices offer significant advantages when built correctly, yet their core strength—being distributed—can also be a key challenge. Distributed systems enhance our flexibility in choosing the technology stack and provide remarkable resilience, high availability, and scalability. However, this same distributed nature introduces complexities that must be carefully managed to fully realize these benefits.
Today, we'll address a fundamental challenge with microservices: communication. Specifically, we'll discuss what happens when two services—Service A and Service B—communicate through a medium, and Service B becomes unavailable while Service A completes its task. We'll explore how to save the state and ensure that Service B processes its part once it comes back online after downtime.
In a distributed system, managing state and ensuring reliable communication between services is crucial. When Service B is down, we need mechanisms to ensure that Service A's actions are not lost and that Service B can pick up where it left off once it is operational again. Several strategies can be employed to handle this scenario:
Message Queuing: Use message brokers like RabbitMQ, Kafka, AWS SQS or Azure Service Bus to queue the requests from Service A. The message queue will hold these requests until Service B is back online and can process them.
Event Sourcing: Implement event sourcing to record all changes to the state as a sequence of events. Service A can publish events to an event store, and Service B can consume these events and update its state accordingly once it is available.
Circuit Breaker Pattern: Use a circuit breaker pattern to detect failures and encapsulate the logic of preventing a failure from constantly recurring. If Service B is down, the circuit breaker trips and Service A can either retry after some time or log the request for later processing.
Retry Mechanisms: Implement retry mechanisms with exponential backoff in Service A. This way, Service A will attempt to resend the request to Service B after a specified interval, increasing the interval with each attempt.
Fallback Strategies: Define fallback strategies where Service A can take alternative actions or log the state to a persistent storage to be processed later.
Today, we're going to explore how to use RabbitMQ with the outbox pattern to address communication issues in microservices. This approach helps ensure that messages between services are reliably delivered, even if one of the services is temporarily unavailable.
What is RabbitMQ?
RabbitMQ is a message broker software that facilitates the exchange of messages between applications, enabling asynchronous communication. It allows messages to be queued, stored, and processed reliably by decoupled components in a distributed system.
What is the Outbox Pattern?
The Outbox Pattern is a design pattern used to ensure reliable message delivery in distributed systems.
It involves storing messages in an "outbox" table within the same database transaction as the business data changes. A separate process then reads from this table and publishes the messages to a message broker, ensuring that messages are not lost even if the broker is temporarily unavailable.
What’s the issue we’re trying to solve?
Microservices usually communicate asynchronously via some kind of medium, like a message broker. This allows them to be decoupled from each other, resilient, and scalable (for example, we can add more consuming instances). However, there are a few potential problems with this approach:
What happens if the broker is down?
What happens if the end service is down?
Many brokers save their state, so even if they go down, you can count on them to restore their state when they come back online, which addresses the first issue. However, for the second issue, we need to take additional measures. Today, we’ll see how we can use the Outbox pattern to solve this.
For the rest of this post, I will use a few snippets from the solution that I’ve prepared. Please make sure to clone the code from the repository by clicking on the link below so you can follow along.
( Click here to get the source code from GitHub to follow along )
Also, you need to make sure that you have RabbitMQ running. The easiest way to do so is by using Docker to run it inside a container.
docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3.13-managementThe source code provides a simplified example of an ordering microservice and a notification microservice. The ordering service is responsible for creating orders and pushing them into a queue. The notification service takes those orders and sends emails to the users to inform them that their order is on its way. Simple as it is, this example perfectly illustrates the issues we described above.
So how are we fixing the problems from the image above?
The solution lies in how the ordering service handles the pushing of a created order event. Instead of directly pushing the event into a queue—which might not be available—the service saves the order in its own database and creates a transaction that also includes an entry in another table, the outbox table.
public Order? CreateOrder(Order order)
{
using (var transaction = context.Database.BeginTransaction())
{
context.Orders.Add(order);
context.SaveChanges();
var outbox = new OrderOutbox
{
AggregateId = order.Id,
AggregateType = "Order",
EventType = "OrderCreated",
EventPayload = JsonSerializer.Serialize(order),
DateTimeOffset = DateTimeOffset.Now,
Processed = false
};
context.Outbox.Add(outbox);
context.SaveChanges();
transaction.Commit();
}
return context.Orders.FirstOrDefault(o => o.Id == order.Id);
}(OrderService.cs)
Why is this done? Because the outbox table is then used by the OrderOutboxProcessor, a hosted service running every 20 seconds. This processor queries the outbox table and pushes any unprocessed entries into the queue.
private void ProcessOutbox(object state)
{
using (var scope = _serviceProvider.CreateScope())
{
var _context = scope.ServiceProvider.GetRequiredService<OrderingDbContext>();
try
{
// Create RabbitMQ queue called created-orders
var factory = new ConnectionFactory() { HostName = "localhost" };
using var connection = factory.CreateConnection();
using var channel = connection.CreateModel();
channel.QueueDeclare(queue: "created-orders",
durable: true,
exclusive: false,
autoDelete: false,
arguments: null);
// Process outbox
var entries = _context.Outbox.Where(x => x.Processed == false).Take(10).ToList();
foreach (var entry in entries)
{
// push to created-orders queue
var body = Encoding.UTF8.GetBytes(entry.EventPayload);
channel.BasicPublish(exchange: "",
routingKey: "created-orders",
basicProperties: null,
body: body);
// mark as processed
entry.Processed = true;
}
_context.Outbox.UpdateRange(entries);
_context.SaveChanges();
}
catch (Exception ex)
{
Console.WriteLine("An error occurred while processing the outbox: " + ex.Message);
}
}
}(OrderOutboxProcessor.cs)
This approach ensures that even if the queue is down, the messages will be pushed once the queue is back online, thus making the system much more resilient.
Then the NotificationsService (which is subscribed to the created orders queue) will receive the new event through the OrderReceivedProcessor class and will send the emails.
public OrderReceivedProcessor(IServiceProvider serviceProvider)
{
_serviceProvider = serviceProvider;
var factory = new ConnectionFactory() { HostName = "localhost" };
_connection = factory.CreateConnection();
_channel = _connection.CreateModel();
_channel.QueueDeclare(queue: "created-orders",
durable: true,
exclusive: false,
autoDelete: false,
arguments: null);
}
public Task StartAsync(CancellationToken cancellationToken)
{
Console.WriteLine("Order Received Processor Service is starting.");
var consumer = new EventingBasicConsumer(_channel);
consumer.Received += (model, ea) =>
{
var body = ea.Body.ToArray();
var message = Encoding.UTF8.GetString(body);
Console.WriteLine("New order received {0}", message);
Console.WriteLine("Sending notification email to the user...");
};
_channel.BasicConsume(queue: "created-orders",
autoAck: true,
consumer: consumer);
return Task.CompletedTask;
}(OrderReceivedProcessor.cs)
Let’s try our resilience
Now that you have docker, the source code and RabbitMQ installed let’s see how we can see our resiliency in action. To do so, please follow the following steps.
Stop the RabbitMQ container (this is important for our example!)
Run both projects
A few things should happen now. Since our message broker is down the communication between the services won’t happen until it’s back up.
OrderingService is running, but the hosted service responsible for processing the outbox is erroring out because the queue is unreachable.
The same is true for the NotificationsService which is trying to reconnect to RabbitMQ every minute.
Now, open OrderingService.http file and create a new order by clicking on Send request.
You’ll receive something similar to the following indicating a new order has been created.
You can check the order has been created by clicking on the GET request.
Now click on the outbox.
As you can see, the entry for this order has not been processed yet, indicated by the processed key having a value of false. This is because RabbitMQ is currently down. However, we now have a record of our orders that will eventually be picked up by our processes once the queue is back up. Let's try that now. Start the RabbitMQ container again and see what happens.
docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3.13-managementIf you check the NotificationsService now you’ll see something similar to the following.
As you can see from the image, our system picked up right where it left off after RabbitMQ was down and then brought back up.
The OrderReceivedProcessor attempted to connect to RabbitMQ and successfully established the connection. It then processed the previously unprocessed order, identified by the ID 1 and the product "MacBook Pro 14".
The processor subsequently sent a notification email to the user, ensuring that the order processing and user notification resumed smoothly without any data loss. This demonstrates the resilience of our system in handling temporary outages.
Also, if we check the outbox endpoint again we’ll now see that our order has been processed.
This simple example shows one way we can ensure that our microservices remain resilient and reliable even in the face of potential system failures, such as the message queue being temporarily unavailable.
By using the Outbox pattern, we can guarantee that important messages are not lost and are processed as soon as the system is back online, thereby maintaining the integrity and continuity of our microservices architecture.