Designing a Reliable Event Publishing Architecture with the Transactional Outbox Pattern

Transactional Outbox Pattern for Data Consistency in Event-Driven Systems

In event-driven systems, atomicity between database updates and message publication can break down, leading to data consistency issues. These inconsistencies can undermine service reliability, and the transactional outbox pattern is commonly used to address them. This pattern ensures consistency between the database and the message broker, preventing issues like data loss or duplicate message delivery.

1. Concept of the Transactional Outbox Pattern

The transactional outbox pattern is a technique that manages both data and event records within a single transaction. It guarantees that updates made to the database are reliably communicated through a message broker, thereby enhancing the trustworthiness of an event-driven architecture. With this pattern, services can remain decoupled while maintaining accurate data flow.

In microservices and event-driven systems, the following issues frequently arise:

• False notifications after payment failure: A failure during payment processing might still trigger a “payment successful” message, misleading the user into thinking the transaction was completed.
• Duplicate payments due to message delivery failure: Even when a payment succeeds, the corresponding message might not be published, causing the user to retry the payment due to missing confirmation.

Beyond these examples, a range of event-handling failures and consistency problems can occur. The transactional outbox pattern addresses such challenges by ensuring consistency between the database and the message broker. It reduces the risk of message delivery errors and contributes to system stability and reliability.

The transactional outbox pattern typically involves the following steps:

1. Transactionally store events: Event payloads are recorded in an outbox table within the same transaction as the primary database update.
2. Poll and publish events: Using a scheduler or Change Data Capture (CDC), the system regularly polls the outbox table for pending events and publishes them to a message broker (e.g., Kafka).
3. Update event status after delivery: Once the message is successfully delivered, the event status in the outbox table is updated to prevent duplicates.
4. Event consumption by downstream services: Subscribers process the event to trigger workflows such as payment processing, email delivery, or notifications.

The transactional outbox pattern is a robust solution for ensuring data consistency and system reliability in microservices environments. By relying on atomic transactions and retry mechanisms, it effectively mitigates common pitfalls in asynchronous communication.

2. Implementation of the Transactional Outbox Pattern

In this section, we'll walk through a simple scenario to implement the transactional outbox pattern step by step. The example focuses on sending a welcome email after a new user successfully completes registration.

1. Save user data: Upon receiving a signup request, the user's information is persisted in the database using a standard transaction.
2. Write outbox event: Within the same transaction, an outbox entry is inserted to represent the email dispatch event. This ensures atomicity and data consistency.
3. Poll pending events: A batch process or CDC (Change Data Capture) mechanism scans the outbox table for events marked as PENDING. CDC enables near real-time processing of database changes.
4. Publish message request: The scanned events are passed to a message broker producer (e.g., Kafka, RabbitMQ).
5. Mark event as completed: Once the message is successfully published, the event’s status in the outbox table is updated to COMPLETED, preventing duplicate dispatch.
6. Send to message broker: The producer sends the event to the message broker, making it available to subscribed systems.
7. Consume message: A downstream service subscribed to the topic receives the message and handles the corresponding operation.
8. Trigger email service: The consumer processes the message and calls the email service to send the welcome email.
9. Send confirmation email: The email is delivered to the user, completing the onboarding process.

2-1. Define Outbox Table Schema

The outbox table stores events alongside database changes, ensuring consistency between the database and the message broker.

create table outbox
(
    id             bigint                                not null primary key,
    aggregate_type varchar(255)                          not null,
    aggregate_id   bigint                                not null,
    event_type     varchar(255)                          not null,
    payload        json                                  not null,
    status         varchar(50) default 'PENDING'         not null,
    retry_count    int         default 0                 not null,
    created_at     datetime    default current_timestamp not null,
    updated_at     datetime    default current_timestamp not null on update current_timestamp
);

• id: Unique identifier for the event
• aggregate_type: The domain or entity type the event is associated with
• aggregate_id: Identifier of the associated entity
• event_type: Type of the event
• payload: JSON-formatted message body
• status: Processing status (PENDING, COMPLETED, etc.)
• retry_count: Number of delivery attempts
• created_at, updated_at: Timestamps for auditing and retry scheduling

2-2. Outbox Pattern in Action

To simplify the demonstration, we’ve excluded complex logic such as validation or encryption and focused solely on the outbox pattern. Here is the controller handling user registration:

@PostMapping("/signup")
public ResponseEntity<?> userSignupApi(
        @RequestBody UserSignupRequest request) {
    User user = userService.registerUser(request.username(), request.password());
    return ResponseEntity
            .ok(ApiResponse.success(user, "User registered successfully"));
}

A typical registration request looks like this:

POST /users/signup HTTP/1.1
Content-Type: application/json
Host: localhost:8080
Connection: close
User-Agent: RapidAPI/4.2.8 (Macintosh; OS X/15.3.0) GCDHTTPRequest
Content-Length: 53

{"username":"ongs@gmail.com","password":"trasactionaloutbox"}

The core business logic is executed in the service layer, where the user and outbox event are created and persisted within the same transaction:

@Transactional
public User registerUser(String username, String password) {
    try {
        User user = User.register(username, password);
        user = userRepository.save(user);

        Email email = Email.composeWelcomeEmail(user);
        OutboxEvent event = OutboxEvent.createUserRegisteredEvent(user, email);
        event = outboxEventRepository.save(event);

        log.info("registerUser: Successfully registered user - userId={}, username={}, eventId={}",
                user.getId(),
                user.getUsername(),
                event.getId());

        return user;
    } catch (Exception e) {
        throw new RuntimeException("Failed to create outbox event for user registration", e);
    }
}

If any part of the transaction fails, the operation is rolled back, preventing inconsistent state between the database and the outbox.

The payload field stores the full event message in JSON format:

{
  "email": {
    "id": 225941051972583586,
    "to": "ongs@gmail.com",
    "body": "Hello!\nThank you for registering with us. Enjoy your experience! 🎉\n\nBest,\nJin.\n",
    "userId": 225941051955806370,
    "subject": "[catsriding] Welcome to Our Service!"
  },
  "eventId": 225941051972649120
}

Once stored, the application returns a successful response, indicating the transaction is complete and the event is ready for publication.

A scheduler then periodically polls the outbox table and dispatches pending events:

@Scheduled(fixedRate = 5000)
public void dispatch() {
    List<OutboxEvent> events = outboxEventRepository.findAllByPendingStatus("PENDING", 3);
    for (OutboxEvent event : events) {
        outboxEventProcessor.publishEvent(event);
    }
}

Each event is processed individually to ensure safe and consistent message delivery:

@Transactional(propagation = Propagation.REQUIRES_NEW)
public void publishEvent(OutboxEvent event) {
    try {
        outboxEventRepository.save(event.start());
        String serializedPayload = objectMapper.writeValueAsString(event.getPayload());
        outboxEventPublisher.publish("user-registration-events", serializedPayload);

        log.info("process: Sent event: eventId={}, status={}", event.getId(), event.getStatus());
    } catch (Exception e) {
        throw new RuntimeException(e);
    }
}

Failed messages can be retried later by tracking status and retry count, further increasing reliability.

Once published, the message is consumed by a Kafka consumer and routed to the email service:

@KafkaListener(topics = "user-registration-events", groupId = "email-service")
public void consume(String message) {
    try {
        outboxEventConsumer.consume(message);
        log.info("consume: Successfully consumed user registration event");
    } catch (Exception e) {
        log.error("consume: Failed to consume message", e);
    }
}

The consumer deserializes the message and triggers the actual email delivery. While this step lies outside the outbox pattern’s scope, it demonstrates how event-driven services handle downstream responsibilities.

@Transactional
public void sendEmail(String message) {
    try {
        OutboxPayload payload = objectMapper.readValue(message, OutboxPayload.class);
        emailService.sendEmail(payload.email());

        OutboxEvent event = outboxEventRepository.findById(payload.eventId());
        event = outboxEventRepository.save(event.complete());

        log.info("consume: Successfully consumed outbox event - eventId={}, status={}",
                event.getId(),
                event.getStatus());
    } catch (JsonProcessingException e) {
        log.error("consume: Failed to consume outbox event - message={}", message);
    }
}

At this point, the full flow is complete. A new user signs up, triggering the creation of a PENDING outbox event:

id	event_type	payload	status	retry_count
225941051972649122	UserRegistered	{...}	PENDING	0

Once dispatched, the event status is updated to COMPLETED:

id	event_type	payload	status	retry_count
225941051972649122	UserRegistered	{...}	COMPLETED	0

Finally, the email service processes the message and delivers the welcome email.

This architecture improves scalability and maintainability while promoting loose coupling and reliability across services.

3. Conclusion

In this article, we explored how the Transactional Outbox Pattern can be used to address data consistency issues between a database and a message broker. Using a user registration scenario followed by welcome email delivery, we implemented a system that ensures event publication remains consistent within a single atomic transaction.

This pattern effectively prevents discrepancies between data changes and event publishing, enabling reliable and stable event-driven processing. However, the current implementation lacks robust handling of failure scenarios. To strengthen reliability, additional mechanisms—such as retry logic and failure compensation—should be incorporated.

Furthermore, in asynchronous workflows involving external systems (e.g., payment gateways), the transactional outbox alone may not be sufficient. In such cases, Saga Pattern is a powerful companion strategy that ensures consistency across distributed transactions. It coordinates a series of local transactions while still leveraging the outbox pattern to maintain consistency in event publication at each step.

In the next article, we will take a closer look at the Saga Pattern and how it helps maintain process-level consistency across distributed systems. 🎯