MSA Data mgmt patterns - Saga Pattern

Problem

When each service has its own database and a business transaction spans multiple services, how do we ensure data consistency across services? For example, for an e-commerce application where customers have a credit limit, the application must ensure that a new order will not exceed the customer’s credit limit. Since Orders and Customers are in different databases, the application cannot simply use a local ACID transaction.

Solution

The Saga pattern is the solution to implementing business transactions spanning multiple microservices.

A Saga is basically a sequence of local transactions. For every transaction performed within a Saga, the service performing the transaction publishes an event. The subsequent transaction is triggered based on the output of the previous transaction. And if one of the transactions in this chain fails, the Saga executes a series of compensating transactions to undo the impact of all the previous transactions. It can be implemented in two ways:

1. Choreography — When there is no central coordination, each service produces and listens to another service’s events and decides if an action should be taken or not.
2. Orchestration — An orchestrator (object) takes responsibility for a saga’s decision making and sequencing business logic.

The Saga pattern is one of the microservice design patterns that allow you to manage such transactions using a sequence of local transactions. Each transaction is followed by an event that triggers the next transaction step.

If one transaction fails, the saga pattern triggers a rollback transaction compensating for the failure.

Example: Managing multiple eCommerce transactions with a Saga pattern

Here’s an example of an eCommerce application that consists of multiple transactions for orders, payment, inventory, shipping, and notifications. Once an order is generated for a specific product, the next transaction for the payment and the inventory update is initialized.

If the transaction for inventory update fails, for example, due to unavailability of a product, a rollback is triggered. If the transaction for inventory update is successful further transactions are initialized.

Moreover, Saga transactions don’t need to know about the command or the role of other transactions. This allows developers to build simplified business logic with clear separation of concerns.

This pattern is suggested for applications where ensuring data consistency is critical without tight coupling. Likewise, it’s less suitable for applications with tight coupling.

MSA Data mgmt patterns - Shared Database per Service

Problem

We have talked about one database per service being ideal for microservices, but that is possible when the application is greenfield and to be developed with DDD. But if the application is a monolith and trying to break into microservices, denormalization is not that easy. What is the suitable architecture in that case?

Solution

A shared database per service is not ideal, but that is the working solution for the above scenario. Most people consider this an anti-pattern for microservices, but for brownfield applications, this is a good start to break the application into smaller logical pieces. This should not be applied for greenfield applications. In this pattern, one database can be aligned with more than one microservice, but it has to be restricted to 2-3 maximum, otherwise scaling, autonomy, and independence will be challenging to execute.

Shared Database per Service

Use a (single) database that is shared by multiple services. Each service freely accesses data owned by other services using local ACID transactions.

Example

The OrderService and CustomerService freely access each other’s tables. For example, the OrderService can use the following ACID transaction ensure that a new order will not violate the customer’s credit limit.

BEGIN TRANSACTION

…

SELECT ORDER_TOTAL

 FROM ORDERS WHERE CUSTOMER_ID = ?

…

SELECT CREDIT_LIMIT

FROM CUSTOMERS WHERE CUSTOMER_ID = ?

…

INSERT INTO ORDERS …

…

COMMIT TRANSACTION

The database will guarantee that the credit limit will not be exceeded even when simultaneous transactions attempt to create orders for the same customer.

The benefits of this pattern are:

·         A developer uses familiar and straightforward ACID transactions to enforce data consistency

·         A single database is simpler to operate.

The drawbacks of this pattern are:

· Development time coupling - a developer working on, for example, the OrderService will need to coordinate schema changes with the developers of other services that access the same tables. This coupling and additional coordination will slow down development.

·   Runtime coupling - because all services access the same database they can potentially interfere with one another. For example, if long running CustomerService transaction holds a lock on the ORDER table then the OrderService will be blocked.

·    Single database might not satisfy the data storage and access requirements of all services.

MSA Database Patterns - Database per Service

Problem

There is a problem of how to define database architecture for microservices. Following are the concerns to be addressed:

·         Services must be loosely coupled. They can be developed, deployed, and scaled independently.

·         Business transactions may enforce invariants that span multiple services.

·         Some business transactions need to query data that is owned by multiple services.

·         Databases must sometimes be replicated and shared to scale.

·         Different services have different data storage requirement

 Solution

To solve the above concerns, One database per microservice must be designed; it must be private to that service only. It should be accessed by the microservice API only. It cannot be accessed directly by other services.

There are a few different ways to keep a service’s persistent data private. You do not need to provision a database server for each service. For example, if you are using a relational database then the options are:

·         Private-tables-per-service – each service owns a set of tables that must only be accessed by that service

·         Schema-per-service – each service has a database schema that’s private to that service

·         Database-server-per-service – each service has its own database server.

Private-tables-per-service and schema-per-service have the lowest overhead. Using a schema per service is appealing since it makes ownership clearer. Some high throughput services might need their own database server.

Using a database per service has the following benefits:

·         Helps ensure that the services are loosely coupled. Changes to one service’s database does not impact any other services.

·         Each service can use the type of database that is best suited to its needs. For example, a service that does text searches could use ElasticSearch. A service that manipulates a social graph could use Neo4j.

Using a database per service has the following drawbacks:

·         Implementing business transactions that span multiple services is not straightforward. Distributed transactions are best avoided because of the CAP theorem. Moreover, many modern (NoSQL) databases don’t support them.

·         Implementing queries that join data that is now in multiple databases is challenging.

·         Complexity of managing multiple SQL and NoSQL databases.

There are various patterns/solutions for implementing transactions and queries that span services:

·         Implementing transactions that span services - use the Saga pattern.

·         Implementing queries that span services:

o    API Composition - the application performs the join rather than the database. For example, a service (or the API gateway) could retrieve a customer and their orders by first retrieving the customer from the customer service and then querying the order service to return the customer’s most recent orders.

o    Command Query Responsibility Segregation (CQRS) - maintain one or more materialized views that contain data from multiple services. The views are kept by services that subscribe to events that each services publishes when it updates its data. For example, the online store could implement a query that finds customers in a particular region and their recent orders by maintaining a view that joins customers and orders. The view is updated by a service that subscribes to customer and order events.

It is a good idea to create barriers that enforce this modularity. You could, for example, assign a different database user id to each service and use a database access control mechanism such as grants. Without some kind of barrier to enforce encapsulation, developers will always be tempted to bypass a service’s API and access it’s data directly.