Modernization Data Persistence
Modernization Data Persistence
Amazon's trademarks and trade dress may not be used in connection with any product or service
that is not Amazon's, in any manner that is likely to cause confusion among customers, or in any
manner that disparages or discredits Amazon. All other trademarks not owned by Amazon are
the property of their respective owners, who may or may not be affiliated with, connected to, or
sponsored by Amazon.
AWS Prescriptive Guidance Enabling data persistence in microservices
Table of Contents
Introduction ..................................................................................................................................... 1
Targeted business outcomes ...................................................................................................................... 3
Patterns for enabling data persistence .......................................................................................... 4
Database-per-service pattern ..................................................................................................................... 4
API composition pattern ............................................................................................................................. 6
CQRS pattern ................................................................................................................................................. 8
Event sourcing pattern ............................................................................................................................. 11
Amazon Kinesis Data Streams implementation ............................................................................. 12
Amazon EventBridge implementation .............................................................................................. 13
Saga pattern ................................................................................................................................................ 14
Shared-database-per-service pattern .................................................................................................... 16
FAQ ................................................................................................................................................. 18
When can I modernize my monolithic database as part of my modernization journey? ............. 18
Can I keep a legacy monolithic database for multiple microservices? ............................................ 18
What should I consider when designing databases for a microservices architecture? ................. 18
What is a common pattern for maintaining data consistency across different
microservices? ............................................................................................................................................. 18
How do I maintain transaction automation? ....................................................................................... 19
Do I have to use a separate database for each microservice? .......................................................... 19
How can I keep a microservice’s persistent data private if they all share a single database? ...... 19
Resources ........................................................................................................................................ 20
Related guides and patterns .................................................................................................................. 20
Other resources ......................................................................................................................................... 20
Document history .......................................................................................................................... 21
Glossary .......................................................................................................................................... 22
Modernization terms ................................................................................................................................. 22
iii
AWS Prescriptive Guidance Enabling data persistence in microservices
Organizations constantly seek new processes to create growth opportunities and reduce time
to market. You can increase your organization's agility and efficiency by modernizing your
applications, software, and IT systems. Modernization also helps you deliver faster and better
services to your customers.
• Decompose monoliths into microservices – Use patterns to break down monolithic applications
into microservices.
• Integrate microservices – Integrate the newly created microservices into a microservices
architecture by using Amazon Web Services (AWS) serverless services.
• Enable data persistence for microservices architecture – Promote polyglot persistence among
your microservices by decentralizing their data stores.
Although you can use a monolithic application architecture for some use cases, modern application
features often don't work in a monolithic architecture. For example, the entire application
can't remain available while you upgrade individual components, and you can't scale individual
components to resolve bottlenecks or hotspots (relatively dense regions in your application's data).
Monoliths can become large, unmanageable applications, and significant effort and coordination is
required among multiple teams to introduce small changes.
Legacy applications typically use a centralized monolithic database, which makes schema changes
difficult, creates a technology lock-in with vertical scaling as the only way to respond to growth,
and imposes a single point of failure. A monolithic database also prevents you from building
the decentralized and independent components required for implementing a microservices
architecture.
Previously, a typical architectural approach was to model all user requirements in one relational
database that was used by the monolithic application. This approach was supported by popular
1
AWS Prescriptive Guidance Enabling data persistence in microservices
relational database architecture, and application architects usually designed the relational schema
at the earliest stages of the development process, built a highly normalized schema, and then sent
it to the developer team. However, this meant that the database drove the data model for the
application use case, instead of the other way round.
By choosing to decentralize your data stores, you promote polyglot persistence among your
microservices, and identify your data storage technology based on the data access patterns
and other requirements of your microservices. Each microservice has its own data store and
can be independently scaled with low-impact schema changes, and data is gated through the
microservice’s API. Breaking down a monolithic database is not easy, and one of the biggest
challenges is structuring your data to achieve the best possible performance. Decentralized
polyglot persistence also typically results in eventual data consistency, and other potential
challenges that require a thorough evaluation include data synchronization during transactions,
transactional integrity, data duplication, and joins and latency.
This guide is for application owners, business owners, architects, technical leads, and project
managers. The guide provides the following six patterns to enable data persistence among your
microservices:
• Database-per-service pattern
• API composition pattern
• CQRS pattern
• Event sourcing pattern
• Saga pattern
• For steps to implement the saga pattern by using AWS Step Functions, see the pattern
Implement the serverless saga pattern by using AWS Step Functions on the AWS Prescriptive
Guidance website.
• Shared-database-per-service pattern
The guide is part of a content series that covers the application modernization approach
recommended by AWS. The series also includes:
2
AWS Prescriptive Guidance Enabling data persistence in microservices
You should expect the following six outcomes from promoting data persistence among your
microservices:
Topics
• Database-per-service pattern
• CQRS pattern
• Saga pattern
• Shared-database-per-service pattern
Database-per-service pattern
Loose coupling is the core characteristic of a microservices architecture, because each individual
microservice can independently store and retrieve information from its own data store. By
deploying the database-per-service pattern, you choose the most appropriate data stores (for
example, relational or non-relational databases) for your application and business requirements.
This means that microservices don't share a data layer, changes to a microservice's individual
database do not impact other microservices, individual data stores cannot be directly accessed
by other microservices, and persistent data is accessed only by APIs. Decoupling data stores also
improves the resiliency of your overall application, and ensures that a single database can't be a
single point of failure.
In the following illustration, different AWS databases are used by the “Sales,” “Customer,” and
“Compliance” microservices. These microservices are deployed as AWS Lambda functions and
accessed through an Amazon API Gateway API. AWS Identity and Access Management (IAM)
policies ensure that data is kept private and not shared among the microservices. Each microservice
uses a database type that meets its individual requirements; for example, "Sales" uses Amazon
Aurora, "Customer" uses Amazon DynamoDB, and "Compliance" uses Amazon Relational Database
Service (Amazon RDS) for SQL Server.
Database-per-service pattern 4
AWS Prescriptive Guidance Enabling data persistence in microservices
• It might be challenging to implement complex transactions and queries that span multiple
microservices or data stores.
• Your data stores must meet two of the CAP theorem requirements: consistency, availability, or
partition tolerance.
Database-per-service pattern 5
AWS Prescriptive Guidance Enabling data persistence in microservices
Note
If you use the database-per-service pattern, you must deploy the API composition pattern
or the CQRS pattern to implement queries that span multiple microservices.
1. An API gateway serves the "/customer" API, which has an "Orders" microservice that tracks
customer orders in an Aurora database.
2. The "Support" microservice tracks customer support issues and stores them in an Amazon
OpenSearch Service database.
3. The "CustomerDetails" microservice maintains customer attributes (for example, address, phone
number, or payment details) in a DynamoDB table.
4. The “GetCustomer” Lambda function runs the APIs for these microservices, and performs an in-
memory join on the data before returning it to the requester. This helps easily retrieve customer
information in one network call to the user-facing API, and keeps the interface very simple.
The API composition pattern offers the simplest way to gather data from multiple microservices.
However, there are the following disadvantages to using the API composition pattern:
• It might not be suitable for complex queries and large datasets that require in-memory joins.
• Your overall system becomes less available if you increase the number of microservices
connected to the API composer.
• Increased database requests create more network traffic, which increases your operational costs.
CQRS pattern
The command query responsibility segregation (CQRS) pattern separates the data mutation, or the
command part of a system, from the query part. You can use the CQRS pattern to separate updates
and queries if they have different requirements for throughput, latency, or consistency. The CQRS
pattern splits the application into two parts—the command side and the query side—as shown in
the following diagram. The command side handles create, update, and delete requests. The
query side runs the query part by using the read replicas.
CQRS pattern 8
AWS Prescriptive Guidance Enabling data persistence in microservices
1. The business interacts with the application by sending commands through an API. Commands
are actions such as creating, updating or deleting data.
2. The application processes the incoming command on the command side. This involves
validating, authorizing, and running the operation.
3. The application persists the command’s data in the write (command) database.
4. After the command is stored in the write database, events are triggered to update the data in
the read (query) database.
5. The read (query) database processes and persists the data. Read databases are designed to be
optimized for specific query requirements.
6. The business interacts with read APIs to send queries to the query side of the application.
7. The application processes the incoming query on the query side and retrieves the data from the
read database.
You can implement the CQRS pattern by using various combinations of databases, including:
• Using relational database management system (RDBMS) databases for both the command and
the query side. Write operations go to the primary database and read operations can be routed
to read replicas. Example: Amazon RDS read replicas
CQRS pattern 9
AWS Prescriptive Guidance Enabling data persistence in microservices
• Using an RDBMS database for the command side and a NoSQL database for the query side.
Example: Modernize legacy databases using event sourcing and CQRS with AWS DMS
• Using NoSQL databases for both the command and the query side. Example: Build a CQRS event
store with Amazon DynamoDB
• Using a NoSQL database for the command side and an RDBMS database for the query side, as
discussed in the following example.
In the following illustration, a NoSQL data store, such as DynamoDB, is used to optimize the
write throughput and provide flexible query capabilities. This achieves high write scalability on
workloads that have well-defined access patterns when you add data. A relational database, such
as Amazon Aurora, provides complex query functionality. A DynamoDB stream sends data to a
Lambda function that updates the Aurora table.
Implementing the CQRS pattern with DynamoDB and Aurora provides these key benefits:
• DynamoDB is a fully managed NoSQL database that can handle high-volume write operations,
and Aurora offers high read scalability for complex queries on the query side.
CQRS pattern 10
AWS Prescriptive Guidance Enabling data persistence in microservices
• DynamoDB provides low-latency, high-throughput access to data, which makes it ideal for
handling command and update operations, and Aurora performance can be fine-tuned and
optimized for complex queries.
• Both DynamoDB and Aurora offer serverless options, which enables your business to pay for
resources based on usage only.
• DynamoDB and Aurora are fully managed services, which reduces the operational burden of
managing databases, backups and scalability.
• You implemented the database-per-service pattern and want to join data from multiple
microservices.
• Your read and write workloads have separate requirements for scaling, latency, and consistency.
• Eventual consistency is acceptable for the read queries.
Important
The CQRS pattern typically results in eventual consistency between the data stores.
By choosing this pattern, you can identify and reconstruct the application’s state for any point in
time. This produces a persistent audit trail and makes debugging easier. However, data becomes
eventually consistent and this might not be appropriate for some use cases.
This pattern can be implemented by using either Amazon Kinesis Data Streams or Amazon
EventBridge.
In the following illustration, Kinesis Data Streams is the main component of a centralized event
store. The event store captures application changes as events and persists them on Amazon Simple
Storage Service (Amazon S3).
1. When the "/withdraw" or "/credit" microservices experience an event state change, they publish
an event by writing a message into Kinesis Data Streams.
2. Other microservices, such as "/balance" or "/creditLimit," read a copy of the message, filter it for
relevance, and forward it for further processing.
1. "OrderPlaced" events are published by the "Orders" microservice to the custom event bus.
2. Microservices that need to take action after an order is placed, such as the "/route" microservice,
are initiated by rules and targets.
3. These microservices generate a route to ship the order to the customer and emit a
"RouteCreated" event.
4. Microservices that need to take further action are also initiated by the "RouteCreated" event.
5. Events are sent to an event archive (for example, EventBridge archive) so that they can be
replayed for reprocessing, if required.
6. Historical order events are sent to a new Amazon SQS queue (replay queue) for reprocessing, if
required.
7. If targets are not initiated, the affected events are placed in a dead letter queue (DLQ) for
further analysis and reprocessing.
Important
If you use the event sourcing pattern, you must deploy the Saga pattern to maintain data
consistency across microservices.
Saga pattern
The saga pattern is a failure management pattern that helps establish consistency in distributed
applications, and coordinates transactions between multiple microservices to maintain data
consistency. A microservice publishes an event for every transaction, and the next transaction is
initiated based on the event's outcome. It can take two different paths, depending on the success
or failure of the transactions.
The following illustration shows how the saga pattern implements an order processing system
by using AWS Step Functions. Each step (for example, “ProcessPayment”) also has separate
Saga pattern 14
AWS Prescriptive Guidance Enabling data persistence in microservices
steps to handle the success (for example, "UpdateCustomerAccount") or failure (for example,
"SetOrderFailure") of the process.
• The application needs to maintain data consistency across multiple microservices without tight
coupling.
• There are long-lived transactions and you don’t want other microservices to be blocked if one
microservice runs for a long time.
Saga pattern 15
AWS Prescriptive Guidance Enabling data persistence in microservices
Important
The saga pattern is difficult to debug and its complexity increases with the number of
microservices. The pattern requires a complex programming model that develops and
designs compensating transactions for rolling back and undoing changes.
For more information about implementing the saga pattern in a microservices architecture, see
the pattern Implement the serverless saga pattern by using AWS Step Functions on the AWS
Prescriptive Guidance website.
Shared-database-per-service pattern
In the shared-database-per-service pattern, the same database is shared by several microservices.
You need to carefully assess the application architecture before adopting this pattern, and make
sure that you avoid hot tables (single tables that are shared among multiple microservices). All
your database changes must also be backward-compatible; for example, developers can drop
columns or tables only if objects are not referenced by the current and previous versions of all
microservices.
In the following illustration, an insurance database is shared by all the microservices and an IAM
policy provides access to the database. This creates development time coupling; for example,
a change in the "Sales" microservice needs to coordinate schema changes with the "Customer"
microservice. This pattern does not reduce dependencies between development teams, and
introduces runtime coupling because all microservices share the same database. For example,
long-running "Sales" transactions can lock the "Customer" table and this blocks the "Customer"
transactions.
Shared-database-per-service pattern 16
AWS Prescriptive Guidance Enabling data persistence in microservices
• You don't want too much refactoring of your existing code base.
• You enforce data consistency by using transactions that provide atomicity, consistency, isolation,
and durability (ACID).
• You want to maintain and operate only one database.
• Implementing the database-per-service pattern is difficult because of interdependencies among
your existing microservices.
• You don’t want to completely redesign your existing data layer.
Shared-database-per-service pattern 17
AWS Prescriptive Guidance Enabling data persistence in microservices
FAQ
This section provides answers to commonly raised questions about enabling data persistence in
microservices.
Resources
Other resources
• Application modernization with AWS
• Build highly available microservices to power applications of any size and scale
• Cloud-native application modernization with AWS
• Cost optimization and innovation: An introduction to application modernization
• Developer guide: Scale with microservices
• Distributed data management – Saga Pattern
• Implementing microservice architectures using AWS services: Command query responsibility
segregation pattern
• Implementing microservice architectures using AWS services: Event sourcing pattern
• Modern applications: Creating value through application design
• Modernize your applications, drive growth and reduce TCO
Document history
The following table describes significant changes to this guide. If you want to be notified about
future updates, you can subscribe to an RSS feed.
Added a link for implement We updated the Home February 23, 2021
ing the saga pattern with and Saga pattern sections
Step Functions with the link to the pattern
Implement the serverless
saga pattern by using AWS
Step Functions from the AWS
Prescriptive Guidance website.
21
AWS Prescriptive Guidance Enabling data persistence in microservices
Modernization terms
anti-pattern
A frequently used solution for a recurring issue where the solution is counter-productive,
ineffective, or less effective than an alternative.
business capability
What a business does to generate value (for example, sales, customer service, or marketing).
Microservices architectures and development decisions can be driven by business capabilities.
For more information, see the Organized around business capabilities section of the Running
containerized microservices on AWS whitepaper.
domain-driven design
An approach used to modernize and upgrade operational technology (OT) systems to better
serve the needs of the manufacturing industry. A historian is a type of database that is used to
collect and store data from various sources in a factory.
microservice
A small, independent service that communicates over well-defined APIs and is typically
owned by small, self-contained teams. For example, an insurance system might include
microservices that map to business capabilities, such as sales or marketing, or subdomains,
such as purchasing, claims, or analytics. The benefits of microservices include agility, flexible
Modernization terms 22
AWS Prescriptive Guidance Enabling data persistence in microservices
scaling, easy deployment, reusable code, and resilience. For more information, see Integrating
microservices by using AWS serverless services.
microservices architecture
An approach to building an application with independent components that run each application
process as a microservice. These microservices communicate through a well-defined interface
by using lightweight APIs. Each microservice in this architecture can be updated, deployed,
and scaled to meet demand for specific functions of an application. For more information, see
Implementing microservices on AWS.
modernization
Transforming an outdated (legacy or monolithic) application and its infrastructure into an agile,
elastic, and highly available system in the cloud to reduce costs, gain efficiencies, and take
advantage of innovations. For more information, see Strategy for modernizing applications in
the AWS Cloud.
modernization readiness assessment
Applications that run as a single service with tightly coupled processes. Monolithic applications
have several drawbacks. If one application feature experiences a spike in demand, the
entire architecture must be scaled. Adding or improving a monolithic application’s features
also becomes more complex when the code base grows. To address these issues, you can
use a microservices architecture. For more information, see Decomposing monoliths into
microservices.
polyglot persistence
Independently choosing a microservice’s data storage technology based on data access patterns
and other requirements. If your microservices have the same data storage technology, they can
encounter implementation challenges or experience poor performance. Microservices are more
easily implemented and achieve better performance and scalability if they use the data store
Modernization terms 23
AWS Prescriptive Guidance Enabling data persistence in microservices
best adapted to their requirements. For more information, see Enabling data persistence in
microservices.
split-and-seed model
A pattern for scaling and accelerating modernization projects. As new features and product
releases are defined, the core team splits up to create new product teams. This helps scale your
organization’s capabilities and services, improves developer productivity, and supports rapid
innovation. For more information, see Phased approach to modernizing applications in the AWS
Cloud.
strangler fig pattern
A small DevOps team that you can feed with two pizzas. A two-pizza team size ensures the best
possible opportunity for collaboration in software development. For more information, see the
Two-pizza team section of the Introduction to DevOps on AWS whitepaper.
Modernization terms 24