Published in Dev Genius

Oleg Potapov Follow

Mar 14 · 8 min read · Listen

Save

Microservices communication: Fetching data

from another service

The problem we’ll talk about is quite common, as microservices can’t be fully
independent. But very often one service needs some data from another one to be
able to invoke the business logic or to return this data to the client. In the classical
implementation of microservices architecture each service has its own database, so
to get this information there should be a method to connect.
First I will mention two solutions that are commonly used but their main purpose is
different.

1. UI composition
Open in app
UI composition is great mainly because it’s very simple for backend developers.
They have to do literally nothing to achieve the goal. It may work when there is a
need to combine data from several services and the logic behind it can be
performed on the client side. But, obviously, it’s not always possible and it’s hard to
imagine the application for which it may be enough to use only this way of fetching
data. Still, it may cover some use cases and is worth mentioning.

2. Aggregate service

Another way is to have a separate service that calls all the source services, fetching
data from each of them. Then it combines the received data into one data structure
and returns it to the client. This functionality is what the API Gateway service is
usually responsible for. This pattern is very common in microservices architecture
and works in most cases related to fetching data for serving GET requests. But it is
very limited when you have to apply business logic and, usually, it’s not a
functionality of the Gateway service, as all the domain-related logic is contained in
the dedicated microservice.

So, these two patterns mentioned above are well-known and widely used but they
are not a good fit for the specified problem. That’s why there are other ways.

Direct call to the other service

This is the most obvious way to solve the task. If you need some data from a
particular service, just ask for it! Most of the services have an API and this API can
be used to fetch data.
 25

Direct service calls

At first glance, it looks simple: the first service sends a request to the second. It may
be HTTP request, gRPC call, or something else. The second service fetches data
from the database, maybe handles it somehow and returns in the specified format.
However it only looks simple and sometimes it doesn’t work as planned.

The main thing you should occupy yourself with is coupling. Now two services are
coupled to each other not only by data; there is also a temporal coupling[1]. What’s
that? Temporal coupling occurs when one service has to wait for the response from
another and can’t continue processing without it. It means that when Service 2 is not
available, Service1 also can’t handle its requests, even though it’s alive and healthy.
And that means that some Service3 that calls Service1 can’t process its requests as
well. Such situations are called cascading failures. Such failures violate the whole
system’s work, so, obviously, we should avoid them. To do that developers and
architects invented a lot of cool patterns, such as Circuit Breaker[2], but it solves
only a part of the problem. Temporal coupling is still one of the most serious
problems with asynchronous communication and it’s almost impossible to get rid of
it altogether.

Another thing to consider is microservices API versioning. If you are not familiar
with it, welcome to a world full of dangers and surprises. There are a lot of
recommendations on how to do this versioning properly: Semantic Versioning[3],
URL versioning, using headers, etc. But it still may become a pain in the neck
anytime, regardless on which side you are — API creator or API consumer. As an API
creator, you don’t want to break all the clients with the next microservice deploy,
thus you should maintain not only the latest version of the API, but also the previous
one (or the previous 10?). When the service is developing and actively changing it
may be tricky to maintain its backward compatibility. As an API consumer, you
should be confident that nothing is broken after the API version is changed. How to
automate it? Here we move to the next question to think about.

Automated testing is what we can’t live without developing big programming

products. But what is the point of having services that are well-tested in isolation but
can’t communicate properly? And testing service communication is not that simple
and requires proper infrastructure. One of the possible solutions is Contract
Testing[4].

Having all that in mind we can make a list of advantages and disadvantages of this
approach.

Pros:

received data is always up-to-date

data-owning microservice can apply additional logic before sending the

response

data encapsulation is not violated

Cons:

temporal coupling

additional infrastructure work — circuit breakers, service discovery

need to deal with API versioning

may be hard to test

Embedded library
From the direct service calls diagram one could probably conclude that there is one
redundant network call. Why can’t the service receive the information directly from
the place it’s stored? The main obstacle to it is one of the backbone microservices
principles — don’t share a database across all the services. You will be on a safer side
if you use Database per service pattern[5] instead. According to this pattern, there
should be a dedicated database for each service with the exclusive rights to access.
But what’s wrong with the shared database? One of them is data encapsulation.
When there is a service specifically responsible for working with the database, it
encapsulates the internal data structure. But when there are several accessing
services, each of them should know this structure and how to query the database.
The solution is to create another layer of encapsulation. It may be a package, library
or SDK included into the service codebase.

Embedded library call

This package (or module, or whatever) may be developed and maintained by the
team responsible for data, but included into the other services as a dependency and
deployed with them. This module may contain read-only database access and some
tiny logic of fetching and composing the data.

Unfortunately, behind all this simplicity there are a lot of hidden problems and the
number of possible pitfalls may be even more than in the previous approach. Now
that you don’t have to worry about microservices versions, you’ve got a new problem
to deal with — a necessity to maintain library versions. In some cases it may be
simpler for the library developers, because not every service release will require the
library update. But sometimes maintaining several versions of the library can be the
same mess as maintaining several microservice API versions.

Another thing to keep an eye on is the way this dependency is managed. It should be
included into the service via package manager or another tool. It can become a
problem when two services are implemented using different programming
languages. In such case library developers will have to implement the client with
another language or use some kind of cross-language libraries.

There is one more issue related to encapsulation. Even though other services don’t
directly use data schema internals, the developers of the corresponding service still
should not relax. Making changes to the database schema they should make sure
that these changes won’t break the previous version of the library.

There may be a lot of other not-obvious and unexpected pitfalls, especially when the
product becomes huge enough. One such thing, for example, might be the number
of database connections. If each instance of the client library keeps one open
connection and each service contains several libraries, the number of connections
may get too large and even reach the limit.

And here goes a brief sum up.

Pros:

received data is always up-to-date

no additional infrastructure or circuit breakers required

client library may contain additional data-fetching logic

usually easier to start with

Cons:

possible schema-related issues

need to deal with library versioning

 dependency management tools required

security — all the client library copies use database connection credentials

Local data projection

You may also come across unusual cases of fetching data not assigned to your
service. It may be some kind of search or you may need to do table join of such data
with the data in your service. Although there may be a lot of ways to manipulate
data, usually the responsible service provides only one way to fetch the data and
only one format. Your service may not run with this format, fetching options, or
even the structure of the data. All these problems can be solved with building the
local data projection.

Local data projection

Data is saved in the Service2 database and then transferred to the Service1 database.
The copy is saved there in the same or changed format. Then Service 1 can use this
data, sending requests only to its local database, and if Service2 or its database is not
available it won’t violate Service2 in any way.
The first self-evident peril here is a propagation delay. There is no guarantee that the
data was not already changed at the moment of request. Another question is how to
copy changes from one database to another. It can be done with the help of database
tools (e.g. replication), message brokers, or Change-Data-Capture instruments (you
can read more about it in my previous article). In any way it will require additional
work on implementation and maintenance.

Here is the list of pros and cons of this approach.

Pros:

more flexibility with data format and data structure

no coupling with other services, the service depends only on data in its local
database

ability to work with data from the local database, e.g. making joins, map-reduce,
etc.

easier testing

Cons:

propagation delay, need to handle eventual consistency

additional work to implement and maintain reliable changes propagation

process

data duplication, additional disk space is used

Conclusion

So we have three options to choose from. Depending on the specific nature of the
project and the task to solve, all of them may be the best or the worst choice. As for
the third option (local projection) I wouldn’t use it normally. It may come in handy
in rare situations when you need to change data structure or make a join and thus
it’s worth mentioning.

Choosing between embedded library and direct calls, I would consider the size of
the project and the team working on it. When the team is not big enough and the
whole project is stored in the monorepo, an embedded library will probably be an
easier option to start with. But when the project contains thousands of
microservices implemented with different technologies, the drawbacks will
probably outweigh the advantages and it would be better to use direct calls.

Links

1. https://en.wikipedia.org/wiki/Coupling_(computer_programming)

2. https://en.wikipedia.org/wiki/Circuit_breaker_design_pattern

3. https://semver.org/

4. https://microservices.io/patterns/testing/service-integration-contract-test.html

5. https://microservices.io/patterns/data/database-per-service.html

6. https://medium.com/@oleg0potapov/events-patterns-message-relay-with-
change-data-capture-1da9b584758e

Microservices Microservice Architecture Versioning Change Data Capture

Shared Database

Sign up for DevGenius Updates

By Dev Genius

Get the latest news and update from DevGenius publication Take a look.

Emails will be sent to [email protected]. Not you?

Get this newsletter