WHITE PAPER

RESILIENCE IN DISTRIBUTED
SYSTEMS

Abstract
With the rapid increase in the number of internet users and frequent
changes in consumer trends, traditional systems have no choice but
to scale out, distribute, and decentralize. To give you an idea of the
extend of scaling involved, Facebook and YouTube each would have
had 40,000 to 45,000 hits from desktop users alone in the 5 seconds
it took you to read this paragraph. [5].
With the peta and exa bytes of data being generated every day and
the growing adoption of IoT, this scaling is only going to become
more exponential and systems will need to scale out and depend on
each other more than ever.
Distributed systems are driven by various Architecturally Significant
Requirements (ASRs) [24] and one such ASR is resilience.
This paper is about the SPEAR (Software Performance Engineering
and ARchitecture services) team’s perspective on Application
Resilience in distributed systems – what it means in simple terms,
how to study it, the factors affecting it, and what patterns/best
practices can help us in improving the same.
 Who is this document for? design of a persistent database will be to effectively, with minimal disruption to
effectively manage secondary disk. business operations.
This is presented from the perspective of a
lead application designer or an architect, What is resilience? Why is this important? – Because it affects
but there are some theories and methods business, trust and lives.
Resilience of an application, in simple
for IT managers, developers, architects, A downtime of 1 minute at Amazon can
language, is the capability of the
tech leads, and program managers who result in a potential loss of USD 234,000
application to spring back to an acceptable
are looking to understand and improve through their online channels alone. [6][7]
operational condition after it faces an
resilience in a distributed system. [8] A technical glitch caused an outage on
event affecting its operating conditions.
Some basic definitions first. the New York Stock Exchange, leading to a
[ ‘capability’ - what you have inside your drop in share prices and indexes.
What is a distributed system? systems to put them back in acceptable
Can you imagine an outage in a critical
operating condition),
A distributed system is one where multiple hospital system, air traffic control system,
components of a system are physically ‘event’ – failure of responsibility of some core trading platform or on a police or
or logically separated and governed module within the application or a failure emergency contact system?
by common and component-specific of responsibility of some dependent
system or a force majeure situation] Some metrics like RPO (Recovery Point
requirements.
Objective), RTO (Recovery Time Objective),
We say common because if you are A ‘failure of responsibility’ or simply MTTR (Mean Time to Recovery), Number of
designing a system to be 99.99% a failure could be a breach of SLA or failures/bugs detected in the system, SLA
available you cannot have a crucial some sort of agreement regarding an variance, etc., are some ways to measure
cache component in the system which application. It could be big, like a failure resilience of a system. We will not be
is available only 97%. If that is the case, of Amazon Route 53 services or a bug in going into detail in this paper about the
you need to have a trade-off in place to the implementation of the Set interface measurement and tools used but take a
make sure the service availability is met in which behaves unexpectedly under normal look at the various failures and patterns
spite of only 97% availability of the cache operating conditions. which can be used to improve the MTTR or
component. The flipside of resilience is all about RPO or RTO of a system.

We say component specific because the understanding and preparing for failures. Let’s begin with a couple of modeling
design of the cache component is driven So resilience can also be defined as the techniques needed for studying resilience
by speed and will be leveraging much capability of the system to understand and – Flow and failure modeling.
of the primary storage disk, whereas the manage failures occurring in the system

Flow modeling lets the user select the food (A), check out study we need to look at the alternate
the same (B), check coupons (C) via an flows. Alternate flows are the control
Traditionally we study the various flows in external service, recalculate the checkout and data flows which are taken by the
the system via use case analysis, control/ amount (B) if there are any discounts, select application if an unexpected behavior
data flows, sequence diagrams etc., but will an address (D), external service for making occurs. So to understand and design
this linear and branching flow study really payment (E1) which in turn automatically alternate flows we need to include the list
help? places an order (E2) via the restaurant API. of implicit services and dependencies at
Let’s consider the control flow of this each step.
While this is more of a happy flow or ideal
example: an online food ordering website flow of a business operation, for resilience

Figure 1 Control Flow of Order Booking

External Document © 2019 Infosys Limited

For example, let’s look at the context takes the form of a reticulated mesh, Figure (2) can help in understanding the
of Step B – the check-out service. If we which requires us to study the various various stages of failure handling in a
examine carefully, the check-out service, events which can occur in each of these system.
apart from coupon and address service, services, and then design and architect the
FP – Fault Prevention - Capabilities present
inherently banks on: application to handle the same.
in the system to handle known failures that
1. The DNS services. With the current linear and branching are expected to occur in the system.
model this is difficult and we should start
2. The infrastructure services like FD – Fault Detection – Capabilities present
using a multi-dimensional graphical model
in the system to detect a fault. tfd - time gap
a. Operating System when trying to understand the control
between the actual fault occurrence and
b. CPU and data flows. This perspective is very
the detection of the same.
important to design resilient systems.
c. RAM FI – Fault Isolation – Capability of the
Where to start – Cyclic and acyclic graphs,
d. Storage – Disk, File system to isolate the fault and treat it
predictive models like PGM (Probabilistic
separately so that the normal functioning
e. Network Graph Modeling) etc., are some of the
of the system is not affected. If there is no
places to start. [21]
f. Any other infra components like fault isolation mechanism, then the time
routers, switches, firewalls, etc., Failure modeling taken to treat the fault would affect the
normal processing of the system. tfi – It is
3. The trust domain established by Once we have a fair understanding of the the time taken by the system to isolate the
the security context (authentication, multi-dimensional flow of control and data fault after it has been detected.
authorization, tokens etc.,) in the application, we need to perform
a failure modeling exercise – where we FT – Fault Treatment – Capability present
4. Persistent Data Storage Services.
list down the kinds of known failures and in the system to treat a fault once it has
5. The state of the service (is it configured perform a what-if analysis and incorporate been detected and/or isolated. tft – It is the
correctly and running correctly). failure handling mechanisms in the system. time taken to categorize the type of fault
and treat it accordingly. For example, faults
6. Technology of the service (Language, [ ‘what-if’ – a what-if analysis is essentially like a dependent system not reachable
frameworks, dependent libraries, etc.) an exercise to simulate a failure from via network can be retried, but a fault
7. Etc . the known list and check if there are like ‘order message cannot be parsed’ is
mechanisms present in the system to something which will fail no matter how
If we observe all the services some are
handle it. many times it is retried. Such faults cannot
shared between multiple components and
For example, what if there is a network be treated and will be logged and failed
some are not. Some are external and some
failure? Do we have an alternate network gracefully.
are internal. Some are in the same layer as
the application and some are not. Studying or retry of services at regular intervals
these dependencies and relations quickly implemented?]

Figure 2 Failure Handling in a software system

External Document © 2019 Infosys Limited

 FR – Fault Recovery – Capability present in Key failure categories library which is built based on Actor model.
the system to recover from the fault after
Let us see some failure categories and Architecture and design is a vast topic and
it has occurred and/or has been treated.
check what mechanisms can be put in to retain brevity, we recommend the below
If the fault cannot be treated or if the
place to handle them efficiently. material for further reading on this.
treatment of the fault affects the overall
system state, then recovery mechanisms I. Architecture and design Issues: Where to start? - Martin Fowler, Chaos
are needed to set the system back to engineering groups, EIP patterns by Gregor
A poorly designed or architected Hohpe and Bobby Woolf, highscalability.
operational state. The only goal of the
distributed software can lead to various com, cloud resiliency patterns, Erlang’s
recovery mechanism(s) is to get the system
issues in the system today, tomorrow or ‘Let it Fail’ [17], digital twins, bulkhead, 8
back up and running in normal operating
any time till the end of life of the software. fallacies of distributed computing, stand-in
mode.
Remember, “Prevention is always better processing, claim-check, throttling, sidecar,
In one of the architecture assessments we than cure”. circuit breaker, fencing, redundancy,
found that the key orchestrating system
Couple of examples are mentioned below: auto scaling, stateless architectures,
was running on a fault-tolerant box, but
Infrastructure as Code, reconciliation
when we did a what-if analysis on the 1. Intelligent retries – “Intelligent Retries”
strategies, Policy centralization, Immutable
physical failure of the system, we found decide the strategy to retry a failed
infrastructure, event/service meshes and
that the time taken to achieve the RTO operation. For example, retry with a
mesh based architectures (like MASA),
and RPO was getting affected. As a result, timeout, logarithmic retry, arithmetic retry,
traffic mirroring, Unbreakable pipelines,
we recommended adding an additional geometric retries, priority retries, failover
streaming patterns, Byzantine fault,
machine to handle box failure. retries, etc.
consensus algorithms like raft or Paxos, CEP
Where to start - FMEA (Failure Mode Effects Retries can make an application fault- (Complex Event Processing), real-time and
Analysis), Event and Fault tree analysis, etc., tolerant, that is, if a module fails to connect near real-time replication strategies, EDA
are some places to start. to a service, it silently fails-over to another (Event Driven Architecture), choreography
equivalent service which can fulfil the same patterns, distributed transaction handling
A highly resilient system should
request. This is called “idempotent failover” patterns (like SAGA), data bus concepts
scientifically study each fault, categorize
[4] and is employed in many stateless from LinkedIn or MySQL or MongoDB, etc .
them based on severity and risk, have
services.
proper mechanisms in place to handle the Daunted by this list? Don’t be, and you are
system in case of a failure, and decrease the 2. Actor model [14] – Actor model is an not alone. Understanding the requirements
time taken to detect or treat a fault. alternate, highly concurrent and resilient of the distributed software and making
model where actors never lock on a single intelligent choice of algorithms and
resource like shared memory or shared patterns will solve many forthcoming
object, instead talking to each other via issues.
messages. Akka is one such open source

External Document © 2019 Infosys Limited

II. Failure risk analysis: example, a network failure is anticipated study the failure and develop a solution for
and recovery mechanisms are in place to the same.
To build more confidence in the system it handle the same.
is important that a failure categorization Quadrant 3 – In this case it is not only
is done accordingly and risk is assessed. Quadrant 2 – It is possible that some of the difficult to find the unknown failures in the
The following quadrant-based segregation failures/bugs lie unearthed in the system system, as setting up such testing context
helps in understanding a distributed and if found earlier in the system could can be difficult, it is possible that such
system. have been isolated and handled better by failures have no known solutions that can
adding the required design and processes. be applied.
Quadrant 1 - All failures, once they occur An unknown failure once unearthed
or are known failure scenarios, are studied Quadrant 4 - This case is when we know
becomes a known failure and after
and solutions are provided to handle them some failures may occur but truly there
studying it, is moved into Quadrant 1.
through various architectural choices and is no solution which can be applied.
design mechanisms. The same needs to For example, an unearthed design bug of Force majeure situations like acts of god,
be built into various testing strategies infinite retry occurs only when the network or any man-made disaster fall into this
to ensure that any changes made in the goes down. Such scenarios could be category and such risks are to be covered
system are tested against these known missed in the usual testing methods. Using in contracts
failures and are handled accordingly. For failure injection techniques, it is possible to

Figure 3 Known and Unknown Failure domains

III. Testing methods: testing are – Chaos Monkey, Simian Army, systems and tools today also allow us to
gremlin etc. detect and rollback the deployments made
The application landscape has changed in case of an error found in production.
and the world is getting rapidly rewritten Another way is to employ shift left testing
in code for tomorrow, even as we speak. strategy where the testing methods can The testing tools and methods today are
start engaging very early in the cycle sophisticated and continually evolving.
To keep up with this change, it is and act as a gateway to promote the Embracing these new testing methods,
imperative that software testing methods code. Tool chains in the DevOps, CI/CD tools, and processes is imperative for
have to be made more robust. One such pipeline should integrate testing tools and building a robust system.
method is Chaos testing [15] which is the practically all kinds of testing, from unit,
process of testing failures in a distributed performance, security, integration etc., can IV. Deployment issues:
system by injecting known failures in the be executed and studied in a controlled An intelligent deployment strategy can
systems and observing the behavior. environment and it can be decided if the prevent issues from being caused due to
For example, inject a JVM memory software or the patch can go live. Tools software problems. Let’s look at a couple of
exception in a remote JVM instance and like kubernetes, JMeter, Jenkins, Docker, examples below:
observing the response time of the system. Dynatrace etc., enable us to model newer
1. Blue/Green deployment – The idea is
testing approaches. Robust monitoring
Some tools which can be used for chaos to maintain two production environments

External Document © 2019 Infosys Limited

 - blue and green - which are identical, sees beyond programming a specific task of DevOps and usage of DevOps tools.
with one of them being active (say blue). and puts together the multiple facets of Incorporate it as part of the software
A new feature is deployed into the inactive what he writes. development-to-deployment cycle.
environment (green) and tested and once
To signify the importance of this, similar to VIII. Security issues:
it is ‘Okay’, then the traffic is switched from
the Y2K bug, a software bug which couldn’t
blue to green. The next feature is deployed Security is a significant area where failures
detect the year 2010 rendered up to 30
into blue and traffic is routed through blue occur in the system. Cyber security has
million debit cards in Germany unusable.
again. This ensures that at any point of always been a battle of today with the
[1][2]
time, the system is operable and no new vulnerabilities of yesterday.
features rupture the business operations A well aligned developer will be able to
Unfortunately, hackers today are more
after go live. write software which will not have such
targeted and motivated than ever before.
issues.
2. Canary releases – This strategy is similar Example: In January 2018, three major
to blue/green deployment but a release is VI. Architecturally Significant banks in Netherlands were victims of
only made to a subset of the infrastructure Requirements study a DDoS attack which resulted in slow
or a scalable but downsized identical It is important that the architecturally response times and service outages. [9]
deployment. Initially some % of the users significant requirements (ASR) derived
are routed to the new deployment. Once Following are some of the preventive
during the initial requirement or
the new feature is stable, all users will be mechanisms that can be incorporated from
‘solutioning’ phase are studied carefully
routed to the new version. an application and process perspective to
and related to logic and math. If there
avoid such a scenario–
For example, we all do see pop-ups in are specific SLAs and metrics which the
or mobile apps and web apps asking us software needs to meet, capturing them a. In case of a security threat, isolation
if we want to try the new feature or the beforehand and asking the software of application or services from the
new beta version of a software. When builders to ‘mind them’ while developing network should be possible.
we opt ‘Yes’, we will be routed to the new will help. b. Security checks, OWASP (Open
feature while users who opted ‘No’ will be For example, instead of just stating the Web Application Security Project)
routed to the old version. Practices like system should be up 99.99%, it should compliance checks, app scans, and
this ensures that feature roll outs don’t be written in a more clearly articulated customized security testing should be
introduce failures and even if they do, it statement, like ‘the users should be able included in your DevOps pipeline.
doesn’t stop the business from servicing to submit loans 99.99% of the time in the c. SecOps should be inculcated as
the clients. system and should be able to view the practice in the organization. As
V. Alignment of builders: loan status response within 3 seconds of developers and leads deal with
submitting the loan in the system.’ DevOps, Sysadmins and Security
It is important that builders of software
are provided a formal initial alignment or Also, by studying the various ASRs and Architects should come together to
induction process, which educates them on performing a trade-off or a risk analysis define and practice SecOps in their
the nature of distributed systems, the likely matters, because the more we try to organization.
problems they will face, and the common design a perfect failure-free system or For example, it could be decided that all OS
set of patterns which can be used to avoid resilient system, the more the cost goes up. and other software used should always be
those. It is natural that in a distributed Sometimes it is okay to let it fail and retry (n-1) version in production.
system with many moving parts and many later. Such decisions can be taken only if
there is a good study and clear articulation d. Adoption of enterprise-ready, proven,
teams, people tend to fall into what I call
of the ASRs in the system. and a good community-backed
as the ‘farther from repercussion’ problem.
Open Source Software (OSS) is one
Since everyone is farther from the larger
picture in an enterprise and there are VII. Adoption of DevOps: area where security has done well in
terms of publishing quick patches
always deployments and developments A deployment should take care of ease
for vulnerabilities as opposed to
from multiple teams happening at the of testing and provide the team a sense
proprietary software which are
same time, it is possible that the team of ownership and the confidence that
in general influenced by product
‘feels’ that it is ‘someone else’s problem’ and whatever they have developed will work
roadmaps and differences in support
that ‘I have no control on the final picture’ well in production. DevOps processes
levels.
or ‘how do I know this will happen.’ This ensure that the ride from development to
needs to be addressed and avoided. It is deployment phase is not jittery. A lot of e. Block chain, AI, and ML are into the
important that programmers are evolved resources are available about the benefits cybersecurity space and newer models
to developers. A developer is the one who

External Document © 2019 Infosys Limited

 in fields like cognitive and semantic XI. Dependency SLA issues:
security help to identify new patterns
Why this is important - After the discovery When the application we build has
which are difficult to manually detect.
of critical Spectre-NG vulnerability, the dependencies, it is imperative that we read
For example, AI/ML algorithms detect following statement was issued by Intel the fine print of the documents governing
intrusions and anomalies which are – “Protecting our customers’ data and their availability or durability or backup
normally left undetected by rule ensuring the security of our products are mechanisms.
configurations done by humans. critical priorities for us. We routinely work
closely with customers, partners, other For example, a customer was using AWS
f. Distribution of the security context EBS for storing core application data
chipmakers and researchers to understand
between multiple services and without storing snapshots of the same in
and mitigate any issues that are identified,
solutions is a proven strategy to avoid a more durable and available S3 storage.
and part of this process involves reserving
attacks on a singular context. If the fine print of AWS can be read (as
blocks of CVE numbers. We believe strongly
g. Last but not the least, the weakest in the value of coordinated disclosure of March 2019) it reads, “Amazon EBS
part of any cyber security system are and will share additional details on any volumes are designed for an annual failure
humans. Design systems which prevent potential issues as we finalize mitigations. rate (AFR) of between 0.1% - 0.2%, where
this vulnerability. Example of a poor As a best practice, we continue to failure refers to a complete or partial loss
design is having a software storing encourage everyone to keep their systems of the volume, depending on the size and
a password in a format readable by up-to-date.” [11] performance of the volume. This makes
humans or having a single-factor EBS volumes 20 times more reliable than
authentication for key use cases. typical commodity disk drives, which fail
It is also to be mandated that all with an AFR of around 4%. For example, if
h. Multi-factor authentication, maker-
maintenance scripts and contingency you have 1,000 EBS volumes running for
checker process for critical business
scripts written for the software are 1 year, you should expect 1 to 2 will have
services, encryption, OTP mechanisms
kept updated and any release made to a failure. EBS also supports a snapshot
etc., are some of the methods which
production is inclusive of updates made feature, which is a good way to take point-
can avoid human vulnerabilities.
to the contingency scripts and SOP in-time backups of your data” [10]
A Robust system should also handle documents A dependency system having 97%
existing and newer vulnerabilities (sites
like https://oval.cisecurity.org/, https://cve.
X. Transition state issues: availability should not be used for use
cases requiring 99.99% availability. If used,
mitre.org/cve/ and https://nvd.nist.gov/ Many of the failures occur during an
then care has to be taken to incorporate
) are some of the open public sites which inconsistent or transient state of systems.
additional replicas and other mechanisms
publish vulnerabilities. A failure occurring during an environment
to achieve the intended availability.
startup and shutdown, can cause more
Ceteris paribus, a less secure system is
damage to the data and the state of the
more likely to fail. We’ve only scratched
application than a failure occurring during
the surface and this is a topic and study
normal business operating conditions.
on its own. We recommend cyber-security
Extra care should be taken to add testing
practices that address both human and
methods and detailed designs to handle
non-human vulnerabilities and how all
such scenarios.
systems, processes, and humans can be
patched at regular intervals. Transition scenarios are special cases and
the variables governing the system during
IX. Currency issues:
transition states are sometimes different
As a general practice, all software and than normal operational conditions.
products used in production should be
Some examples of transition states
at the most be (n-2) versions or within
are – During DR processes, replication
two years of a major release. This should
backups, reverse DR times, while writing
be a part of product upgrade strategy to
point-in-time disk snapshots to storage, a
avoid foreseen and unforeseen issues. All
false positive event in the system triggers
patches should be taken into a security and
a contingency process and needs to be
operations group -- here is where the role
reversed, etc.
of the SecOps team is vital -- and should be
patched accordingly.

External Document © 2019 Infosys Limited

External Document © 2019 Infosys Limited
XII. Lack of robust monitoring: dependencies and introduce failures when generated every day, it is humanly
the operating environment changes (say a impossible to peruse and correlate issues.
One of the key areas of failure handling is version of the Linux kernel or package of AI and ML pattern recognition algorithms
the ability of the system to monitor itself Linux). On the other end, asking questions can identify, predict, and report failures
and detect a failure or a pattern which can like, “Is my program going to run in a and anomalies from the piles of monitoring
cause potential disruption of services. memory-constrained environment like data.
A good monitoring system today should PoS terminal?” brings out good design and
For example, ML algorithms based on
incorporate event correlation, detect algorithms which are frugal and efficient.
‘survival analysis’ [23] models can be used
and provide graphical information of In short, Mechanical sympathy is to be to predict failures.
the data flows, trace processes across exercised wisely.
distributed systems, predict event streams XVI. Lack of training:
for failure, alert users and systems, trigger XIV. DR switchover processes: It is important that Ops Team and
contingency scripts for automatic recovery, Software should participate in the DR Application Team should train together
collect logs, search through them, detect and reverse DR switch over processes as with mock DR drills.
anomalies in traffic and network, have failure recovery is never an Ops team-only
configurable rule addition capabilities By engaging in such activities, developers
issue. Recovering from failures involves
to identify newer patterns in the logs, realize that the software which they
understanding of the data storage, data
trace flows (this is very important as the have built has failed to cope up with
replication strategy, what is needed to
tracing gets very graphical with multiple the resilience SLA. For example, the lead
recover the state of the system when
dimensions of dependencies), and to could realize that the algorithm used to
it went down, having enough logs to
top it all, have AI and ML capabilities to defer database writes affects RTO. Also
investigate the issues, and software plays a
sort, cluster, learn, and identify patterns most of the application maintenance and
major role in all of this.
from the vast amount of monitoring data contingency scripts are written by the
collected. To have a single monitoring tool For example, writing a software which application team and any scope to reduce
with all these features is very tough and we doesn’t produce enough information to manual work and automate the same can
may need to have a mix of tools. debug issues and understand the failures be done.
is clearly a design flaw which has not
Example: A rare component failure in Training also detects problems like the SOP
factored a DR switchover process and RPO.
a backup switch at a VISA data center document not being fool-proof and while
caused an outage in June 2018, resulting XV. Automation, AI & ML: executing steps, new scenarios or failures
in the failure of 5.2 million card payments. could be encountered.
The ability of the system to recover from
Visa admitted that the software to failures sometimes involves executing XVII. Infrastructure Failures:
automatically detect failure was not in manual procedures and hence is prone to
place and the same was corrected. [13] error, time consuming, and introduces key- Software depends on infrastructure
man dependency. services like network, storage, OS, etc.
The key to resilience is in case of a failure
While choice of architectural design,
event, the system should quickly detect For example, in March 2016 a major outage replication, and deployment strategies
and deploy counter measures aided by a occurred at Telstra, associated to a human- takes care of the software side of issues, the
good monitoring system. made error. IT operations consultant Sam infrastructure side of things needs to be
XIII. Mechanical sympathy: Newman of Thought Works responded handled separately and if the infrastructure
to it, “It’s about the system you create, it’s is down, then everything built on top of
Mechanical sympathy [22] can be not about individuals.” and continued “… it will go down. Redundancy, clustering,
understood as the state of mind Looking for a single cause of failure is like replication, a good monitoring and alerting
a developer is in, when he tries to looking for a single cause of success,”. [12] system, selection of reliable infrastructure
understand the environment where his
Automation scripts and robotic processes etc., are some of the many processes and
code or tool runs and tries to design and
are some of the mechanisms which can tools which are adopted today to handle
optimize the code or tool based on that
be employed in production to avoid infrastructure failures.
information. This can be static or dynamic.
For example, if a developer writes code delays in running recovery mechanism in
to observe the RAM usage dynamically production. Self-healing, auto-healing, and
and performs garbage collection, then self-stabilization mechanisms should be
he is said to have designed his code with adopted as much as possible.
‘mechanical sympathy’. Adoption of AI and ML is important as
On one side, it tends to couple with the with Giga/Tera bytes of logs and events

External Document © 2019 Infosys Limited

 XVIII. Processes – a closer cost cutting, and lack of expertise.
look: Some principles to look at - YAGNI – You
In an enterprise, processes are the key Ain’t Gonna Need It, KISS – Keep it Simple
delivery vehicles and it is important and Short, Data is shared, Compliance
that we take a closer look at the various with Law, Always think ahead of the quick
processes governing software. fix, Don’t use a cannon to kill a mosquito,
Open-Closed principle, Illusion of control,
In one of our assessments we found that Premature optimization is evil, Never
a software was tested and pushed into trust anything which is coming into your
production and the same was also pushed module or system, etc.
to the DR environment without testing,
as it was identical. This gap in testing One example is on the principle of ‘always
and change rollout processes introduces document your design’ - All design and
an additional risk which ‘could’ve been documents at a high level should be
mitigated’ if we had done a basic sanity captured in some sort of document
testing in DR environment. We identified so that it is reviewed. Agile prefers
this and the risk was mitigated. working software over comprehensive
documentation, but never mentions ‘no
In March 2019, operation ShadowHammer documentation’ [18]. Not knowing how a
attacked thousands of users by infecting complex distributed cache works in your
the live update server in ASUS with enterprise will cause an issue, if not today,
malware. According to the verge, the then sometime tomorrow.
origin of this issue was likely a ‘supply
chain’ attack where malicious software Why this is important – Let’s take the
or components are installed on systems general case of attaching a node to a load
before or while building the systems. [16] balancer when one of the servers goes
down. It is a pretty common use case, but
The key lesson here is to pay attention when we simply follow this pattern without
to all processes in the enterprise which understanding the after effects of it, this
directly or indirectly affect the software. will lead to what is called as a ‘black hole
XIX. Primacy of architecture effect’ [19]. A black hole effect happens
principles: when the load balancer basically finds out
that the new server added actually has no
Like they decide on the various good
sessions affined to it and starts routing all
patterns and best practices, it is also
requests to the new server. This new server
equally important that developers and
now ‘sucks’ all requests and hence is over
architects always have an internal compass
flooded with requests causing sudden
pointing to the common principles of
imbalance and slowness.
architecture to guide them at every point
of time. This anti-pattern of just attaching a node
to a load-balancer was later discovered and
It is common that developers and
was fixed by adding various weightage,
architects are getting trained in many
throttling, and limits to the load balancer.
distributed technologies today and with
What principles could have prevented this
the technology marketing heave, it is
during design? How about, always think
common that developers are lost and make
ahead of the quick fix?
choices in technology and design which
may look like it is paying off now or give a In general, good architecture principles
‘feeling’ that it will solve their issues today, guide us to develop good resilient systems
but may eventually cause other issues. and also help weed out anti-patterns in our
Though architecture checks and balances systems. TOGAF framework has published
can be setup to review and approve new some of the key principles [20] and the
designs, sometimes things pass through same can be read from the reference
many such measures due to other forces section below.
like time constraints, delivery constraints,

External Document © 2019 Infosys Limited

Summary
Designing and developing a distributed
software is challenging in many ways, but
the benefits far outweigh the cons and
rest assured, the ecosystem of distributed
system is growing and continues to grow
rapidly, with technology and software
fast evolving to handle the failures and
issues which are envisioned in distributed
systems. Here are some of the golden
takeaways for resilience.

1. Resilience is not related to a singular

context in distributed systems and
is not dependent on humans to be
error-free. All systems, processes,
tools, libraries, software, hardware,
infrastructure, dependent services,
etc., come together to achieve resilient
business operations.

2. There are no failure-free systems. No

system can be declared 100% resilient
or failure-free.

3. Understanding and handling failures

is key to resilience. All processes
governing software - design,
development, deployment, operations
etc., can contribute to a failure.

4. Practically all software built by

major technology companies have
experienced failures and caused
downtime in some form or another,
such as at Facebook, Apple, Boeing,
Amazon Google, Microsoft, Linux to
name a few. So rest assured, you are
not alone.

5. There is no book listing all failures in

software, however there are lists of
things which are known to occur.

6. In a distributed set-up, sometimes

failures are not in your control and
you need to carry on trusting the
dependent systems. You can never be
completely free of failures; you can only
prevent or embrace them. Embracing
them and continually building
mechanisms to handle them is the only
way to make your system more robust.

Resiliency is a process continuum

and should keep evolving from
time to time.

External Document © 2019 Infosys Limited

About SPEAR:
SPEAR stands for Software Performance Engineering and ARchitecture services and is an integral part of Infosys STAR (Solution Technology
and ARchitecture) group aiding primarily the banking and financial vertical in Infosys. We are a large and niche group of Technical and Senior
Technical Architects, specializing in assessing and improving performance, resilience, scalability, availability, and reliability in distributed
systems through our tailored services and offerings.
For more details contact us at: [email protected]

References
1. German bank error - http://content.time.com/time/business/article/0,8599,1952305,00.html
2. Year 2010 problem - https://www.dw.com/en/millions-of-german-bank-cards-hit-by-software-bug/a-5088075
3. Reliability - http://www.cse.msu.edu/~stire/HomePage/Papers/wadsChapter05.pdf
4. Idempotent failover - https://www.springer.com/us/book/9783540407270
5. Traffic stats - https://www.similarweb.com/website/facebook.com#overview
6. Downtime and lag time - https://medium.com/@vikigreen/impact-of-slow-page-load-time-on-website-performance-40d5c9ce568a
7. Downtime of amazon - https://www.forbes.com/sites/kellyclay/2013/08/19/amazon-com-goes-down-loses-66240-per-minute/#5eabc9b6495c
8. Amazon revenue by segment - online sales - https://www.statista.com/statistics/672747/amazons-consolidated-net-revenue-by-segment/
9. 3 Banks DDoS attacks - https://www.cshub.com/attacks/news/incident-of-the-week-ddos-attack-hits-3-banks
10. EBS - AWS - https://aws.amazon.com/ebs/features/#Amazon_EBS_Snapshots
11. Intel’s comments - https://www.theregister.co.uk/2018/10/08/intel_security_commitment/
12. Telstra human error - https://www.news.com.au/technology/gadgets/mobile-phones/telstra-explains-network-outage-as-worker-faces-the-music/
news-story/7e3f2214350094c3c2096ad14f7480ae
13.VISA outage - https://www.cbronline.com/news/visa-outage
14. Actor Model - https://doc.akka.io/docs/akka/2.5/guide/actors-intro.html
15. Chaos Engineering - https://principlesofchaos.org/
16. ASUS attack - https://techhq.com/2019/03/asus-breach-highlights-software-supply-chain-risk/
17. Let it Fail approach - http://ward.bay.wiki.org/view/let-it-fail
18. Agile Manifesto - https://agilemanifesto.org/
19. Black Hole - https://www.brianmadden.com/opinion/Dealing-with-the-Black-Hole-Effect-Throttling-Logons-to-New-Servers
20. TOGAF Architecture principles - http://pubs.opengroup.org/architecture/togaf8-doc/arch/chap29.html
21. Flow Modeling references:
a. FMEA - https://wiki.ece.cmu.edu/ddl/index.php/FMEA
b. PGM - http://pgm.stanford.edu/algorithms/
22. Mechanical Sympathy - https://dzone.com/articles/mechanical-sympathy
23. Survival analysis - https://en.wikipedia.org/wiki/Survival_analysis
24. Architecturally Significant Requirements:
a. https://www.ida.liu.se/~TDDD09/openup/core.tech.common.extend_supp/guidances/concepts/arch_significant_requirements_1EE5D757.html
b. https://www.ibm.com/developerworks/rational/library/4706.html

For more information, contact [email protected]

© 2019 Infosys Limited, Bengaluru, India. All Rights Reserved. Infosys believes the information in this document is accurate as of its publication date; such information is subject to change without notice. Infosys
acknowledges the proprietary rights of other companies to the trademarks, product names and such other intellectual property rights mentioned in this document. Except as expressly permitted, neither this
documentation nor any part of it may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, printing, photocopying, recording or otherwise, without the
prior permission of Infosys Limited and/ or any named intellectual property rights holders under this document.

Infosys.com | NYSE: INFY Stay Connected