Fundamentals of Scalable Computing

Microservices

Dr. Radhika M. Hirannaiah

Department of Computer Science and Engineering
 FUNDAMENTALS OF SCALABLE COMPUTING
Application Architectures - Monolithic
An application architecture describes the approach or patterns and techniques which are used to design and
build an application.
Monolithic Architecture
• Monolithic Architecture is the approach of building the application wherein all the features of the system are put on a
single codebase and are typically with a single database
• In a monolithic, all the components share the same resources and memory space
• It is considered to be a traditional way of building applications.
• A monolithic application is built as a single and indivisible unit. This could be visualized as in the
figure with a client-side UI, Business logic and data access layer together with a database. If
distributed you could have Server side application and a database access there.
• They can be characterized by being large, having long release cycles and having large teams
Typical Challenges
▪ Changes are not easily integrated (whole system may need to be rebuilt)
▪ Scalability challenges
▪ New Technology Adoption
▪ Difficult to adapt to newer practices, devices
▪ Reliability can be a question
FUNDAMENTALS OF SCALABLE COMPUTING
Strengths of Monolithic Architecture

Strengths of Monolithic Architecture

1. Development is quite simple.

2. Testing is very simple - Just launch the application and start end-to-end testing. We can also do test automation
using Selenium without any difficulty.

3. Deploying the monolithic application is straightforward - Just copy the packaged application to the server.

4. Scalability is simple - We only need to have a new instance of the monolithic application and ask the load
balancer to distribute load to the new instance as well. However, as the monolithic application grows in size,
scalability becomes a serious issue.

Monolithic architecture worked successfully for many decades. In fact, many of the most successful and largest
applications were initially developed and deployed as a monolith.
But serious issues like flexibility, reliability and scalability have led to the emergence of microservices architecture.
FUNDAMENTALS OF SCALABLE COMPUTING
Microservices Architecture

What are Microservices ?

Microservices are small services that work together but are independent, and are
components or processes of an application. They have benefits of dynamic scalability,
Reliability etc. as discussed later.

What is an Microservice Architecture

‘’The microservice architectural style is an approach of developing a single
application as a suite of small services, each running in its own process and
communicating with lightweight mechanisms, often an HTTP resource API.

These services are built around business capabilities and independently deployable
by fully automated deployment machinery.

There is a bare minimum of centralized management of these services, which may be

written in different programming languages and use different data storage
technologies. ‘’

- Martin Fowler
FUNDAMENTALS OF SCALABLE COMPUTING
Benefits of Microservices Architecture

1. Flexibility:
Microservices architecture is quite flexible. Different microservices can be developed in different
technologies. Since a microservice is smaller, the code base is quite less, so it’s not that difficult to upgrade the
technology stack versions. Also, we can incrementally adopt a newer technology without much difficulty.

2. Reliability:
Microservices architecture can be very reliable. If one feature goes down, the entire application doesn’t go
down. We can fix the issue in the corresponding microservice and immediately deploy it.

3. Development speed:
Development is pretty fast in microservices architecture. Since the volume of code is much less for a
microservice, it’s not difficult for new team members to understand and modify the code. They become
productive right from the start. Code quality is maintained well. The IDE is much faster. A microservice takes
much less time to start up. All these factors considerably increase developers' productivity.
FUNDAMENTALS OF SCALABLE COMPUTING
Benefits of Microservices Architecture (contd.)

4. Building complex applications:

With microservice architecture, it's easy to build complex applications. If the features of the application
are analyzed properly, we can break it down into independent components which can be deployed
independently. Then, even the independent components can be further broken down into small independent
tasks which can be deployed independently as a microservice. Deciding the boundaries of a microservice can
be quite challenging. It’s actually an evolutionary process, but once we decide on a microservice, it’s easy to
develop, as there are no limitation in technologies.

5. Scalability:
Scalability is a major advantage in microservice architecture. Each microservice can be scaled individually.
Since individual microservices are much smaller in size, caching becomes very effective.

6. Continuous deployment:
Continuous deployment becomes easier. In order to update one component, we have to redeploy only
that particular microservice.
FUNDAMENTALS OF SCALABLE COMPUTING
An example of a Microservices Architecture

Clients cannot call the Services directly and API Gateway

acts as an entry point for clients to forward requests to microservices
Advantages
• All services can be updated without the clients knowledge
• Services can also use messaging protocols which are not web friendly
• API can provide cross-cutting functions like security, load balancing etc.
FUNDAMENTALS OF SCALABLE COMPUTING
Principles Microservices
There are the following principles of Microservices:
▪ Single Responsibility principle
The single responsibility principle states that a class or a module in a program should have only one responsibility.
Any microservice cannot serve more than one responsibility at a time.
▪ Modelled around business domain
Microservice never restrict itself from accepting appropriate technology stack or database. The stack or database is
most suitable for solving the business purpose.
▪ Isolate Failure
The large application can remain mostly unaffected by the failure of a single module. It is possible that a service can
fail at any time. So, it is important to detect failure quickly, if possible, automatically restore failure.
▪ Infrastructure automation
The infrastructure automation is the process of scripting environments. With the help of scripting environment, we
can apply the same configuration to a single node or thousands of nodes. It is also known as configuration
management, scripted infrastructures, and system configuration management.
▪ Deploy independently
Microservices are platform agnostic. It means we can design and deploy them independently without affecting the
other services.
 FUNDAMENTALS OF SCALABLE COMPUTING
Microservices Limitations
Building: Upfront time needs to be spent in terms of identifying dependencies between your services.
Be aware that completing one build might trigger several other builds, due to those dependencies.
You also need to consider the effects that microservices have on your data.
Testing: Integration testing, as well as end-to-end testing, can become more difficult as a failure in one part of the
architecture could cause something a few hops away to fail, depending on how you’ve architected your services to
support one another.
Versioning: When you update to new versions, keep in mind that you might break backward compatibility. You can build
in conditional logic to handle this, but that gets unwieldy and nasty, fast. Alternatively, you could stand up multiple live
versions for different clients, but that can be more complex in maintenance and management.
Deployment: Needs initial investment towards automation as the complexity of microservices becomes overwhelming for
human deployment. Think about how you’re going to roll services out and in what order.
Logging: With distributed systems, you need log management as the scale makes it hard to manage.
Monitoring: This may need to have a centralized view of the system to pinpoint sources of problems.
Debugging: Remote debugging through your local integrated development environment (IDE) isn’t an option and it won’t
work across dozens or hundreds of services. Its difficult with no single answer to how to debug.
Connectivity: Consider service discovery, whether centralized or integrated
https://www.redhat.com/en/topics/cloud-native-apps/what-is-an-application-architecture#microservices-architecture
FUNDAMENTALS OF SCALABLE COMPUTING
Microservices Limitations (others)
FUNDAMENTALS OF SCALABLE COMPUTING
Microservice vs Monolithic Model

Monolithic Architecture
• It is considered to be a traditional way of building applications.
• A monolithic application is built as a single and indivisible unit. Usually,
such a solution comprises a client-side user interface, a server side-
application, and a database.

Microservices Architecture
• While a monolithic application is a single unified unit, a microservices
architecture breaks it down into a collection of smaller independent
units.
• These units carry out every application process as a separate service. So
all the services have their own logic and the database as well as
perform the specific functions.
 FUNDAMENTALS OF SCALABLE COMPUTING
Comparison with SOA
Microservice architecture is generally considered an evolution of SOA as its services are more fine-grained and
function independently of each other.
Service-oriented architecture was largely created as a response to traditional, monolithic approaches to building
applications. SOA breaks up the components required for applications into separate service modules that
communicate with one another to meet specific business objectives.
FUNDAMENTALS OF SCALABLE COMPUTING
Comparison with SOA

SOA Microservice Architecture

Follows “share-as-much-as-possible” architecture approach Follows “share-as-little-as-possible” architecture approach
Importance is on the concept of “bounded context” or Single
Importance is on business functionality reuse
Responsibility
They focus on people, collaboration and freedom of other
They have common governance and standards
options
Uses Enterprise Service bus (ESB) for communication Simple messaging system
They support multiple message protocols They use lightweight protocols such as HTTP/REST etc.
Single-threaded usually with the use of Event Loop features for
Multi-threaded with more overheads to handle I/O
non-locking I/O handling
Maximizes application service reusability Focuses on decoupling
Traditional Relational Databases are more often used Modern Relational Databases are more often used
A systematic change requires modifying the monolith A systematic change is to create a new service
DevOps / Continuous Delivery is becoming popular, but not
Strong focus on DevOps / Continuous Delivery
yet mainstream
THANK YOU

Dr. Radhika M. Hirannaiah

Department of CSE
[email protected]
Fundamentals of Scalable Computing
Message-Oriented Middleware

Dr. Radhika M. Hirannaiah

Department of Computer Science and Engineering
FUNDAMENTALS OF SCALABLE COMPUTING
Communication Mechanism

▪ In a distributed computing environment like the Cloud, where the hardware infrastructure typically
configured as a cluster with different system architectures like a client-server or master-slave or say
a P2P architecture, there exists a significant interactions between the components or machines
involving data flows from one system to another which could be through messages or events.
▪ These data flows could happen using different protocols such as HTTP, AMQP (Advanced Message
Queuing Protocol, or a binary protocol like TCP and with different data formats.
▪ There are also different styles or natures through which these communication may happen like
synchronous or asynchronous or based on the request and number of processors of the service
▪ If the application is a monolith with multiple processes, simple intra-system IPC mechanism may
work to communicate with processes which need to interact, but when built as a set of say
microservices maybe running on different systems, communication mechanisms which can support
Inter-service communication would be essential
Ref: https://docs.microsoft.com/en-us/dotnet/architecture/microservices/architect-microservice-container-applications/communication-in-microservice-architecture
https://chanakaudaya.medium.com/a-futuristic-view-of-building-distributed-systems-with-messaging-560d0652513a
FUNDAMENTALS OF SCALABLE COMPUTING
Enterprise Service Bus

• An enterprise service bus implements a communication system between

mutually interacting software applications in a service-oriented architecture
• ESB represents the piece of software that lies between the business
applications and enables communication among them.
• The enterprise service bus serves an analogous function at a higher level of
abstraction.
• The inner bubbles such as Routing, Security consists of some of the functionality
of ESB.
• The term “enterprise service bus” or ESB refers to the case where the bus
supports the convenient integration of many components, often in different
styles
FUNDAMENTALS OF SCALABLE COMPUTING
Functions of Enterprise Service Bus

The primary functions of an ESB are:

• Route messages between services
• Monitor and control routing of message exchange between services
• Resolve contention between communicating service components
• Control deployment and versioning of services
• Marshal use of redundant services
• Provide commodity services like event handling, data transformation and mapping, message
and event queuing and sequencing, security or exception handling, protocol conversion and
enforcing proper quality of communication service.
 FUNDAMENTALS OF SCALABLE COMPUTING
Enterprise Service Bus – How does it work?

• One does not open a channel between source and destination, but rather inject a message into the bus with enough
information to allow it to be delivered correctly
• The injection is performed by code loaded into each service and represented by the filled ovals as client interfaces
• The use of multiple brokers allows the bus to scale to many clients (services) and large message traffic.
• Note that the brokers are “just” special servers/services that receive messages and perform needed transformations and
routing on them and send out new messages.
• There is a special (simple) case of message buses where the brokers shown are not separate servers but are included in
the client software. Note that such buses support not just point-to-point messaging but broadcast or selective multicast to
many recipient clients (services)
FUNDAMENTALS OF SCALABLE COMPUTING
Interaction styles

Interaction styles can be categorized in two dimensions

First dimension
1. One-to-one : Each client request is processed by exactly one service
2. One-to-many : Each request is processed by multiple services

Second dimension
1. Synchronous : The client expects a timely response from the service and might even block while it waits.
2. Asynchronous : The client doesn’t block, and the response, if any, isn’t necessarily sent immediately.
FUNDAMENTALS OF SCALABLE COMPUTING
One-to-one interaction

The following are the different types of one-to-one interactions

1. Request/response— A service client makes a request to a service and waits for a response. The client
expects the response to arrive in a timely fashion. It might event block while waiting. This is an
interaction style that generally results in services being tightly coupled.

2. Asynchronous request/response— A service client sends a request to a service, which replies

asynchronously. The client doesn’t block while waiting, because the service might not send the
response for a long time.

3. One-way notifications— A service client sends a request to a service, but no reply is expected or sent.
FUNDAMENTALS OF SCALABLE COMPUTING
One-to-many interaction

The following are the different types of one-to-many interactions

- Publish/subscribe— A client publishes a notification message, which is consumed by zero

or more interested services.

- Publish/async responses— A client publishes a request message and then waits for a
certain amount of time for responses from interested services.
FUNDAMENTALS OF SCALABLE COMPUTING
Interaction styles
1.Synchronous (request-response) :
▪ Synchronous messaging means that the system which is
sending the message expects an immediate response from
the target system.
▪ Source system sends a message (request) to the target system and waits (blocks) until it receives
a response from the target system.
▪ Target system may process the message within itself or send the message to another system and
generate a response within a short time period.
▪ To make this communication successful and make sure source system does not waste its
resources in case of a target system failure, there is a timeout configured at the source side so
that it will stop waiting for the response if the timeout is elapsed without receiving a response
from the target system.
▪ There are variants of this like with call-back and full-duplex, which you could go through
https://chanakaudaya.medium.com/a-futuristic-view-of-building-distributed-systems-with-messaging-560d0652513a
FUNDAMENTALS OF SCALABLE COMPUTING
Synchronous (Request – response)
▪ Request-response style communication can be implemented in a blocking manner as well
as non-blocking manner.
▪ Blocking style: Instead of waiting for the response, source system can continue its work
while registering a callback in the source system side so that whenever there is a response
available from the target, this callback gets executed and the source system can continue
processing the response. This is the common pattern of implementation of most request-
response style systems because it does not blocks the resources (CPU) on the source side
instead continue utilizing resources for other tasks.

https://chanakaudaya.medium.com/a-futuristic-view-of-building-distributed-systems-with-messaging-560d0652513a
FUNDAMENTALS OF SCALABLE COMPUTING
Synchronous (Request – response)

▪ The latest improvement in the synchronous or request-response style of messaging is the full-
duplex style of messaging where both source and target systems can communicate via the same
connection at the same time.
▪ In both the above mentioned blocking and callback based synchronous messaging architectures,
one system can communicate at a given time through the connection which we called as half-
duplex or simplex.
▪ But this style of communication was not enough for modern web applications with high data
demands. The protocols like Websockets and HTTP/2 are designed in this manner.
▪ Messaging style improved the performance of the web applications through various technologies
like
▪ Request multiplexing, Header compression, Server Push, Request prioritization

https://chanakaudaya.medium.com/a-futuristic-view-of-building-distributed-systems-with-messaging-560d0652513a
FUNDAMENTALS OF SCALABLE COMPUTING
Interaction styles
2. Asynchronous :
▪ The client doesn’t block, and the response, if any, isn’t necessarily sent immediately
▪ This supports high rates of data flow.
▪ The expectations of these type of messages are
▪ Guaranteed delivery
▪ Extensive processing
▪ Correlation and time series analysis
▪ Decoupling of source and target systems
▪ These could be of the types
▪ Publish Subscribe (based on topics)
▪ Message Queues
▪ Event based real time processing (not planned for discussion)
▪ Batch Processing
▪ Store and Forward https://chanakaudaya.medium.com/a-futuristic-view-of-building-distributed-systems-with-messaging-560d0652513a
FUNDAMENTALS OF SCALABLE COMPUTING
Asynchronous vs Synchronous messaging

Advantages of Asynchronous messaging

- Reduced coupling: The message sender does not need to know about the consumer.

- Multiple subscribers: Using a pub/sub model, multiple consumers can subscribe to receive events.

- Failure isolation: If the consumer fails, the sender can still send messages. The messages will be
picked up when the consumer recovers. Asynchronous messaging can handle intermittent downtime.
Synchronous APIs, on the other hand, require the downstream service to be available or the
operation fails.

- Load leveling: A queue can act as a buffer to level the workload, so that receivers can process
messages at their own rate.
FUNDAMENTALS OF SCALABLE COMPUTING
Asynchronous vs Synchronous messaging

Disadvantages of Asynchronous messaging

- Coupling with the messaging infrastructure: Using a particular messaging infrastructure may cause
tight coupling with that infrastructure. It will be difficult to switch to another messaging
infrastructure later.

- Latency: End-to-end latency for an operation may become high if the message queues fill up.

- Complexity: Handling asynchronous messaging is not a trivial task. For example, handling of
duplicated messages, or correlating request and response messages using a separate response
queue.

- Throughput: If message queues are used, each message requires at least one queue operation and
one dequeue operation. Moreover, queue semantics generally require some kind of locking inside
the messaging infrastructure.
FUNDAMENTALS OF SCALABLE COMPUTING
Messaging models

There are two main paradigms for asynchronous communication :

1. Message Queues

2. Event streams (Pub/Sub)

FUNDAMENTALS OF SCALABLE COMPUTING
Message Queues https://aws.amazon.com/message-queue/

A message queue is a form of asynchronous service-to-service communication used in serverless and microservices
architectures.

Messages are stored on the queue until they are processed and deleted.

Each message is processed only once, by a single consumer.

Message queues can be used to decouple heavyweight processing, to buffer or batch work, and to smooth spiky
workloads.

Examples - Apache ActiveMQ, RabbitMQ, Apache Kafka

FUNDAMENTALS OF SCALABLE COMPUTING
Message Queues (contd.)

• A message queue provides a lightweight buffer which temporarily stores messages, and endpoints
that allow software components to connect to the queue in order to send and receive messages.

• The messages are usually small, and can be things like requests, replies, error messages, or just plain
information.

• To send a message, a component called a producer adds a message to the queue. The message is
stored on the queue until another component called a consumer retrieves the message and does
something with it.

• Many producers and consumers can use the queue, but each message is processed only once, by a
single consumer. For this reason, this messaging pattern is often called one-to-one, or point-to-
point, communications.

• When a message needs to be processed by more than one consumer, message queues can be
combined with Pub/Sub messaging in a fanout design pattern.
FUNDAMENTALS OF SCALABLE COMPUTING
Publish-Subscribe https://aws.amazon.com/pub-sub-messaging/

• Publish/subscribe messaging is another asynchronous service-to-service communication

mechanism.
• In a pub/sub model, any message published to a topic is immediately received by all of the
subscribers to the topic.
• Pub/sub messaging can be used to enable event-driven architectures, or to decouple applications in
order to increase performance, reliability and scalability.
FUNDAMENTALS OF SCALABLE COMPUTING
Publish-Subscribe (contd.)

Four core concepts make up the pub/sub model:

1. Topic – An intermediary channel that maintains a list of subscribers to relay messages to that
are received from publishers. There can be different topic channels & different subscribers can
look at their interested topics

2. Message – Serialized messages sent to a topic by a publisher which has no knowledge of the
subscribers

3. Publisher – The application that publishes a message to a topic

4. Subscriber – An application that registers itself with the desired topic in order to receive the
appropriate messages
FUNDAMENTALS OF SCALABLE COMPUTING
Publish-Subscribe (contd.)

Advantages

1. Loose coupling
Publishers are never aware of the existence of subscribers so that both systems can operate
independently of each other. This methodology removes service dependencies that are present in
traditional coupling.
For example, a client generally cannot send a message to a server if the server process is not
running. With pub/sub, the client is no longer concerned whether or not processes are running on the
server.

2. Scalability
Pub/sub messaging can scale to volumes beyond the capability of a single traditional data center.
This level of scalability is primarily due to parallel operations, message caching, tree-based routing,
and multiple other features built into the pub/sub model.
 FUNDAMENTALS OF SCALABLE COMPUTING
Publish-Subscribe (contd.)

Benefits of Pub-Sub model

3. Eliminate Polling:
Message topics allow instantaneous, push-based delivery, eliminating the need
for message consumers to periodically check or “poll” for new information and
updates.
This promotes faster response time and reduces the delivery latency that can be
particularly problematic in systems where delays cannot be tolerated.

4. Dynamic Targeting
Instead of maintaining a roster of peers that an application can send messages to,
a publisher will simply post messages to a topic.
Then, any interested party will subscribe its endpoint to the topic, and start
receiving these messages.
Subscribers can change, upgrade, multiply or disappear and the system
dynamically adjusts.
 FUNDAMENTALS OF SCALABLE COMPUTING
Publish-Subscribe (contd.)

Benefits of Pub-Sub model (contd.)

5. Decouple and Scale Independently:

Publishers and subscribers are decoupled and work independently from each other,
which allows you to develop and scale them independently.
Adding or changing functionality won’t send ripple effects across the system,
because Pub/Sub allows you to flex how everything uses everything else.

6. Simplify Communication
Communications and integration code is some of the hardest code to write. The
Publish Subscribe model reduces complexity by removing all the point-to-point
connections with a single connection to a message topic, which will manage
subscriptions to decide what messages should be delivered to which endpoints.
Fewer callbacks results in looser coupling and code that’s easier to maintain and
extend.
 FUNDAMENTALS OF SCALABLE COMPUTING
Redis https://aws.amazon.com/redis/

Redis, which stands for Remote Dictionary Server, is a fast, open-source, in-memory key-value data store
for use as a database, cache, message broker, and queue.

Why use Redis?

- All Redis data resides in-memory, in contrast to databases that store data on disk or SSDs. By
eliminating the need to access disks, in-memory data stores such as Redis avoid seek time delays and
can access data in microseconds.
FUNDAMENTALS OF SCALABLE COMPUTING
Oracle publish-subscribe model

• The Oracle Publish-Subscribe Model is a messaging pattern used to facilitate communication between
different applications or systems, particularly in scenarios where data needs to be sent from one system (the
publisher) to multiple recipients (the subscribers).
• It is commonly used in event-driven architectures, message-driven middleware, and real-time data
streaming.
• In Oracle's ecosystem, the publish-subscribe model is commonly applied in technologies like Oracle Advanced
Queuing (AQ), Oracle Cloud Messaging, and Oracle Service Bus, among others.
 FUNDAMENTALS OF SCALABLE COMPUTING
RabbitMQ
RabbitMQ is an open-source message-broker software that originally implemented the Advanced
Message Queuing Protocol (AMQP) and has since been extended to support Streaming Text
Oriented Messaging Protocol (STOMP), MQ Telemetry Transport (MQTT) and other protocols.

Message flow in RabbitMQ

1. The producer publishes a message to an exchange.
2. The exchange receives the message and is now responsible for routing the message. The
exchange takes different message attributes into account, such as the routing key, depending
on the exchange type.
3. Bindings must be created from the exchange to queues. In this case, there are two bindings to
two different queues from the exchange. The exchange routes the message into the queues
depending on message attributes.
4. The messages stay in the queue until they are handled by a consumer
5. The consumer handles the message.
 FUNDAMENTALS OF SCALABLE COMPUTING
Kafka

• Apache Kafka is an open-source distributed stream-processing software

platform.
• Event streaming is the practice of capturing data in real-time from event
sources like databases, sensors, mobile devices and cloud services

• Kafka combines three key capabilities for implementing event streaming

use cases end-to-end:
1. To publish (write) and subscribe to (read) streams of events,
including continuous import/export of data from other systems.
2. To store streams of events durably and reliably for as needed.
3. To process streams of events as they occur or retrospectively.
FUNDAMENTALS OF SCALABLE COMPUTING
Kafka
FUNDAMENTALS OF SCALABLE COMPUTING
Comparison of Messaging and Queuing Systems
THANK YOU

Dr. Radhika M. Hirannaiah

Department of CSE
[email protected]
Fundamentals of Scalable Computing
Workflows in SoA

Dr. Radhika M. Hirannaiah

Department of Computer Science and Engineering
FUNDAMENTALS OF SCALABLE COMPUTING
Workflows in SoA

• Services are the basic unit for constructing distributed

systems.
• An application consists of multiple interacting services
• A “Complete System” could be considered as a “Grid of
services”
• Workflow is used to integrate component grids and services
• Workflow is the approach to “programming the interaction
between services.”
• It describes “service orchestration” • These workflows are often coordinated by
• A workflow refers to the series of automated tasks or various components, such as orchestration
processes that are performed as part of a service's operation engines, to execute business processes or data
or interaction between services within the system. transformations across different services.
 FUNDAMENTALS OF SCALABLE COMPUTING
Key Components in SOA Workflow

• Services: The building blocks of SOA, each responsible for a specific function. These can be web services,
microservices, or other service types.
• Orchestration: This involves the coordination and sequencing of different services to achieve a business
process. Workflow orchestration engines (e.g., BPM engines, workflow managers) are responsible for
organizing and managing the sequence of service calls in a specific workflow.
• Business Process Execution Language (BPEL): A standard for defining workflows in SOA, BPEL is often used
to describe the logic of service interactions, including handling conditions, loops, and exception
management.
• Messaging and Communication: Workflows in SOA rely on asynchronous or synchronous communication
mechanisms, such as SOAP, REST, or message queues, to pass messages between services.
• Process Automation: Services in the workflow can be triggered automatically based on certain events, such
as a new request, time-based conditions, or input from other services.
FUNDAMENTALS OF SCALABLE COMPUTING
Basic Workflows Concepts

• Services are the basic unit for constructing distributed systems.

• Workflow implies a two-level programming model for distributed systems
• Each service is programmed in traditional languages (C, C++, Java and Python) while their interaction is
described by workflow developed using complex Shell (Perl) scripts
• Good workflow systems have been built on Petri Nets

https://aws.amazon.com/what-is/service-oriented-architecture/
FUNDAMENTALS OF SCALABLE COMPUTING
Workflow Standards
• A workflow standard refers to a set of guidelines, rules, and
specifications that govern how workflows should be designed,
implemented, and executed within an organization or across
different systems. These standards aim to ensure that workflows
are efficient, interoperable, scalable, and maintainable. By
adhering to a standard, workflows can be consistently defined,
automated, and integrated across various systems and platforms.
• Web service-related standards proposed by OASIS, OMG and W3C
- interoperability was achieved by complete specification of
service features
• This goal produced heavyweight architectures
• Today, greater emphasis is put on lightweight systems where
interoperability is achieved by ad hoc transformations where
necessary
FUNDAMENTALS OF SCALABLE COMPUTING
Workflow Architecture and Specification

• Most workflow systems have two key components corresponding to the language and runtime
components of any programming environment.
• We can call these components the workflow specification and workflow execution engine
• They are linked by interfaces which could be specified by documents using, for ex: BPEL
Workflow Specification
• Scripting-based workflow systems can specify workflow in traditional language syntax similar to Python,
JavaScript, or Perl.
• One can also directly specify the (XML) interface document driving the execution engine, although that is
pretty low-level. However, most workflow systems use a graphical interface as illustrated in Figure (next
slide)
FUNDAMENTALS OF SCALABLE COMPUTING
Workflow Architecture and Specification

Workflow Specification

• Each step is a “large” activity (i.e., a service) illustrating that workflow is a programming system at a
very different granularity from familiar languages such as C++.
• Scripting languages are often used for specifying coarse-grained operations, which “explains” why
scripting is a popular approach to specifying workflows.
FUNDAMENTALS OF SCALABLE COMPUTING
Workflow Architecture and Specification

Workflow Execution Engine

• A workflow engine tracks which steps an individual process instance
currently resides in, who is assigned to work on a particular task, or
which task needs to be claimed.
• Workflow structure is a collection of vertices and directed edges, each
edge connecting one vertex to another such that there are no cycles.
That is, there is no way to start at some vertex V and follow a sequence
A workflow graph that includes subgraphs
of edges that eventually loops back to that vertex V again
illustrating pipelines and loops
• Most sophisticated workflow systems support hierarchical
specifications—namely the nodes of a workflow can be either services
or collections of services
FUNDAMENTALS OF SCALABLE COMPUTING
Workflow Execution Engine Example

• A workflow execution engine is a

system component that executes
and manages the execution of
workflows according to a defined
specification.
• It orchestrates the sequence of
tasks, manages data flow,
handles exceptions, and ensures
that business processes are
executed smoothly.
• These engines are often used in
Business Process Management
(BPM) systems, Service-Oriented
Architecture (SOA), and other
automation scenarios.
FUNDAMENTALS OF SCALABLE COMPUTING
Amazon Simple Workflow Service

• Amazon Simple Workflow Service (Amazon SWF) makes it easy to build applications that coordinate work across
distributed components.
• In Amazon SWF, a task represents a logical unit of work that is performed by a component of an application.
• Coordinating tasks across the application involves managing intertask dependencies, scheduling, and concurrency
in accordance with the logical flow of the application.
• Amazon SWF gives full control over implementing tasks and coordinating them without worrying about underlying
complexities such as tracking their progress and maintaining their state.
• Using Amazon SWF, you implement workers to perform tasks. These workers run on cloud infrastructure, such as
Amazon EC2
• Amazon SWF stores tasks and assigns them to workers when they are ready, tracks their progress, and maintains
their state, including details on their completion. To coordinate tasks, you write a program that gets the latest state
of each task from Amazon SWF and uses it to initiate subsequent tasks

Ref: https://docs.aws.amazon.com/amazonswf/latest/developerguide/swf-dev-about-workflows.html
THANK YOU

Dr. Radhika M. Hirannaiah

Department of CSE
[email protected]
Fundamentals of Scalable Computing
NoSQL and Speciality Databases

Dr. Radhika M. Hirannaiah

Department of Computer Science and Engineering
FUNDAMENTALS OF SCALABLE COMPUTING
Key-Value database

• The key-value (KV) store is the simplest NoSQL model

• A KV store pairs keys to values in much the same way that a map (or hashtable) would in any
popular programming language
• Some KV implementations permit complex value types such as hashes or lists
• Some KV implementations provide a means of iterating through the keys

Example:
• A file system could be considered a key-value store if you think of the file path as the key and
the file contents as the value
FUNDAMENTALS OF SCALABLE COMPUTING
DynamoDB

• DynamoDB is a cloud-based database available through Amazon Web Services (AWS)

• DynamoDB is ideal for handling large-scale applications, like mobile apps, gaming backends, IoT, and e-commerce
systems, for use cases that require fast read and write operations on large amounts of data.
• All data in DynamoDB is stored in tables that you have to create and define in advance, though tables have some
flexible elements and can be modified later
• DynamoDB requires users to define only some aspects of tables, most importantly the structure of keys and local
secondary indexes, while retaining a schemaless flavor.
• DynamoDB enables users to query data based on secondary indexes rather than solely on the basis of a primary key
• DynamoDB supports only item-level consistency, which is analogous to row-level consistency in RDBMSs
• If consistency across items is a necessity, DynamoDB is not the right choice
• DynamoDB has no concept of joins between tables. Table is the highest level at which data can be grouped and
manipulated, and any join-style capabilities that you need will have to be implemented on the application side
FUNDAMENTALS OF SCALABLE COMPUTING
Key features of DynamoDB

• Scalability: DynamoDB can automatically scale to accommodate workloads of any size without requiring you to
manually manage the scaling process. It automatically adjusts throughput capacity based on demand, and it can
handle millions of requests per second with ease.
• Fully Managed: AWS takes care of all the administrative tasks, including hardware provisioning, setup, configuration,
and patching, so you don’t need to worry about the low-level details. This makes it very developer-friendly.
• Low Latency: DynamoDB offers low-latency data access, usually in the single-digit millisecond range, making it ideal
for applications that require fast responses.
• Flexible Data Model: DynamoDB supports both key-value and document data models. You can store structured or
semi-structured data and retrieve it based on primary key values, which is very efficient for high-throughput
workloads.
• Global Availability: DynamoDB offers global tables, which replicate data across multiple AWS regions for highly
available and low-latency access from anywhere in the world.
FUNDAMENTALS OF SCALABLE COMPUTING
Key features of DynamoDB

• Event-Driven Programming: DynamoDB can trigger AWS Lambda functions in response to data changes using
DynamoDB Streams. This can be used to implement real-time workflows or asynchronously process data changes.
• Consistency Models: DynamoDB supports both eventual consistency and strong consistency. You can choose the
level of consistency you want based on your application’s needs:
• Eventually consistent reads provide lower latency and higher throughput, but the data may be slightly out of
sync for a brief period.
• Strongly consistent reads guarantee that you always get the most up-to-date data, but they can have higher
latency and may reduce throughput.
• Built-in Security: It integrates with AWS Identity and Access Management (IAM) for access control, enabling you to
set granular permissions on who can access data. It also supports encryption at rest and in transit.
• Cost-Effective: DynamoDB uses a pay-per-request pricing model, meaning you only pay for the throughput (read and
write capacity) you use, or you can use on-demand mode to pay based on the actual number of reads and writes
your application performs.
FUNDAMENTALS OF SCALABLE COMPUTING
DynamoDB
• In DynamoDB, tables, items, and attributes are the core components
• A table is a collection of items and each item is a collection of attributes
• DynamoDB uses primary keys to uniquely identify each item in a table and
secondary indexes to provide more querying flexibility
• DynamoDB supports two different kinds of primary keys:
• Partition key – A simple primary key, composed of one attribute known as the
partition key. DynamoDB uses the partition key's value as input to an internal hash
function. The output from the hash function determines the partition (physical
storage internal to DynamoDB) in which the item will be stored.
• Partition key and sort key – Referred to as a composite primary key, this type of key
is composed of two attributes. The first attribute is the partition key and the second
attribute is the sort key. All items with the same partition key value are stored
together in sorted order by sort key value. The Partition Key decides the physical
Location of Data, but the Sort Key then decides the relative logical position of
associated item's record inside that physical location.
FUNDAMENTALS OF SCALABLE COMPUTING
DynamoDB

Secondary Indexes
• Users can create one or more secondary indexes on a table.
• A secondary index lets users query the data in the table using an alternate key, in addition to queries
against the primary key.

Partitions and Data Distribution

• DynamoDB stores data in partitions.
• A partition is an allocation of storage for a table, backed by solid state drives (SSDs) and automatically
replicated across multiple Availability Zones within an AWS Region.
• Partition management is handled entirely by DynamoDB
FUNDAMENTALS OF SCALABLE COMPUTING
DynamoDB Architecture
FUNDAMENTALS OF SCALABLE COMPUTING
DynamoDB Architecture
FUNDAMENTALS OF SCALABLE COMPUTING
DynamoDB Architecture
FUNDAMENTALS OF SCALABLE COMPUTING
DynamoDB Architecture
FUNDAMENTALS OF SCALABLE COMPUTING
DynamoDB Use cases

• E-commerce Platforms: Storing user profiles, cart items, orders, and transaction logs that need to
be accessed rapidly and scale with user traffic.
• IoT Applications: Storing sensor data or device states, where high write throughput is essential.
• Gaming Backends: Storing user sessions, scores, achievements, and real-time leaderboards.
• Content Management Systems: Storing and retrieving large amounts of unstructured content like
media files, product descriptions, or user-generated content.
FUNDAMENTALS OF SCALABLE COMPUTING
DynamoDB use case

Ocado - world’s largest dedicated online grocery retailer

• Using IAM policies each application can have it’s

own name-spaced AWS resources
(microservices)
• Every microservice has it’s own database.
• Decoupled how data is stored from how it’s
accessed.
• Applications as masters of their own state
FUNDAMENTALS OF SCALABLE COMPUTING
DynamoDB use case

Possible Example Use Cases for Ocado:

Product Catalog: A fast and scalable way to manage thousands (or even millions) of grocery items and
associated metadata (pricing, availability, etc.).
Customer Order Data: Storing real-time information about customer orders (e.g., what’s in the basket,
order progress, delivery status).
Personalization: DynamoDB could power personalized shopping experiences, storing customer preferences
or browsing history for recommendation engines.
DynamoDB is utilized in their architecture, for its strengths in scalability, low-latency data access, and
seamless integration with the broader AWS ecosystem.
THANK YOU

Dr. Radhika M. Hirannaiah

Department of Computer Science and Engineering
[email protected]
Fundamentals of Scalable Computing
Containers

Dr. Radhika M. Hirannaiah

Department of Computer Science and Engineering
 FUNDAMENTALS OF SCALABLE COMPUTING
Containers

▪ Linux Containers or LXC is an operating-system-level virtualization method for running multiple isolated
Linux systems (containers) which can run multiple workloads, on a control host running a single linux OS
instance
▪ LXC provides a virtual environment that has its own process and network space and does not create a full-
fledged virtual machine.
▪ LXC uses Linux kernel cgroups and namespace isolation functionality to achieve the same.
Motivation
▪ VMs support the objective of virtualizing the physical systems and allowing multiple tenants/applications to
be isolated and share the physical resources along with Access control
▪ One of the challenges as observed is, this isolation achieved by or provided by VM is expensive
▪ Traditional OSs supporting multiple application processes, but share a disk, with all the processes capable
of seeing the entire filesystem with access control built on top, and also share a network.
▪ Containers using a light weight mechanism, provide this virtualization extending the isolation provided by
our traditional OS Lxc (tools) is an user space toolset for creating & managing Linux Containers – different from LXC Linux containers
FUNDAMENTALS OF SCALABLE COMPUTING
What is Containerization?

• Containerization is the packaging of software code with just the operating system (OS) libraries
and dependencies required to run the code to create a single lightweight executable—called a
container that runs consistently on any infrastructure.
• More portable and resource-efficient than virtual machines (VMs), containers have become the de
facto compute units of modern cloud-native applications.
• Containerization allows developers to create and deploy applications faster and more securely.
• Containers are often referred to as “lightweight,” meaning they share the machine’s operating system
kernel and do not require the overhead of associating an operating system within each application.
• Containers are inherently smaller in capacity than a VM and require less start-up time, allowing far
more containers to run on the same compute capacity as a single VM.
FUNDAMENTALS OF SCALABLE COMPUTING
Container Characteristics
▪ Containers sit on top of a physical server and its host OS.
▪ Each container shares the host OS kernel and, usually, the binaries
and libraries too, but as read-only. This reduces the management
overhead where a single OS needs to be maintained for bug fixes,
patches, and so on.
▪ Containers are thus exceptionally “light”—they are only megabytes
in size and take just seconds to start, versus gigabytes and minutes
for a VM.
▪ Container creation is similar to process creation and it has
speed, agility and portability.
▪ Thus containers have higher provisioning performance
FUNDAMENTALS OF SCALABLE COMPUTING
Benefits of Containerization

• Portability: A container creates an executable package of software that is abstracted away from the
host operating system, and hence, is portable and able to run uniformly and consistently across any
platform or cloud.
• Agility: The open source Docker Engine for running containers started the industry standard for
containers with simple developer tools and a universal packaging approach that works on both Linux
and Windows operating systems.
• Speed: Containers are often referred to as “lightweight,” meaning they share the machine’s
operating system (OS) kernel and are not bogged down with this extra overhead
• Fault isolation: Each containerized application is isolated and operates independently of others. The
failure of one container does not affect the continued operation of any other containers
FUNDAMENTALS OF SCALABLE COMPUTING
Benefits of Containerization

• Security: The isolation of applications as containers inherently prevents the invasion of

malicious code from affecting other containers or the host system.
• Ease of management: A container orchestration platform automates the installation,
scaling, and management of containerized workloads and services. Container orchestration
platforms can ease management tasks such as scaling containerized apps, rolling out new
versions of apps, and providing monitoring, logging and debugging, among other functions.
FUNDAMENTALS OF SCALABLE COMPUTING
VM vs. Containers

• Virtual machines (VMs) are an abstraction of • Containers are an abstraction at the app layer
physical hardware turning one server into that packages code and dependencies together.
many servers. • Multiple containers can run on the same
• The hypervisor allows multiple VMs to run on machine and share the OS kernel with other
a single machine. containers, each running as isolated processes in
• Each VM includes a full copy of an operating user space.
system, the application, necessary binaries
and libraries - taking up tens of GBs.
• VMs can also be slow to boot
 FUNDAMENTALS OF SCALABLE COMPUTING
VM vs. Containers

1. Each VM includes a separate OS image, which adds overhead in memory and storage footprint.
Containers reduce management overhead as they share a common OS, only a single OS needs to be
maintained for bug fixes, patches, and so on.
2. In terms of performance, VMs have to boot when provisioned making it slower, and also have I/O
performance overhead
Containers have higher performance as its creation is similar to process creation, so boots quickly and it
has speed, agility and portability
3. VMs are more flexible as Hardware is virtualized to run multiple OS instances.
Containers run on a single OS and also can support only Ubuntu containers of that type of OS where
its running or containers cannot be of different OS variants
4. VMs consume more resources and come up slower than Containers which come up more quickly
and consume fewer resources
 FUNDAMENTALS OF SCALABLE COMPUTING
Docker
• Docker is a set of PaaS products that use OS-level virtualization to deliver software in packages called containers.
• Docker isn’t a programming language, and it isn’t a framework for building software.
• Docker is a tool that helps solve common problems such as installing, removing, upgrading, distributing, trusting, and running software.
• Docker is an open source project for building, shipping, and running programs
• Docker containers allow us to separate the applications from the infrastructure enabling faster deployment of applications/software
• It significantly reduces the time between writing code and running it in production by providing methodologies for shipping, testing and
deploying code quickly. Docker provides the isolation and security to allow many containers to run on a single server or virtual machine.
• Docker allows these containers to be shared in a way that it would work identically. Its typical to find between 8 -18 containers running
simultaneously on a single server/VM.
• Docker can be used with network applications such as web servers, databases, and mail servers, and with terminal applications including text
editors, compilers, network analysis tools, and scripts.
• In some cases, it’s even used to run GUI applications such as web browsers and productivity software.
• Docker runs Linux software on most systems.
• Docker for Mac and Docker for Windows integrate with common virtual machine (VM) technology to create portability with Windows
and macOS.
• Docker can run native Windows applications on modern Windows server machines.
FUNDAMENTALS OF SCALABLE COMPUTING
Docker Benefits

Portability, Shipping Applications Portability

Developers use Version Control Systems (VCS) like Git. DevOps also uses VCS for docs, scripts and Dockerfiles.
DevOps could use Dockerfile to describes how to build the image, and something like docker-compose.yml to describe how to orchestrate them.
FUNDAMENTALS OF SCALABLE COMPUTING
Docker Architecture

▪ Docker uses a client-server architecture.

▪ The Docker client talks to the Docker daemon, which does the heavy lifting of building, running, and
distributing your Docker containers.
▪ The Docker client and daemon can run on
the same system, or you can connect a
Docker client to a remote Docker daemon.
▪ Docker client and daemon communicate
using a REST API, over UNIX sockets or a
network interface.
▪ Another Docker client is Docker Compose,
that lets you work with applications consisting
of a set of containers.

Reference:https://docs.docker.com/get-started/overview/
FUNDAMENTALS OF SCALABLE COMPUTING
Docker Architecture

The Docker daemon

▪ The Docker daemon (dockerd) listens for Docker API
requests and manages Docker objects such as images,
containers, networks, and volumes.
▪ A daemon can also communicate with other daemons to
manage Docker services.
The Docker client
▪ The Docker client (docker) is the primary way that many
Docker users interact with Docker.
▪ When you use commands such as docker run, the client sends these commands
to ’dockerd’ (docker daemon), which carries them out.
▪ The docker command uses the Docker API.
▪ The Docker client can communicate with more than one daemon.
FUNDAMENTALS OF SCALABLE COMPUTING
Docker Architecture

Docker Host
▪ Docker Host has the Docker Daemon running and can host (like a private hub/registry) or connect
(like to a public docker_hub/registry) to a Docker Registry which stores the Docker Images.
▪ The Docker Daemon running within Docker Host is responsible for the Docker Objects images and
containers.
Docker Objects :
There are objects like
▪ Images
▪ Containers
▪ Networks
▪ Volumes
▪ Plugins and other objects which are created and used while using docker.
FUNDAMENTALS OF SCALABLE COMPUTING
Docker Objects : Images

Images : This is a read-only template with instructions for creating a Docker container. This could be
based on another image (available in the registry) with additional customizations. Eg. Image for a
Webserver .. Original image of Ubuntu customized with installation and configuration of the
webserver which is created only as a read-only image which can be deployed and an application can
be run in the same.
This is done using a Dockerfile a script file which defines the syntax to indicate steps needed to
create the image (and run using a run command).
Each instruction in a Dockerfile creates a layer in the image.
These Dockerfiles are distributed along with software that the author wants to be put into an image.
In this case, you’re not technically installing an image. Instead, you’re following instructions to build
an image.
FUNDAMENTALS OF SCALABLE COMPUTING
Docker Objects : Images – More on Dockerfile
▪ Distributing a Dockerfile is similar to distributing image files using your own distribution mechanisms. A
common pattern is to distribute a Dockerfile with software from common version-control systems like Git.
• If you have Git installed, you can try this by running an example from a public repository:
o git clone https://github.com/dockerinaction/ch3_dockerfile.git
o docker build -t dia_ch3/dockerfile:latest ch3_dockerfile
In this example, you copy the project from a public source repository onto your computer and then build and
install a Docker image by using the Dockerfile included with that project. The value provided to the -t option
of docker build is the repository where you want to install the image.
▪ Building images from Dockerfile is a light way to move projects around that fits into existing workflows. Could
lead to an unfavorable experience in case of a drift in dependencies between the time when the Dockerfile
was authored and when an image is built on a user’s computer.
▪ Docker Images can be removed or cleaned up with
o docker rmi dia_ch3/dockerfile
o rm -rf ch3_dockerfile Docker Commands – Video Ref: https://www.youtube.com/watch?v=nXV6qihj5uw
FUNDAMENTALS OF SCALABLE COMPUTING
Docker Objects : Images - layers

▪ Specification for a Docker Image is stored in Dockerfile

• Should be only one for a container
• Only the definition of the image
▪ Image is built from Dockerfile
▪ Specifies the read-only file systems in which various programs
are installed E.g. web server + libraries
▪ Each instruction in a Dockerfile creates a layer in the image.
▪ A layer is set of files and file metadata that is packaged and
distributed as an atomic unit.
• Internally, Docker treats each layer like an image, and layers are often called intermediate images.
• You can even promote a layer to an image by tagging it.
• Most layers build upon a parent layer by applying filesystem changes to the parent.
• Whenever there is a change in the Dockerfile and the image is rebuilt, only the incremental changes are
rebuilt (making it light weight, small and fast when compared to other virtualized technologies)
FUNDAMENTALS OF SCALABLE COMPUTING
Docker Objects : Images & Docker Registries
▪ Image + temporary R/W file system
• Used as temporary storage
• Deleted when container is destroyed
▪ Multiple containers can use the same image & their own
temporary storage
Docker registries:
• A Docker registry stores Docker images.
• Docker Hub is a public registry that anyone can use, and
Docker is configured to look for images on Docker Hub by
default.
• You can even run your own private registry.
• When you use the docker pull or docker run commands, the
required images are pulled from your configured registry.
• When you use the docker push command, your image is
pushed to your configured registry.
https://docs.docker.com/storage/storagedriver/
FUNDAMENTALS OF SCALABLE COMPUTING
Docker Objects : Containers

▪ Docker Containers are the ready applications created from Docker Images.
Or you can say they are running instances of the Images and they hold the
entire package needed to run the application Or it could also be looked at
as a runnable instance of an image Or an execution environment (sandbox).
▪ We can create, start, stop, move, or delete a container using the Docker API or CLI.
▪ A container can be connected to one or more networks. Storage could be attached to it, or even new image
could be created based on its current state.
▪ A container is relatively well isolated from other containers and its host machine and this isolation can also
be controlled. Processes in the container cannot access non-shared objects of other containers, and can only
access a subset of the objects (like files) on the physical machine.
▪ Docker creates a set of name spaces when a container is created. These namespaces restrict what the objects
processes in a container can see e.g. a subset of files
▪ A container is defined by its image and the configuration options provided to it when its created or started.
When a container is removed, any changes to its state that are not stored in persistent storage will disappear
 FUNDAMENTALS OF SCALABLE COMPUTING
Docker running three containers on a basic Linux computer system

▪ Running Docker means running two

programs in user space.
• The first is the Docker engine that
should always be running.
• The second is the Docker CLI. This is the
Docker program that users interact
with to start, stop, or install software
▪ Each container is running as a child process
of the Docker engine, wrapped with a
container, and the delegate process is
running in its own memory subspace of the
user space.
• Programs running inside a container Docker can be looked at as a set of PaaS products that
use OS-level virtualization to deliver software in
can access only their own memory and
packages called containers.
resources as scoped by the container.
FUNDAMENTALS OF SCALABLE COMPUTING
Running a program with Docker

▪ What happens after running

docker run
▪ The image itself is a collection
of files and metadata which
includes the specific program
to execute and other relevant
configuration details

▪ Running docker run a

second time.
▪ The image is already
installed, so Docker can start
the new container right
away.
FUNDAMENTALS OF SCALABLE COMPUTING
Dockerfile

• A Dockerfile is a script that describes steps for Docker to take to build a new image.
• These files are distributed along with software that the author wants to be put into an image. In this case, you’re not
technically installing an image. Instead, you’re following instructions to build an image.
• Distributing a Dockerfile is similar to distributing image files using your own distribution mechanisms. A common
pattern is to distribute a Dockerfile with software from common version-control systems like Git or Mercurial.
• If you have Git installed, you can try this by running an example from a public repository:
• git clone https://github.com/dockerinaction/ch3_dockerfile.git
• docker build -t dia_ch3/dockerfile:latest ch3_dockerfile
• In this example, you copy the project from a public source repository onto your computer and then build and install a
Docker image by using the Dockerfile included with that project. The value provided to the -t option of docker build is
the repository where you want to install the image.
FUNDAMENTALS OF SCALABLE COMPUTING
Docker namespaces and cgroups
• Namespaces are a feature in the Linux Kernel and fundamental aspect of
containers on Linux.
• Namespaces provide a layer of isolation. Docker uses namespaces of
various kinds to provide the isolation that containers need in order to
remain portable and refrain from affecting the remainder of the host
system. This isolation allows containers to run independently, even though
they share the same operating system kernel.
• Each aspect of a container runs in a separate namespace and its access is
limited to that namespace.
• Cgroups provide resource limitation and reporting capability within the • Common control groups
• CPU
container space. They allow granular control over what host resources are • Memory
• Network Bandwidth
allocated to the containers and when they are allocated. • Disk
• Priority
 FUNDAMENTALS OF SCALABLE COMPUTING
Docker used Namespaces
▪ Docker builds containers at runtime using 10 major system features. Docker commands can be
used to illustrate and modify features to suit the needs of the contained software and to fit the
environment where the container will run.
▪ The specific features are as follows (a few of them are discussed in a little more detail):
• PID namespace—Process isolation through identifiers (PID number) – not aware of what
happens outside its own processes
• UTS namespace—allows for having multiple hostnames on a single physical host - Host and
domain name (as other things like IP can change)
• MNT namespace—isolate a set of mount points such that processes in different namespaces
cannot view each others files. (almost like chroot) (Filesystem access & structure)
• IPC namespace—provides isolation to container process communication over shared memory
and having an ability to invite other processes to read from the shared memory
https://www.redhat.com/sysadmin/7-linux-namespaces
FUNDAMENTALS OF SCALABLE COMPUTING
Docker used Namespaces (Cont.)

• NET namespace—allow processes inside each namespace instance to have access to a new
IP address along with the full range of ports - Network access and structure
• USR namespace—provides user name-UID mapping isolating changes in names from the
metadata within a container
• chroot syscall—Controls the location of the filesystem root
• cgroups—Controlling and accounting resources –rather than a hierarchical cgroup creation,
there is a new cgroup with a root directory created to isolate resources and not allow
navigation
• CAP drop—Operating system feature restrictions
• Security modules—Mandatory access controls

https://www.redhat.com/sysadmin/7-linux-namespaces
FUNDAMENTALS OF SCALABLE COMPUTING
Cgroups (or Control groups)

• Cgroup is a linux feature to limit, police, and account the resource usage for a set of processes.
Docker uses cgroups to limit the system resources.
• Cgroups - provides mechanisms (fine grain control) to allocate, monitor and limit resources such
as CPU time, system memory, block IO or disk bandwidth, network bandwidth, or combinations
of these resources — among user-defined groups of tasks (processes) running on a system.
• Cgroups works on resource types. So it works by dividing resources into groups and then assigning
tasks to those groups, deny access to certain resources, and even reconfigure our cgroups
dynamically on a running system.
• When you install Docker binary on a linux box like ubuntu it will install cgroup related packages
and create subsystem directories.
• Hardware resources can be appropriately divided up among tasks and users, increasing overall
efficiency.
FUNDAMENTALS OF SCALABLE COMPUTING
VM vs Docker

More resource usage Less resource usage

 FUNDAMENTALS OF SCALABLE COMPUTING
Container Software

1. Docker
2. AWS Fargate
3. Google Kubernetes Engine
4. Amazon ECS
5. LXC
6. Microsoft Azure
7. Google Cloud Platform
8. Core OS
THANK YOU

Dr. Radhika M. Hirannaiah

Department of Computer Science and Engineering
[email protected]
FUNDAMENTALS OF SCALABLE COMPUTING
More on Docker

• Running Docker means running two programs in user space.

• The first is the Docker engine that should always be
running.
• The second is the Docker CLI. This is the Docker program
that users interact with to start, stop, or install software
• Each container is running as a child process of the Docker
engine, wrapped with a container, and the delegate process
is running in its own memory subspace of the user space.
• Programs running inside a container can access only
their own memory and resources as scoped by the

container.
FUNDAMENTALS OF SCALABLE COMPUTING
Running a program with Docker

What happens after running docker run

• The image itself is a collection of files
and metadata which includes the
specific program to execute and other
relevant configuration details

Running docker run a second time.

• The image is already installed,
so Docker can start the new
container right away.
FUNDAMENTALS OF SCALABLE COMPUTING
Docker images

• A layer is set of files and file metadata that is packaged and

distributed as an atomic unit.
• Internally, Docker treats each layer like an image, and layers are
often called intermediate images.
• You can even promote a layer to an image by tagging it.
• Most layers build upon a parent layer by applying filesystem
changes to the parent.
FUNDAMENTALS OF SCALABLE COMPUTING
Containers (Cont.)

A basic computer stack

running two programs
that were started from
the command line

• Notice that the command-line interface, or CLI, runs in what is called user space
memory, just like other programs that run on top of the operating system.
• Ideally, programs running in user space can’t modify kernel space memory.
• Broadly speaking, the operating system is the interface between all user programs
and the hardware that the computer is running on.
Fundamentals of Scalable Computing
Distributed systems and Trends - DevOps

Dr. Radhika M. Hirannaiah

Department of Computer Science and Engineering
FUNDAMENTALS OF SCALABLE COMPUTING
The dawn of DevOps

• Before DevOps, developing and operating software were essentially two separate jobs, performed by two
different groups of people.
• Developers wrote software, and they passed it on to operations staff, who ran and maintained the software in
production
• Software development was a very specialist job, and so was computer operation, and there was very little
overlap between these two roles
• The two departments had quite different goals and incentives, which often conflicted with each other.
• Developers tended to focus on shipping new features quickly, while operations teams cared mostly about
making services stable and reliable over the long term
• The origins of the DevOps movement lie in attempts to bring these two groups together: to collaborate, to
share understanding, to share responsibility for systems reliability and software correctness, and to improve the
scalability of both the soft- ware systems and the teams of people who build them.
FUNDAMENTALS OF SCALABLE COMPUTING
What does DevOps mean?

• It’s nothing more than a modern label for existing good practice in software development which
fosters greater collaboration between development and operations
• Four key pillars of DevOps - culture, automation, measurement, and sharing (CAMS).
• Organizations practicing DevOps have a culture that embraces collaboration, rejects siloing
knowledge between teams, and comes up with ways of measuring how they can be constantly
improving
• Another way to break it down is called the DevOps trinity: people and culture, process and
practice, and tools and technology.
• The most important thing to understand about DevOps is that it is primarily an organizational,
human issue, not a technical one
 FUNDAMENTALS OF SCALABLE COMPUTING
Definition

§ DevOps is a set of practices that combines software development and IT operations

(development and operations) where each of them are independent specialist jobs.
§ It aims to shorten the system development life cycle (SDLC) and increase an organizational
ability to deliver (Deploy and Support) applications and services
§ DevOps is complementary with Agile software development; several DevOps aspects came
from Agile methodology
FUNDAMENTALS OF SCALABLE COMPUTING
DevOps Definition (Cont.)

§ DevOps is a set of practices that automates the processes between software

development and IT teams, in order that they can build, test, and release software faster
and more reliably.

Operations Org
Development Org

Have quite different goals

and incentives, which often
conflicted with each other.

§ It follows Plan, Build, Deploy, Operate cycle.

§ Docker and Kubernetes are 2 of 10 most used DevOps tools now. Git is another one.
CLOUD COMPUTING
SDLC – Software Development Life Cycle

• Customer generates requirements

• Do a design
• Different teams work on different components
• Integrated together
• Then tested for
• Functionality
• Similarity to customer deployment
• Then for deployment, need to talk to customers IT team.
• Manages deployment, scaling, run time issues with the application
• Software Development Life Cycle (SDLC) is a framework that defines tasks performed at
each step in the development process. From inception to completion, it details out; how
to plan, build, and maintain specific software. SDLC is a crucial consideration before the
actual software development process.
FUNDAMENTALS OF SCALABLE COMPUTING
SDLC

DEV OPS
FUNDAMENTALS OF SCALABLE COMPUTING
Make the balance between Dev & Ops

In pre-cloud era, requires a lot of manual intervention like procuring machines, etc.

With cloud – automation is possible

FUNDAMENTALS OF SCALABLE COMPUTING
Advantages

§ Faster Time to Market – checkins more often, so can build and test more often.
§ Quality of Code/Release - improved because of frequent testing
§ Integration Often
§ Deploy Often – changes reach the customers often
§ Automated – Repeatable and faster
§ Every one is happy (Hopefully!!)
FUNDAMENTALS OF SCALABLE COMPUTING
Automation : Principles of Continuous Delivery

Principle #1 Principle #3
Every build is a potential release Automate wherever possible

Principle #4
Principle #2 Have automated tests you can trust
Eliminate manual bottlenecks
FUNDAMENTALS OF SCALABLE COMPUTING
Keep it Simple

• Reduce risk of Release

• Make small changes and Test it every cycle
• Small changes can make the difference for Ops & Dev
FUNDAMENTALS OF SCALABLE COMPUTING
Automate all the things

§ Programming Model to enable scale

§ Manage source code
§ Reproducible Build
§ Build on a Prod-Like environment
§ No more “Works on my machine”
§ Test
§ Testing reduces risks
§ Make you more confident
§ Deploy
§ Deploy to Dev
§ Deploy to QA
§ Deploy to Pre-Prod
§ Prod
 FUNDAMENTALS OF SCALABLE COMPUTING
DevOps Pipeline

Typical pipeline for containerized applications might look like the following:
1. A developer pushes his/her code changes to Git.
2. The build system automatically builds the current version of the code and runs some sanity tests.
3. If all tests pass, the container image will be published into the central container registry.
4. The newly built container is deployed automatically to a staging environment.
5. The staging environment undergoes some automated acceptance tests.
6. The verified container image is deployed to production.
FUNDAMENTALS OF SCALABLE COMPUTING
Continuous Integration

Continuous Integration (CI) is the practice of routinely integrating code changes into the main branch of a repository, and
testing the changes, as early and often as possible. Ideally, developers will integrate their code daily, if not multiple times a
day.
Benefits of CI:
• Shorter integration cycles
• Better visibility of what others are doing leading to greater communication
• Issues are caught and resolved early
• Better use of time for development rather than debugging
• Early knowledge that your code works along with others' changes
• Ultimately, enabling the team to release software faster
FUNDAMENTALS OF SCALABLE COMPUTING
Continuous Development

§ This includes continuous code development, continuous integration and continuous Build
§ Developers distributed across geographies
§ Consistent coding style
§ Need to keep the changes in sync
§ Can’t rely on mailing individual changes
§ Everyone needs to have a common view of the code.
§ Continuous integration (CI): the automatic merging of the code, Requires every module to compile
(Build) correctly and then do sanity testing of developers’ changes against the mainline branch.
§ If you’re making changes on a branch that would break the build when merged to the mainline,
continuous integration will let you know that right away, rather than waiting until you finish your
branch and do the final merge.
§ Tool : Source Code Version Management Systems like git, ClearCase, Maven, ANT, Jenkins, Travis CI,
or Drone are often used to automate build pipelines
FUNDAMENTALS OF SCALABLE COMPUTING
Continuous Delivery / Continuous Testing / Continuous Deployment

Continuous delivery is the frequent shipping of sanitized builds to a given environment (such as test or
production) for automatic developments.
§ This could be done using tools like Jenkins the open source Java based application which automates
many parts of the software delivery pipeline
Continuous Testing
§ Continuous testing is the process of executing predominantly automated tests as part of the
software delivery pipeline for validating the code to ensure that the application works as envisaged
and is free of bugs or defects and can be continuously deployed
§ These tests are typically designed to be executed with minimum wait times and provide the earliest
possible feedback & support detection (or prevention) of the risks that are most critical for the
business or organization that is associated with the software release candidate.
(Cont.)
FUNDAMENTALS OF SCALABLE COMPUTING
Continuous Delivery / Continuous Testing / Continuous Deployment

Continuous testing (Cont.)

This would involve activities like
§ Copy individual binaries to staging environment to the right directories
§ Configure it correctly, ensure dependent libraries are installed
§ Execute different categories of the planned tests (e.g. Integration tests, functionality, performance
or acceptance tests) (Could use tools like Junit, Selenium)
§ Generate reports
§ Continuous deployment (CD) is the automatic deployment of successful builds to production
managed centrally.
§ Developers should be able to deploy new versions by either pushing a button, or merging a merge
request, or pushing a Git release tag.
 FUNDAMENTALS OF SCALABLE COMPUTING
Continuous Deployment (Cont.)

§ This is when the application is actually used

by the customer Setup the environment
§ Specify the type of infrastructure that you
need in terms of machine types – Chef
Provisioning (Machines, storage)
§ Consider Immutable infrastructure which
does not change enabling easier automation
§ Specify the configuration that you need –
Puppet, Chef Recipe/Cookbooks
§ The set of docker images and their scaling
policy, setup load balancer etc. Jenkins: Is the
workflow automation
Tools
server
Tools to setup the infrastructure: Chef/Puppet/Kubernetes
Docker – create a docker image with all dependencies for each of the services
FUNDAMENTALS OF SCALABLE COMPUTING
Continuous Delivery/Deployment

• Continuous Delivery (CD) ensures that every code change is tested and ready for the production
environment, after a successful build.
• CI ensures every code is committed to the main code repository whereas CD ensures the system
is in an executable state at all times, after changes to code
• The primary goal of Continuous Delivery is to make software deployments painless, low-risk
events that can be performed at any time, on demand
• The actual process of automatic deployment in the target customer environment comes under
Continuous Deployment (CD)
• Continuous Delivery can work only if Continuous Integration is in place
FUNDAMENTALS OF SCALABLE COMPUTING
DevOps Tools
FUNDAMENTALS OF SCALABLE COMPUTING
Jenkins

• Jenkins is a very widely adopted CD tool. There is also a newer dedicated side project for running Jenkins in
Kubernetes cluster, JenkinsX.
• Jenkins is a self-contained, open source automation server which can be used to automate all sorts of tasks
related to building, testing, and delivering or deploying software.
• Jenkins can be installed through native system packages, Docker, or even run standalone by any machine with
a Java Runtime Environment (JRE) installed.
• Jenkins is a highly extensible product whose functionality can be extended through the installation of plugins.
• There are a vast array of plugins available to Jenkins.
• Plugins are the primary means of enhancing the functionality of a Jenkins environment to suit organization- or
user-specific needs.
• There are over a thousand different plugins which can be installed on a Jenkins controller and to integrate
various build tools, cloud providers, analysis tools, and much more.
FUNDAMENTALS OF SCALABLE COMPUTING
CI/CD Tools
FUNDAMENTALS OF SCALABLE COMPUTING
DevOps Metrics

• Metrics give insights into what's happening at all stages of the DevOps pipeline, from design to development to
deployment.
• Metrics are objective measures. They strengthen the feedback loops that are essential to DevOps.
• Collecting metrics and displaying them via dashboards or scorecards should be automated.
• It's important to map these metrics to business needs.
FUNDAMENTALS OF SCALABLE COMPUTING
Cloud Native applications

• The term cloud native has become an increasingly popular shorthand way of talking about modern
applications and services that take advantage of the cloud, containers, and orchestration, often
based on open source software

Characteristics of cloud native applications:

• Automatable - applications are to be deployed and managed by machines
• Ubiquitous and flexible - decoupled from physical resources so that they can be moved easily from one node
to another
• Resilient and scalable - highly available through redundancy and graceful degradation
• Dynamic - perform rolling updates to smoothly upgrade services without ever dropping traffic.
• Observable - monitoring, logging, tracing of metrics
• Distributed - composed of multiple, cooperating, distributed microservices
FUNDAMENTALS OF SCALABLE COMPUTING
Cloud Native applications

References:
https://www.guru99.com/jenkins-
tutorial.html

https://www.guru99.com/devops-
tutorial.html

References: (useful for Cloud lab

experiment)
https://www.jenkins.io/doc/
https://www.jenkins.io/doc/book/pipeli
ne/
https://jenkins-x.io/
THANK YOU

Dr. Radhika M. Hirannaiah

Department of Computer Science and Engineering
[email protected]
Fundamentals of Scalable Computing
Distributed systems and Trends
Orchestration and Kubernetes

Dr. Radhika M. Hirannaiah

Department of Computer Science and Engineering
FUNDAMENTALS OF SCALABLE COMPUTING
Container orchestration
• Container orchestration is a process that automates the
deployment, management, scaling, networking, and availability
of container-based applications.
• It also includes the process of deploying containers on a
compute cluster consisting of multiple nodes.
• It is all about managing the lifecycles of containers, especially
in large, dynamic environments.
• Container orchestration can be used in any environment where
you use containers, to help deploy the same application across
different environments without needing to redesign it.
• Orchestration tools extend lifecycle management capabilities to
complex, multi-container workloads deployed on a cluster of
machines. https://devopedia.org/container-orchestration
FUNDAMENTALS OF SCALABLE COMPUTING
Container orchestration

• Container orchestrator: a piece of software designed to join together many different machines into
a cluster: a kind of unified compute substrate, which appears to the user as a single very powerful
computer on which containers can run.
• Operations teams, too, find their workload greatly simplified by containers.
• Instead of having to maintain a sprawling estate of machines of various kinds, architectures, and
operating systems, all they have to do is run a container orchestrator

Container orchestration is a process that automates the deployment, management, scaling,

networking, and availability of container-based applications.
FUNDAMENTALS OF SCALABLE COMPUTING
Container orchestration

• The terms orchestration and scheduling are often used

loosely as synonyms.
• Strictly speaking, though, orchestration in this context
means coordinating and sequencing different activities
in service of a common goal (like the musicians in an
orchestra).
• Scheduling means managing the resources available and
assigning workloads where they can most efficiently be
run.
FUNDAMENTALS OF SCALABLE COMPUTING
Container orchestration (Cont.)

• A third important activity is cluster management: joining multiple physical or virtual servers into a
unified, reliable, fault-tolerant, apparently seamless group.
• The term container orchestrator usually refers to a single service that takes care of scheduling,
orchestration and cluster management.
• Containers providing this computation environment is considered Immutable
• Kubernetes is the most pervasive container orchestration platform to address these challenges.
• Kubernetes is an open-source container management (orchestration) tool.
• It’s container management responsibilities include container deployment, scaling & descaling
of containers & container load balancing.
• It lets you manage complex application deployments quickly in a predictable and reliable
manner.
FUNDAMENTALS OF SCALABLE COMPUTING
Container orchestration (Cont.)

• Container orchestration is used to control and automate many tasks:

• Provisioning and deployment of containers
• Redundancy and availability of containers
• Scaling up or removing containers to spread application load evenly across host infrastructure
• Movement of containers from one host to another if there is a shortage of resources in a host, or if a
host dies
• Allocation of resources between containers
• External exposure of services running in a container with the outside world
• Load balancing of service discovery between containers
• Health monitoring of containers and hosts
• Configuration of an application in relation to the containers running it
FUNDAMENTALS OF SCALABLE COMPUTING
Where does Container orchestration fit within the system stack?

Container Orchestration mediates between the apps/services and the container runtimes
Functional Aspects
• Service Management: Labels (represents
Key-Value pairs that are attached to objects
such as pods and used to specify identifying
attributes of objects), groups (collection of
related functionality), namespaces,
dependencies, load balancing, readiness
checks.
• Scheduling: Allocation, replication,
(container) resurrection, rescheduling, rolling
deployment, upgrades, downgrades.
• Resource Management: Memory, CPU, GPU,
volumes, ports, IPs.
FUNDAMENTALS OF SCALABLE COMPUTING
Container orchestration – Tools
1. Docker Swarm: Provides native clustering functionality for Docker
containers, which turns a group of Docker engines into a single, virtual
Docker engine.
2. Google Container Engine: Google Container Engine, built on Kubernetes,
can run Docker containers on the Google Cloud.
3. Kubernetes: An orchestration system for Docker containers. It handles
scheduling and manages workloads based on user-defined parameters.
4. Amazon ECS: The ECS supports Docker containers and lets you run
applications on a managed cluster of Amazon EC2 instances.
5. Azure Container Service (ACS): ACS lets you create a cluster of
virtual machines that act as container hosts along with master
machines that are used to manage your application containers.
6. Cloud Foundry’s Diego: Container management system that
combines a scheduler, runner, and health manager.
FUNDAMENTALS OF SCALABLE COMPUTING
Kubernetes Basics (K8)
• Containerisation has brought a lot of flexibility for developers in terms of managing the
deployment of the applications.
• More granular application consists of more components and hence requires some sort of
management for those.
• Kubernetes as seen earlier is a container or microservice platform that orchestrates
computing, networking, and storage infrastructure workloads. Kubernetes extends how we
scale containerized applications so that we can enjoy all the benefits of a truly immutable
infrastructure
• Need a solution to
• take care of scheduling the deployment of a certain number of containers to a specific
node
• manage networking between the containers
• follow the resource allocation http://kubernetes.io/docs/concepts/overview/what-is-kubernetes/
• move containers around as they grow and much more.
 FUNDAMENTALS OF SCALABLE COMPUTING
Kubernetes Basics (Cont.)
1. Basic Building objects
• Replication of components ▪ Pod. A group of one or more containers.

• Auto-scaling ▪ Service. An abstraction that defines a logical set of pods as well as the policy and an endpoint for
accessing them.
• Load balancing ▪ Volume. An abstraction that lets us persist data. (This is necessary because containers are
ephemeral—meaning data is deleted when the container is deleted.)
• Rolling updates ▪ Namespace. A segment of the cluster dedicated to a certain purpose, for example a certain project or
team of devs. Typically used to create multiple virtual Kubernetes clusters within a single cluster
• Logging across components 2. Controllers which are several higher-level abstractions. Basic K8S objects include -
▪ ReplicaSet (RS). Ensures the desired amount of pod is what’s running.
• Monitoring and health checking ▪ Deployment. Offers declarative updates for pods in an RS.
• Service discovery ▪ StatefulSet. A workload API object that manages stateful applications, such as databases.
▪ DaemonSet. Ensures that all or some worker nodes run a copy of a pod. This is useful for daemon
• Authentication applications like Fluentd.
▪ Job. Creates one or more pods, runs a certain task(s) to completion, then deletes the pod(s)

Kubernetes does the things that the very best system administrator would do: automation, failover,
centralized logging, monitoring. It takes what we’ve learned in the DevOps community and makes it the
default, out of the box.
- Kelsey Hightower, Staff Developer Advocate, Google Cloud Platform at Google .
FUNDAMENTALS OF SCALABLE COMPUTING
Kubernetes Architecture

Pod: container
or group of
containers

kubelet: get pod

controller-mgr: config and invoke
run controllers, Docker to run
e.g., containers
namespace
manager
FUNDAMENTALS OF SCALABLE COMPUTING
Kubernetes Architecture

Kubernetes targets the management of elastic applications that consist of multiple microservices
communicating with each other. Often those microservices are tightly coupled forming a group of
containers that would typically, in a non-containerized setup run together on one server. This group, the
smallest unit that can be scheduled to be deployed through K8s is called a pod.

The master node is responsible for the management of Kubernetes cluster. This is the entry point of all
administrative tasks. The master node is the one taking care of orchestrating the worker nodes, where the
actual services are running.
 FUNDAMENTALS OF SCALABLE COMPUTING
Kubernetes Building blocks - Pod
• Kubernetes targets the management of elastic applications that consist of multiple microservices communicating with
each other.
• Often those microservices are tightly coupled forming a group of containers that would typically, in a non-containerized
setup run together on one server.
• This group, the smallest unit that can be scheduled to be deployed through K8s is called a pod. Pod is a single or group
of containers that share storage and network within a Kubernetes configuration, telling those containers how to
behave.
• Pods share storage, Linux namespaces, cgroups, IP addresses and communicate with each other over localhost
networking
• Each pod is assigned an IP address on which it can be accessed by other pods within a cluster. Applications within a pod
have access to shared volumes – helpful for when you need data to persist beyond the lifetime of a pod.
• They are co-located, hence share resources and are always scheduled together.
• Pods are not intended to live long. They are created, destroyed and re-created on demand, based on the state of the
server and the service itself.
 FUNDAMENTALS OF SCALABLE COMPUTING
Kubernetes Building Blocks : Pod (Cont)

▪ A Pod is scheduled or assigned to a node by the scheduler and the kubelet starts creating containers for that
Pod using a container runtime. There are three possible container states: Waiting, Running, and Terminated.
▪ Pods follow a defined lifecycle, starting in the Pending phase, moving through Running if at least one of its
primary containers starts OK, and then through either the Succeeded or Failed phases depending on
whether any container in the Pod terminated in failure.
▪ Pods are only scheduled once in their lifetime. Once a Pod is scheduled (assigned) to a Node, the Pod runs on
that Node until it stops or is terminated
▪ Whilst a Pod is running, the kubelet is able to restart containers to handle some kind of faults. Within a Pod,
Kubernetes tracks different container states and determines what action to take to make the Pod healthy
again.
▪ Volumes exists as long as that specific Pod (with that exact UID) exists unless its an persistent volume for
shared storage between the containers
▪ Pods do not, by themselves, self-heal. If a Pod is scheduled to a node that then fails, the Pod is deleted;
https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/
FUNDAMENTALS OF SCALABLE COMPUTING
Kubernetes - Service

Services is coupling of a set of pods to a policy by which to access them. Services are used to
expose containerized applications to originations from outside the cluster
▪ Pods in a Kubernetes deployment are regularly created and destroyed, causing their IP
addresses to change constantly. It will create discoverability issues for the deployed application,
making it difficult for the application frontend to identify which pods to connect.
▪ To address this K8s introduced the concept of a service, which is an abstraction on top of a
number of pods which has a Virtual IP address assigned to it which is exposed and used by the
service proxies running on top of it.
▪ Other services communicate with the service via this Virtual IP address, and the service keeps
track of the changes in IP addresses and DNS names of the pods and map them to this virtual IP.
▪ A Kubernetes service is a logical collection of pods in a Kubernetes cluster.
▪ This is where we can configure load balancing for our numerous pods and expose them via a
service.
FUNDAMENTALS OF SCALABLE COMPUTING
Kubernetes Components – Master Node

Master Node
API server
etcd storage
controller-manager
Scheduler
FUNDAMENTALS OF SCALABLE COMPUTING
Kubernetes Components – Worker Node

Docker
Kubelet
kube-proxy
Kubectl
FUNDAMENTALS OF SCALABLE COMPUTING
Kubernetes Components – Master Node

Master Node
The master node is responsible for the management of Kubernetes cluster. This is the entry point of all
administrative tasks. The master node is the one taking care of orchestrating the worker nodes, where the
actual services are running.
It exposes the API, schedules deployments, and generally manages the cluster.
Let's dive into each of the components of the master node.
API server (Kube-apiserver)
The API server is the entry points for all the REST commands used to control the cluster. It processes the
REST requests, validates them, and executes the bound business logic. The result state has to be persisted
somewhere (etcd component of the master node).
FUNDAMENTALS OF SCALABLE COMPUTING
Kubernetes Components – Master Node (Cont.)

etcd storage

• etcd is a simple, distributed, consistent key-value stores all cluster data (Can be run on the same
server as a master node or on a dedicated cluster) .
• It’s mainly used for shared configuration and service discovery.
• It provides a REST API for CRUD operations as well as an interface to register watchers on specific
nodes, which enables a reliable way to notify the rest of the cluster about configuration changes.
• CRUD - Create, Read, Update, and Delete, which are four primitive database operations.
• An example of data stored by Kubernetes in etcd is jobs being scheduled, created and deployed,
pod/service details and state, namespaces and replication information, etc.
FUNDAMENTALS OF SCALABLE COMPUTING
Kubernetes Components - Master Node (Cont.)
Kube controller-manager
• Optionally we can run different kinds of controllers inside the master node.
• A controller uses API server to watch the shared state of the cluster and makes corrective changes to the current
state to change it to the desired one.
• An example of such a controller is the Replication controller, which takes care of the number of pods in the system.
The replication factor is configured by the user, and it's the controller’s responsibility to recreate a failed pod or
remove an extra-scheduled one.
• Other examples of controllers are endpoints controller, namespace controller, and service accounts controller.
Kube- Scheduler
• Has information on the resources available. Schedules new pods on worker nodes
• Deploys configured pods and services onto the nodes
• The scheduler has the information regarding resources available on the members of the cluster, as well as the ones
required for the configured service to run and hence is able to decide where to deploy a specific service.
FUNDAMENTALS OF SCALABLE COMPUTING
Kubernetes Components – Worker Node

Worker node
The pods are run here, so the worker node contains all the necessary services to manage the networking
between the containers, communicate with the master node, and assign resources to the containers
scheduled.

Docker
• Docker runs on each of the worker nodes, and runs the configured pods.
• It takes care of downloading the images and starting the containers.
FUNDAMENTALS OF SCALABLE COMPUTING
Kubernetes Components - Worker Node (Cont.)

Kubelet

• kubelet gets the configuration of a pod from the API server and ensures that
the described containers are up and running.
• This is the worker service that’s responsible for communicating with the
master node.
• It also communicates with etcd, to get information about services and write
the details about newly created ones.
 FUNDAMENTALS OF SCALABLE COMPUTING
Kubernetes Components - Worker Node (Cont.)

kube-proxy

• kube-proxy acts as a network proxy and a load balancer for a service on a single worker node.
• It takes care of the network routing for TCP and UDP packets.

Kubectl

• A command line tool to communicate with the API service and send commands to the master
node.

Container runtime - Runs containers.

FUNDAMENTALS OF SCALABLE COMPUTING
Kubernetes Functioning

• Kubernetes, you will describe the configuration of an application using say a JSON file.
The configuration file tells the configuration management tool where to find the
container images, how to establish a network, and where to store logs.
• When deploying a new container, the container management tool automatically
schedules the deployment to a cluster and finds the right host, taking into account any
defined requirements or restrictions. The orchestration tool then manages the
container’s lifecycle based on the specifications that were determined in the compose
file.
FUNDAMENTALS OF SCALABLE COMPUTING
Kubernetes Benefits

• Horizontal scaling. Scale your application as needed from command line or UI.
• Automated rollouts and rollbacks. Roll out changes in a controlled fashion while monitoring the health of
your application in the container and if something goes wrong, K8S automatically rolls back the change.
• Service discovery and load balancing. Kubernetes can expose a containers using a DNS name or IP and based
on the load, can do load balancing by distribute traffic.
• Storage orchestration. Automatically mount local or public cloud or a network storage.
• Secret and configuration management. Kubernetes lets you store and manage sensitive information, such as
passwords, OAuth tokens, and SSH keys. You can deploy and update secrets and application configuration
without rebuilding your container images, and without exposing secrets in your stack configuration.
• Self-healing. The platform heals many problems: restarting failed containers, replacing and rescheduling
containers as nodes die, killing containers that don’t respond to your user-defined health check, and waiting
to advertise containers to clients until they’re ready.
• Batch execution. Manage your batch and Continuous Integration workloads and replace failed containers.
• Automatic binpacking. Automatically schedules containers based on resource requirements like CPU and
memory which the containers need along with other constraints into the same host
 FUNDAMENTALS OF SCALABLE COMPUTING
Docker and Kubernetes – A Brief Comparison

• Docker is a containerization platform, and Kubernetes is a container orchestrator for container platforms like
Docker.
• Docker provided an open standard for packaging and distributing containerized applications while
Kubernetes will coordinate and schedule these containers , seamlessly upgrade an application without any
interruption of service, monitor the health of an application, know when something goes wrong and
seamlessly restart it.
• Docker’s own native clustering solution for Docker containers is Docker Swarm , which has the advantage of
being tightly integrated into the ecosystem of Docker, and uses its own API.
• Like most schedulers, Docker Swarm provides a way to administer a large number of containers spread
across clusters of servers.
• Its filtering and scheduling system enables the selection of optimal nodes in a cluster to deploy
containers. https://www.sumologic.com/blog/kubernetes-vs-docker/
FUNDAMENTALS OF SCALABLE COMPUTING
Docker and Kubernetes – A Brief Comparison (Cont.)

• Kubernetes is a comprehensive system for automating deployment, scheduling and scaling of

containerized applications, and supports many containerization tools such as Docker.
• Kubernetes is the market leader and the standardized means of orchestrating containers and
deploying distributed applications.
• Kubernetes can be run on a public cloud service or on-premises, is highly modular, open
source, and has a vibrant community.
• Companies of all sizes are investing into it, and many cloud providers offer Kubernetes as a
service

https://www.civo.com/blog/kubernetes-vs-docker-a-comprehensive-comparison
THANK YOU

Dr. Radhika M. Hirannaiah

Department of Computer Science and Engineering
[email protected]
Fundamentals of Scalable Computing
Distributed systems and Trends
- ZooKeeper
Dr. Radhika M. Hirannaiah
Department of Computer Science and Engineering
FUNDAMENTALS OF SCALABLE COMPUTING
Apache Zookeeper

What is Apache Zookeeper?

• ZooKeeper is an open source Apache project that provides a centralized service for providing
configuration information, naming, synchronization and group services over large clusters in distributed
systems.
• The goal is to make these systems easier to manage with improved, more reliable propagation of
changes
• ZooKeeper is a high-performance coordination service for distributed applications. It exposes common
services in a simple interface so you don't have to write them from scratch
• Interaction with ZooKeeper occurs by way of Java™ or C interface time.
FUNDAMENTALS OF SCALABLE COMPUTING
Apache Zookeeper

Why is Zookeeper needed?

• It addresses the need for a simple distributed co-ordination process,
where message-based communication can be hard to use
• It supports use of locks correctly where it is hard to use
• It provides reliability and availability by addressing single point of
failures using an ensemble of nodes
• It ensures atomicity of the distributed transactions and ensures the
applications run consistently by ensuring that the transactions either
succeed or failed completely but without transaction is partial
• It helps to maintain a standard hierarchical namespace like files and
directories
FUNDAMENTALS OF SCALABLE COMPUTING
Apache ZooKeeper: Objectives

§ Simple, Robust, Good Performance

§ High Availability, High throughput, Low Latency

§ Tuned for Read dominant workloads

§ Familiar models and interfaces

§ Wait-Free: A slow/failed client will not interfere with the requests of a fast client
FUNDAMENTALS OF SCALABLE COMPUTING
Apache ZooKeeper – What is Apache ZooKeeper
• An open source, high-performance coordination service for distributed applications having many hosts.
• It automates this process of management of the hosts and allows developers to focus on building software
features rather than worry about its distributed nature.
• It provides a Centralized coordination service that can be used by distributed applications to maintain
configuration information, perform distributed locks & synchronization and enable group services.
• It uses a shared hierarchical name space of data registers (called znode’s) to coordinate distributed processes.
• Exposes common services in simple interface:
§ Naming
§ configuration management
§ locks & synchronization
§ group services
.. developers don't have to write them from scratch ..Build your own on it for specific needs.
• For reliability, three copies of the ZooKeeper can be run so that it does not become a single point of failure.
• Apache Kafka uses ZooKeeper to manage configuration, Apache HBase uses ZooKeeper to track the status of
distributed data.
FUNDAMENTALS OF SCALABLE COMPUTING
Apache Zookeeper – How does it work?

• ZooKeeper allows distributed processes to coordinate with each other through a shared hierarchical
namespace which is organized similar to a standard file system
• The namespace consists of data registers - called znodes and these are similar to files and
directories.
• Unlike a typical file system, which is designed for storage, ZooKeeper data is kept in-memory, which
means ZooKeeper can achieve high throughput and low latency numbers.

• Like the distributed processes it coordinates, ZooKeeper itself is intended to be replicated over a set
of hosts called an ensemble.
FUNDAMENTALS OF SCALABLE COMPUTING
Apache Zookeeper – How does it work?

• The servers that make up the ZooKeeper service must all know about each other. They maintain an in-
memory image of state, along with a transaction logs and snapshots in a persistent store.
• As long as a majority of the servers are available, the ZooKeeper service will be available.
• Clients connect to a single ZooKeeper server. The client maintains a TCP connection through which it
sends requests, gets responses, gets watch events, and sends heart beats.
• If the TCP connection to the server breaks, the client will connect to a different server.
• ZooKeeper stamps each update with a number that reflects the order of all ZooKeeper transactions.
Subsequent operations can use the order to implement higher-level abstractions, such as
synchronization primitives.
FUNDAMENTALS OF SCALABLE COMPUTING
Apache ZooKeeper Features
1. Updating the Node’s Status: updates every node that allows it to store updated information about
each node across the cluster.
2. Managing the Cluster: manage the cluster in such a way that the status of each node is maintained
in real time, leaving lesser chances for errors and ambiguity.
3. Naming Service: attaches a unique identification to every node which is quite similar to the DNA
that helps identify it.
4. Automatic Failure Recovery: locks the data while modifying which helps the cluster recover it
automatically if a failure occurs in the database.
 FUNDAMENTALS OF SCALABLE COMPUTING
Apache ZooKeeper: Architecture

Server : The server sends an acknowledgement when any client connects. A leader is elected at startup.
All servers store a copy of the data tree (discussed later) in memory

Client: Client is one of the nodes in the distributed application cluster. It helps you to accesses
information from the server. Every client sends a message to the server at regular intervals that helps
the server to know that the client is alive.
In the case when there is no response from the connected server, the client automatically redirects the
message to another server.
FUNDAMENTALS OF SCALABLE COMPUTING
Apache ZooKeeper: Architecture

Leader: One of the servers is designated a Leader. It gives all the information to the clients as well as an
acknowledgment that the server is alive. It would perform automatic recovery if any of the connected
nodes failed.
Follower: Server node which follows leaders instruction is called a follower.
§ Client read requests are handled by the correspondingly connected ZooKeeper server
§ The client writes requests are handled by the ZooKeeper leader.
Ensemble/Cluster: Group of ZooKeeper servers which is called ensemble or a Cluster. You can use
ZooKeeper infrastructure in the cluster mode to have the system at the optimal value when you are
running the Apache.
ZooKeeper WebUI: If you want to work with ZooKeeper resource management, then you need to use
WebUI. It allows working with ZooKeeper using the web user interface, instead of using the command
line. It offers fast and effective communication with the ZooKeeper application.
FUNDAMENTALS OF SCALABLE COMPUTING
Apache ZooKeeper: Data Model

• The data model follows a Hierarchal namespace (like a file system) as discussed earlier
• A node is a system where the cluster runs and each Zookeeper Data Model

node is created with a “znode”

• Each znode has data (Configurations, Status, Counters, Parameters,
Location information) and may or may-not have children
• Keeps metadata information
• Does not store big datasets
• ZNode paths:
• Are Canonical, slash-separated and absolute
• Do not use any relative references

The entire ZooKeeper tree is kept in main memory

 https://www.simplilearn.com/introduction-to-zookeeper-tutorial
FUNDAMENTALS OF SCALABLE COMPUTING
Apache ZooKeeper: Types of Znodes
1. Persistent znodes
• Permanent and have to be deleted explicitly by the client.
• Will persist even after the session that created the znode is terminated.
2. Ephemeral znodes
• Temporary and Exists as long as the session that created the znode is active.
• Deleted automatically when the client session creating them ends.
• Used to detect the termination of a client.
• Alerts, known as a watch, can be set up to detect the deletion of the znode.
• These nodes can’t have children
Sequence Nodes (Unique Naming)
• Append a monotonically increasing counter to the end of path
• Applies to both persistent & ephemeral nodes
Note that when a znode is created, its type is specified and it cannot be changed later.
FUNDAMENTALS OF SCALABLE COMPUTING
Apache Zookeeper – Data model and the hierarchical namespace
• The namespace provided by ZooKeeper is much like that of a standard file system but are more like a distributed,
consistent shared memory that is hierarchically organized like a file system.
• A name is a sequence of path elements separated by a slash (/), and every znode has a parent except the root ("/"). Every
node in ZooKeeper's namespace is identified by a path
• Unlike standard file systems, each node in a ZooKeeper namespace can have data associated with it as well as children. A
znode cannot be deleted if it has any children.
• Znodes maintain a stat structure that includes version numbers for data changes, ACL changes, and timestamps, to allow
cache validations and coordinated updates.
• Each time a znode's data changes, the version number increases.
• Whenever a client retrieves data it also receives the version of the data.
• The data stored at each znode in a namespace is read and written atomically.
• Reads get all the data bytes associated with a znode and a write replaces all the data.
• Each node has an Access Control List (ACL) that restricts who can do what.
• If the version it supplies doesn't match the actual version of the data, the update will fail
• All updates made by ZooKeeper are totally ordered. It stamps each update with a sequence called zxid (ZooKeeper
Transaction Id). Each update will have a unique zxid.
FUNDAMENTALS OF SCALABLE COMPUTING
Apache Zookeeper – Data model and the hierarchical namespace

Ephemeral nodes:
• These znodes exists as long as the session that created the znode is active. When the session ends the znode is deleted.
Watches:
• ZooKeeper supports the concept of watches. Clients can set a watch on a znode.
• A watch will be triggered and removed when the znode changes.
• When a watch is triggered, the client receives a packet saying that the znode has changed.
• If the connection between the client and one of the ZooKeeper servers is broken, the client will receive a local notification.
Guarantees:
• Sequential Consistency - Updates from a client will be applied in the order that they were sent.
• Atomicity - Updates either succeed or fail. No partial results.
• Single System Image - A client will see the same view of the service regardless of the server that it connects to. i.e., a client will never
see an older view of the system even if the client fails over to a different server with the same session.
• Reliability - Once an update has been applied, it will persist from that time forward until a client overwrites the update.
• Timeliness - The clients view of the system is guaranteed to be up-to-date within a certain time bound.
FUNDAMENTALS OF SCALABLE COMPUTING
Apache ZooKeeper

Each Znode has version number, is incremented every time its data changes
• setData and delete take version as input, operation succeeds only if client’s version is equal to server’s one
FUNDAMENTALS OF SCALABLE COMPUTING
ZooKeeper Reads/Writes/Watches
ZooKeeper Service

Leader

Server Server Server Server Server

Client Client Client Client Client Client Client Client

• Reads: between the client and single server

• Writes: sent between servers first, get consensus, then respond back
• Watches: between the client and single server
• A watch on a znode basically means “monitoring it”
FUNDAMENTALS OF SCALABLE COMPUTING
Apache Zookeeper – API

• One of the design goals of ZooKeeper is providing a very simple programming interface.
• As a result, it supports only these operations:
1. create : creates a node at a location in the tree
2. delete : deletes a node
3. exists : tests if a node exists at a location
4. get data : reads the data from a node
5. set data : writes data to a node
6. get children : retrieves a list of children of a node
7. sync : waits for data to be propagated
FUNDAMENTALS OF SCALABLE COMPUTING
Apache Zookeeper – Implementation

• The replicated database is an in-memory database containing the entire

data tree.
• Updates are logged to disk for recoverability, and writes are serialized to
disk before they are applied to the in-memory database.
• Every ZooKeeper server services clients.
• Clients connect to exactly one server to submit requests.
• Read requests are serviced from the local replica of each server database.
• All write requests from clients are forwarded to a single server, called
the leader. The rest of the ZooKeeper servers, called followers, receive
message proposals from the leader and agree upon message delivery.
• The messaging layer takes care of replacing leaders on failures and syncing
followers with leaders.
 FUNDAMENTALS OF SCALABLE COMPUTING
Implementation of ZooKeeper

ZooKeeper server services clients

• Request Processor
• Atomic Broadcast
• Zab Protocol
• Replicated Database

Clients connect to exactly one server to submit requests

• read requests served from the local replica
• write requests are processed by an agreement protocol (an elected server leader initiates
processing of the write request)
FUNDAMENTALS OF SCALABLE COMPUTING
Advantages of ZooKeeper

● Simple distributed coordination process

● Synchronization − Mutual exclusion and co-operation between server processes. This process
helps in Apache HBase for configuration management.
● Ordered Messages
● Serialization − Encode the data according to specific rules. Ensure your application runs
consistently. This approach can be used in MapReduce to coordinate queue to execute running
threads.
● Reliability
● Atomicity − Data transfer either succeeds or fails completely, no transaction is partial.

https://www.tutorialspoint.com/ZooKeeper/ZooKeeper_overview.htm
FUNDAMENTALS OF SCALABLE COMPUTING
Disadvantages of ZooKeeper

● Other solutions like consul have features like service discovery baked in, with health checks in
the loop, while ZooKeeper does not.
● The ephemeral node lifecycle is tied to TCP connection; it can happen that TCP connection is
up, but the process on the other side just went out of memory or otherwise not functioning
properly
● It’s Complex
FUNDAMENTALS OF SCALABLE COMPUTING
When to use Apache Zookeeper

● Naming service - Naming Service is a specific service whose aim is to provide a consistent and uniform naming of
resources, thus allowing other programs or services to localize them and obtain the required metadata for
interacting with them
● Configuration management - store and manage configuration settings of the entire system and its components
● Data Synchronization - Data synchronization is the process of establishing consistency among data from a source
to a target data storage and vice versa and the continuous harmonization of the data over time.
● Leader election - Leader election is the process of designating a single process as the organizer of some task
distributed among several computers.
● Message queue - Typically used for inter-process communication, or for inter-thread communication within the
same process.
● Distributed Locks – Generally used as Globally synchronous locks, which means at any snapshot in time no two
clients think they hold the same lock
FUNDAMENTALS OF SCALABLE COMPUTING
Apache Zookeeper – Use cases
THANK YOU

Dr. Radhika M. Hirannaiah

Department of Computer Science and Engineering
[email protected]
Fundamentals of Scalable Computing
Distributed systems and Trends
- Case Study – Netflix
Dr. Radhika M. Hirannaiah
Department of Computer Science and Engineering
FUNDAMENTALS OF SCALABLE COMPUTING
Case Study - Netflix

• Netflix has over 222 million subscribers in 180+ countries.

• Serves over 6 billion hours of content per month, globally, to nearly every country in the world.
• Netflix subscribers consume about 15% of the world's internet bandwidth traffic
• Netflix subscribers spend an average of 3.2 hours per day watching content on the platform.
• Netflix annual net income for 2021 was $5.116 billion, a 85.28% increase from 2020.
• Netflix reported a net income of $1.6 billion in Q1 of 2022, now $2.3billion ($9.3billion in Q1
2024)

https://aws.amazon.com/solutions/case-studies/netflix
https://dev.to/gbengelebs/netflix-system-design-backend-architecture-10i3
FUNDAMENTALS OF SCALABLE COMPUTING
Case Study - Netflix

• The overall Netflix system consists of three main parts.

• Open Connect Appliances(OCA) - Open Connect is Netflix’s custom global content delivery
network(CDN). These OCAs servers are placed inside internet service providers (ISPs) and
internet exchange locations (IXPs) networks around the world to deliver Netflix content to users.
• Client - A client is any device from which you play the video from Netflix. This consists of all the
applications that interface with the Netflix servers.
• Backend - This includes databases, servers, logging frameworks, application monitoring,
recommendation engine, background services, etc... When the user loads the Netflix app all
requests are handled by the backend server in AWS Eg: Login, recommendations, the home page,
users history, billing, customer support. Some of these backend services include (AWS EC2
instances, AWS S3, AWS DynamoDB, Cassandra, Hadoop, Kafka, etc).
FUNDAMENTALS OF SCALABLE COMPUTING
Case Study – Netflix Architecture
FUNDAMENTALS OF SCALABLE COMPUTING
Case Study – Netflix CDN

• Netflix performs compression and transcoding of the

original video file.
• The file is split into chunks using parallel workers in AWS
for faster processing.
• Each chunk is subdivided across several resolutions and
internet speeds.
• These chunks are stored on Amazon S3.
• During the off-peak period. The files are transferred to
open connect boxes, which is Netflix's custom content
delivery network spread out across the world.
• These boxes are capable of communicating and sharing
content between themselves.
FUNDAMENTALS OF SCALABLE COMPUTING
Case Study – Netflix Backend Architecture

• Netflix operates in two clouds Amazon Web Services and

Open Connect (Netflix content delivery network)
• Backend includes databases, servers, logging frameworks,
application monitoring, recommendation engine, background
services, etc...
• When the user loads the Netflix app all requests are handled
by the backend server in AWS Eg: Login, recommendations,
the home page, users history, billing, customer support.
• Some of these backend services include (AWS EC2 instances,
AWS S3, AWS DynamoDB, Cassandra, Hadoop, Kafka, etc).
FUNDAMENTALS OF SCALABLE COMPUTING
Case Study – Netflix Backend Architecture

• Netflix is one of the major drivers of microservices architecture.

• Every component of their system is a collection of loosely coupled services
collaborating
• The Client sends a Play request to a Backend running on AWS. Netflix uses
Amazon's Elastic Load Balancer (ELB) service to route traffic to its services.
• AWS ELB will forward that request to the API Gateway Service. Netflix uses Zuul
as its API gateway, which is built to allow dynamic routing, traffic monitoring,
and security, resilience to failures at the edge of the cloud deployment
• Application API component is the core business logic behind Netflix operations.
• Play API will call a microservice or a sequence of microservices to fulfill the
request.
• The data coming out of the Stream Processing Pipeline can be persistent to
other data stores such as AWS S3, Hadoop HDFS, Cassandra, etc.
FUNDAMENTALS OF SCALABLE COMPUTING
Case Study – Netflix API Gateway

• Netflix uses Amazon's Elastic Load Balancer (ELB) service to route traffic to
services.
• ELB’s are set up such that load is balanced across zones first, then
instances.
• Netflix uses Zuul as its API gateway, It handles all the requests and
performs the dynamic routing of microservice applications. It works as a
front door for all the requests.
• The Cloud Gateway team at Netflix runs and operates more than 80
clusters of Zuul 2, sending traffic to about 100 (and growing) backend
service clusters which amount to more than 1 million requests per second.
FUNDAMENTALS OF SCALABLE COMPUTING
Case Study – Netflix Application API

• Supposing the user clicks on play for the latest episode of peaky
blinders, the request will be routed to the playback API.
• The API in turn calls several microservices under the hood.
• Some of these calls can be made in parallel because they don’t
depend on each other. Others have to be sequenced in a specific
order.
• The API contains all the logic to sequence and parallelize the calls as
necessary.
• The device, in turn, doesn’t need to know anything about the
orchestration that goes on under the hood when the customer clicks
“play”
FUNDAMENTALS OF SCALABLE COMPUTING
Case Study – Netflix Container Management

• Titus is a container management platform that provides scalable and

reliable container execution and cloud-native integration with Amazon
AWS.
• It is a framework on top of Apache Mesos, a cluster management
system that brokers available resources across a fleet of machines.
• Titus is run in production at Netflix, managing thousands of AWS EC2
instances and launching hundreds of thousands of containers daily for
both batch and service workloads. Just think of it as the Netflix version
of Kubernetes.
• Titus runs about 3 million containers per week.
FUNDAMENTALS OF SCALABLE COMPUTING
Case Study – Netflix Datastores

EVCache
• A cache's primary purpose is to increase data retrieval
performance by reducing the need to access the underlying
slower storage layer
• A cache typically stores a subset of data transiently
• EVCache is a caching solution that is mainly used for AWS EC2
infrastructure for caching frequently used data
EVCache is an abbreviation for:
• Ephemeral - The data stored is for a short duration as specified
by its TTL (Time To Live).
• Volatile - The data can disappear any time (Evicted).
• Cache - An in-memory key-value store.
FUNDAMENTALS OF SCALABLE COMPUTING
Case Study – Netflix Datastores
MySQL
• Netflix uses AWS EC2 instances of MYSQL for its Billing infrastructure.
Billing infrastructure is responsible for managing the billing state of
Netflix members.
• The payment processor needed the ACID capabilities of an RDBMS to
process charge transactions.
Apache Cassandra
• Cassandra is a free and open-source distributed wide column store
NoSQL database designed to handle large amounts of data across many
commodity servers, providing high availability with no single point of
failure.
• Netflix uses Cassandra for its scalability, lack of single points of failure,
and cross-regional deployments.
• Netflix stores all kinds of data across their Cassandra DB instances. All
user collected event metrics are stored on Cassandra.
FUNDAMENTALS OF SCALABLE COMPUTING
Case Study – Netflix Stream Processing

Kafka
• Netflix embraces Apache Kafka® as the de-facto standard for its
eventing, messaging, and stream processing needs. Kafka acts as
a bridge for all point-to-point and Netflix Studio wide
communications.
Apache Chukwe- Analyzing Streaming Data
• Apache Chukwe is an open-source data collection system for
collecting logs or events from a distributed system. It is built on
top of HDFS and Map-reduce framework.
Apache Spark - Analyzing Streaming Data
• Netflix uses Apache Spark and Machine learning for Movie
recommendation
FUNDAMENTALS OF SCALABLE COMPUTING
Case Study – Netflix Stream Processing
FUNDAMENTALS OF SCALABLE COMPUTING
Case Study – Netflix deployed on AWS

Infrastructure services provided by AWS to deliver its content securely and reliably at a massive scale
Netflix uses:
• Amazon Elastic Compute Cloud (Amazon EC2) for computing resources
• Amazon Relational Database Service (Amazon RDS) for database instances
• Amazon Simple Storage Service (Amazon S3) for storage
• Amazon CloudFront for the content delivery network (CDN) bandwidth capacity
• Elastic Load Balancing for load balancing across multiple data center locations
• Auto Scaling Groups for automatically adding or removing servers depending upon fluctuating demand from
customers
• DynamoDB NoSQL database as an alternative choice for highly scalable applications that require rapid access
time without traditional relational database management system (RDBMS) features such as transactions or
referential integrity constraints
FUNDAMENTALS OF SCALABLE COMPUTING
Case Study – Netflix

References:
1. Netflix System Design - https://dev.to/gbengelebs/netflix-system-design-how-netflix-onboards-new-content-2dlb
2. Netflix System Design Backend Architecture - https://dev.to/gbengelebs/netflix-system-design-backend-architecture-
10i3
3. How Netflix is using AWS and its own CDN - https://ayushhsinghh.medium.com/case-study-how-netflix-is-using-aws-
and-its-own-cdn-64ca6282eda0
4. https://www.cloudthat.com/resources/blog/how-netflix-uses-aws-to-deliver-a-personalized-and-interactive-viewing-
experience
5. Netflix Case Study – AWS - https://aws.amazon.com/solutions/case-studies/netflix-case-study/
6. Netflix on AWS - https://aws.amazon.com/solutions/case-studies/innovators/netflix/
7. https://www.linkedin.com/pulse/how-netflix-uses-aws-services-ajinkya-khandave/
8. https://www.linkedin.com/pulse/case-study-netflix-using-aws-cloud-abhinav-singh/
THANK YOU
Dr. Radhika M. Hirannaiah
Department of Computer Science and Engineering
[email protected]