Data

Engineering
System Design
Core Concepts

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS SYSTEM DESIGN?

System design is the process of defining

1
the architecture, components, modules,
interfaces, and data for a system to
satisfy specified requirements. It involves
both high-level architecture and
detailed design.

Designing a news feed system for a

social media application.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A SCALABLE SYSTEM?

A scalable system is one that can

2
handle increased load without
compromising performance by adding
resources.

Horizontal scaling by adding more

servers to handle increased traffic.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

HOW DO YOU APPROACH A

SYSTEM DESIGN INTERVIEW?

Understand the requirements, define the

3
scope, outline a high-level architecture,
dive into detailed design for key
components, and address potential
bottlenecks and trade-offs.

Designing a URL shortener: Start with the

core functionality, then address
storage, scalability, and fault tolerance.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS THE DIFFERENCE BETWEEN

HORIZONTAL AND VERTICAL
SCALING?
Horizontal scaling (scaling out)

4
involves adding more machines to a
system, while vertical scaling (scaling
up) involves adding more power (CPU,
RAM) to an existing machine.

Adding more servers to a web

application (horizontal) vs. upgrading
the server’s hardware (vertical).

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT ARE THE ADVANTAGES OF

HORIZONTAL SCALING?

It improves fault tolerance, allows for

5
easier load distribution, and avoids the
limitations of a single machine.

Distributing web traffic across multiple

servers using a load balancer.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A LOAD BALANCER?

A load balancer distributes incoming

6
network traffic across multiple servers to
ensure no single server becomes
overwhelmed.

Using an AWS Elastic Load Balancer to

manage web traffic.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

HOW DOES A LOAD BALANCER

IMPROVE SYSTEM RELIABILITY?

By distributing traffic, it ensures that if

7
one server fails, the load balancer can
redirect traffic to other available servers,
thus maintaining service availability.

In a web application with three servers,

if one fails, the load balancer redirects
traffic to the remaining two.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS CACHING?

Caching is a technique to store

8
frequently accessed data in a
temporary storage location to speed up
subsequent data retrievals.

Using Redis to cache database query

results.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT ARE THE TYPES OF CACHES?

There are client-side caches, server-side

9
caches, and distributed caches.

Browser cache (client-side),

Memcached (server-side), and Redis
(distributed).

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A CDN (CONTENT

DELIVERY NETWORK)?

A CDN is a network of servers distributed

10
geographically to deliver static content
to users from the nearest server location,
reducing latency.

Using Cloudflare CDN to serve static

assets like images and CSS files.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

HOW DOES A CDN IMPROVE

PERFORMANCE?

By reducing the distance between users

11
and the content, decreasing load times
and reducing bandwidth usage.

Serving a video from a server located

closer to the user, resulting in faster
loading times.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS DATABASE REPLICATION?

Database replication is the process of

12
copying data from one database server
(master) to another (slave) to ensure
high availability and fault tolerance.

Using MySQL replication to maintain a

backup database server.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT ARE THE BENEFITS OF

DATABASE REPLICATION?

It improves read performance, ensures

13
high availability, and provides data
redundancy.

A master-slave setup where the master

handles writes, and slaves handle
reads.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS SHARDING?

Sharding is a database partitioning

14
technique that divides a large database
into smaller, more manageable pieces,
called shards, which can be distributed
across multiple servers.

Splitting user data across different

databases based on user ID ranges.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT ARE THE CHALLENGES OF

SHARDING?

Challenges include data consistency,

15
complex queries across shards, and re-
sharding data when a shard grows too
large.

Implementing a consistent hashing

algorithm to evenly distribute data
across shards.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A MESSAGE QUEUE?

A message queue is a component used

16
for communication between processes,
allowing them to send and receive
messages asynchronously.

Using RabbitMQ to manage

background tasks in a web application.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT ARE THE BENEFITS OF USING

A MESSAGE QUEUE?

It enables asynchronous processing,

17
improves system resilience, and
decouples system components.

Processing user signup emails in the

background while the main application
handles user requests.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A MICROSERVICES
ARCHITECTURE?
Microservices architecture is an
architectural style that structures an

18
application as a collection of loosely
coupled services, each with its own
functionality and data storage.
Breaking down a monolithic e-
commerce application into individual
services for inventory, payment, and
user management.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT ARE THE BENEFITS OF

MICROSERVICES?

Benefits include improved modularity,

19
easier scaling, and better fault isolation.

Like scaling the payment service

independently of the inventory service
in an e-commerce application.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A MONOLITHIC
ARCHITECTURE?

A monolithic architecture is a single-tier

20
software application where all
components are interconnected and
interdependent.

A traditional web application where the

frontend, backend, and database are
tightly integrated.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT ARE THE DRAWBACKS OF A

MONOLITHIC ARCHITECTURE?

Drawbacks include difficulty in scaling

21
individual components, tight coupling,
and challenges in maintaining and
deploying the application.

Updating one part of the application

requires redeploying the entire system.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS EVENTUAL CONSISTENCY?

Eventual consistency is a consistency

22
model used in distributed systems
where updates to a database will
propagate to all nodes, but not
immediately.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

HOW DO YOU ENSURE DATA

CONSISTENCY IN A DISTRIBUTED
SYSTEM?
Techniques include using consensus

23
algorithms (e.g., Paxos, Raft), distributed
transactions, and ensuring idempotent
operations.

The implementation of the Raft

algorithm is used to manage leader
election and data replication.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS CAP THEOREM?

CAP theorem states that in a distributed

data store, you can only achieve two out
of the following three guarantees:

24
Consistency, Availability, and Partition
tolerance.

Choosing between strong consistency

and availability in a distributed
database during network partitions.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A KEY-VALUE STORE?

A key-value store is a type of NoSQL

database that stores data as a

25
collection of key-value pairs.

Using Redis or Amazon DynamoDB to

store session data.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT ARE THE ADVANTAGES OF

KEY-VALUE STORES?

They offer fast read and write

26
operations, are easy to scale, and
provide flexible data models.

For example storing user preferences in

Redis helps in quick retrieval.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A DOCUMENT STORE?

A document store is a type of NoSQL

27
database that stores data as
documents, typically in JSON or BSON
format.

Using MongoDB to store user profiles

with varying attributes.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A GRAPH DATABASE?

A graph database is designed to store

28
and query data modeled as graphs, with
nodes, edges, and properties.

Using Neo4j to manage social network

data with complex relationships.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT ARE THE BENEFITS OF GRAPH

DATABASES?

They excel at handling complex

29
relationships, enable efficient traversal
queries, and provide a flexible schema.

Finding shortest paths and

recommendations in a social network
using Neo4j.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A COLUMN-FAMILY
STORE?

A column-family store is a type of NoSQL

30
database that stores data in columns
rather than rows, optimized for read and
write operations.

Using Apache Cassandra for time-

series data storage.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT ARE THE BENEFITS OF

COLUMN-FAMILY STORES?

They offer high write throughput,

31
horizontal scalability, and are suitable
for time-series data and real-time
analytics.

Storing and retrieving logs in Apache

Cassandra.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A DISTRIBUTED FILE

SYSTEM?

A distributed file system (DFS) is a file

32
system that allows access to files from
multiple hosts, providing redundancy
and fault tolerance.

Using Hadoop Distributed File System

(HDFS) for big data storage.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A DATA WAREHOUSE?

A data warehouse is a centralized

33
repository for storing large volumes of
structured and semi-structured data,
optimized for query and analysis.

Example using Amazon Redshift or

Google BigQuery for business analytics.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS ETL (EXTRACT,

TRANSFORM, LOAD)?

ETL is a process in data warehousing

34
that involves extracting data from
various sources, transforming it to fit
operational needs, and loading it into a
target data store.

Using Apache NiFi to extract data from

databases, transform it, and load it into
HDFS.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A DATA LAKE?

A data lake is a storage repository that

35
holds vast amounts of raw data in its
native format until it is needed.

Using Amazon S3 as a data lake for

storing structured and unstructured
data.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT ARE THE BENEFITS OF A DATA

LAKE?

Benefits include scalability, flexibility, and

36
the ability to store diverse data types.

Storing log files, images, and structured

data together in Amazon S3.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS STREAM PROCESSING?

Stream processing involves processing

37
data in real-time as it flows from one
source to another, allowing immediate
insights and actions.

Using Apache Kafka and Apache Flink

to process real-time clickstream data.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT ARE THE ADVANTAGES OF

STREAM PROCESSING?

It enables real-time analytics, reduces

38
data latency, and supports event-driven
architectures.

Real-time fraud detection in financial

transactions using Apache Kafka and
Apache Storm.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS BATCH PROCESSING?

Batch processing involves processing

39
data in large blocks or batches at
scheduled intervals.

Using Apache Hadoop to run nightly ETL

jobs.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS THE DIFFERENCE BETWEEN

BATCH PROCESSING AND STREAM
PROCESSING?
Batch processing deals with large

40
volumes of data processed at intervals,
while stream processing deals with
continuous data processing in real-time.

Using Apache Hadoop for batch

processing and Apache Kafka for
stream processing.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A LAMBDA
ARCHITECTURE?
A lambda architecture is a data-
processing architecture designed to

41
handle massive quantities of data by
taking advantage of both batch and
stream-processing methods.
Combining Apache Hadoop for batch
processing and Apache Storm for real-
time processing in a lambda
architecture.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT ARE THE COMPONENTS OF A

LAMBDA ARCHITECTURE?

The components include a batch layer

42
for processing large datasets, a speed
layer for real-time processing, and a
serving layer to merge the results.

Using Hadoop for batch processing,

Kafka for real-time data, and HBase as
the serving layer.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS THE KAPPA

ARCHITECTURE?

The kappa architecture simplifies the

43
lambda architecture by using only
stream processing to handle both real-
time and historical data.

Using Apache Kafka and Apache Flink

to process all data as streams.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A NOSQL DATABASE?

A NoSQL database is a non-relational

44
database designed to handle large
volumes of unstructured or semi-
structured data with flexible schemas.

Using MongoDB or Cassandra for

storing large-scale data with varying
structures.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT ARE THE TYPES OF NOSQL

DATABASES?

Types include key-value stores,

45
document stores, column-family stores,
and graph databases.

Redis (key-value), MongoDB

(document), Cassandra (column-
family), Neo4j (graph).

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A RELATIONAL
DATABASE?

A relational database is a type of

46
database that organizes data into
tables with rows and columns, using SQL
for data management.

Using MySQL or PostgreSQL for

structured data storage.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT ARE THE BENEFITS OF

RELATIONAL DATABASES?

They provide strong consistency, ACID

47
transactions, and support complex
queries and relationships.

Using SQL joins to query related data

across multiple tables in PostgreSQL.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS ACID COMPLIANCE?

ACID stands for Atomicity, Consistency,

Isolation, and Durability, which are

48
properties that ensure reliable database
transactions.

A banking system using ACID

transactions to ensure accurate fund
transfers.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS BASE CONSISTENCY?

BASE stands for Basically Available, Soft

49
state, and Eventual consistency, which
are properties of some NoSQL databases
that prioritize availability over strict
consistency.

Using Cassandra for high availability

and eventual consistency in a
distributed database.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A DISTRIBUTED SYSTEM?

A distributed system is a system whose

50
components are located on different
networked computers, which
communicate and coordinate to
achieve a common goal.

Using a microservices architecture with

services running on multiple servers.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT ARE THE CHALLENGES OF

DISTRIBUTED SYSTEMS?

Challenges include network latency,

51
fault tolerance, data consistency, and
synchronization.

Implementing consensus algorithms

like Raft to manage distributed state.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS FAULT TOLERANCE?

Fault tolerance is the ability of a system

52
to continue operating properly in the
event of the failure of some of its
components.

Using redundant servers and data

replication to ensure high availability.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

HOW DO YOU ACHIEVE HIGH

AVAILABILITY?

Techniques include using load

53
balancers, redundant systems, failover
mechanisms, and data replication.

Implementing database replication and

load balancing for a web application.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A CONSENSUS
ALGORITHM?

A consensus algorithm is a process in

54
computer science used to achieve
agreement on a single data value
among distributed processes or
systems.

Using the Raft algorithm to elect a

leader in a distributed system.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS THE RAFT ALGORITHM?

Raft is a consensus algorithm designed

55
to be understandable and
implementable, used to manage a
replicated log in distributed systems.

Implementing Raft for leader election

and log replication in a distributed
database.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS THE PAXOS ALGORITHM?

Paxos is a family of protocols for solving

56
consensus in a network of unreliable or
faulty processors.

Using Paxos for achieving consensus in

a distributed system with unreliable
nodes.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A LEADER ELECTION?

Leader election is the process of

57
designating a single node as the
organizer of some task distributed
among several nodes.

Using ZooKeeper for leader election in a

distributed system.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A QUORUM?

A quorum is the minimum number of

58
votes needed for a distributed
transaction to be committed.

Using a majority quorum (n/2 + 1) in a

distributed database to ensure data
consistency.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A WRITE-AHEAD LOG

(WAL)?

A write-ahead log is a technique used in

59
databases to ensure data integrity by
logging changes before applying them
to the database.

Using WAL in PostgreSQL to ensure data

is not lost during a crash.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS STRONG CONSISTENCY?

Strong consistency ensures that all

61
nodes in a distributed system see the
same data at the same time after a
write operation.

Using a relational database like

PostgreSQL to ensure strong
consistency.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A DATA PARTITION?

Data partitioning involves dividing a

62
database into smaller, more
manageable pieces, which can be
stored across multiple servers.

Sharding a user database by user ID to

distribute data across different servers.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A DISTRIBUTED HASH

TABLE (DHT)?

A DHT is a decentralized distributed

65
system that provides a lookup service
similar to a hash table, where data is
distributed across multiple nodes.

Using a DHT for peer-to-peer file

sharing in BitTorrent.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A GOSSIP PROTOCOL?

A gossip protocol is a method of peer-

66
to-peer communication where nodes
periodically exchange state information
to achieve eventual consistency.

Using a gossip protocol in Cassandra

for node state information exchange.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A LEADER-FOLLOWER
PATTERN?

The leader-follower pattern is a

67
replication strategy where one node
(leader) handles all writes, and other
nodes (followers) replicate the leader's
data.

Using leader-follower replication in

Kafka for high availability.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A DISTRIBUTED LOCK?

A distributed lock is a mechanism to

68
synchronize access to a shared resource
across multiple nodes in a distributed
system.

Using ZooKeeper to implement

distributed locks for coordination.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A HEARTBEAT IN
DISTRIBUTED SYSTEMS?

A heartbeat is a periodic signal sent by a

69
node to indicate its presence and
operational status to other nodes in a
distributed system.

Using heartbeats in Kubernetes to

monitor node health.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A PARTITION TOLERANCE?

Partition tolerance is the ability of a

70
distributed system to continue
functioning even when network
partitions occur, isolating parts of the
system.

Using eventual consistency to handle

network partitions in a distributed
database.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS THE DIFFERENCE BETWEEN

SYNCHRONOUS AND
ASYNCHRONOUS REPLICATION?
Synchronous replication requires that

71
data be written to all replicas before
acknowledging the write, while
asynchronous replication allows for
acknowledgement before all replicas
are updated.

Synchronous replication in a financial

transaction system for data integrity vs.
asynchronous replication in a content
delivery network for performance.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A READ REPLICA?

A read replica is a copy of a database

that is used to offload read traffic from

72
the primary database, improving
performance and availability.

Using read replicas in Amazon RDS to

handle increased read traffic.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS DATA CONSISTENCY?

Data consistency ensures that data is

73
the same across all nodes in a
distributed system after a write
operation.

Using strong consistency in a

distributed database to ensure
accurate data retrieval.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A ROLLBACK?

A rollback is the process of undoing

75
changes made to a database during a
transaction if an error occurs, ensuring
data integrity.

Using rollback in SQL transactions to

revert changes if an error occurs during
a multi-step process.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A DISTRIBUTED CACHE?

A distributed cache is a caching

76
mechanism where data is stored across
multiple nodes to improve scalability
and performance.

Using Redis Cluster to distribute cache

data across multiple nodes.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A CIRCUIT BREAKER

PATTERN?

The circuit breaker pattern is a design

77
pattern used to detect failures and
prevent cascading failures in distributed
systems by stopping the flow of requests
to a failing service.

Implementing a circuit breaker in a

microservices architecture to handle
service failures.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A BULKHEAD PATTERN?

The bulkhead pattern isolates

78
components of a system into separate
pools to prevent a failure in one
component from affecting others.

Using separate thread pools for

different services in a microservices
architecture to isolate failures.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A FALLBACK MECHANISM?

A fallback mechanism provides an

79
alternative action or data source when a
service call fails, ensuring system
resilience.

Returning cached data or a default

response when a primary service call
fails.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS IDEMPOTENCY?

Idempotency ensures that performing

80
an operation multiple times has the
same effect as performing it once, which
is crucial for reliable systems.

Using unique transaction IDs to ensure

that duplicate payment requests do not
result in multiple charges.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A SERVICE DISCOVERY?

Service discovery is the process of

81
automatically detecting services and
their endpoints in a distributed system,
facilitating communication between
components.

Using Consul or Eureka for service

discovery in a microservices
architecture.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A PROXY SERVER?

A proxy server acts as an intermediary

82
for requests from clients seeking
resources from other servers, providing
benefits like load balancing, caching,
and security.

Using Nginx as a reverse proxy to

distribute traffic to backend servers.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A REVERSE PROXY?

A reverse proxy is a type of proxy server

83
that retrieves resources on behalf of a
client from one or more servers, often
used for load balancing and security.

Using HAProxy to distribute incoming

HTTP requests to multiple web servers.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A CDN EDGE SERVER?

A CDN edge server is a server located at

84
the edge of a network that stores
cached content, delivering it to users
from a location closer to them.

Using Cloudflare edge servers to deliver

static content to users with low latency.

Shwetank Singh
GritSetGrow - GSGLearn.com
 DATA ENGINEERING - SYSTEM DESIGN

WHAT IS A MICROSERVICES
GATEWAY?

A microservices gateway is an API

85
gateway that manages and routes
requests to the appropriate
microservice, often handling cross-
cutting concerns like authentication and
rate limiting.

Using Kong or Apigee as a

microservices gateway to manage API
traffic.

Shwetank Singh
GritSetGrow - GSGLearn.com