ACKNOWLEDGEMENT

I would like to express our sincere gratitude to these dignitaries, who are with us in

the journey of my Summer Internship “AWS Data Engineering Virtual Internship.”

I am utmost grateful to galgotias university Department

of Computer Science and Engineering for paving a path for me and assisting

me to meet the requirements that are needed to complete this internship.

I would also like to express our sincere thanks to my friends and family for their

moral support though out our journey.

This whitepaper is designed to help architects, data scientists, and developers navigate big data
analytics

options in the AWS cloud. It provides an overview of relevant AWS services, covering ideal usage

patterns, cost models, performance, durability, availability, scalability, elasticity, interfaces, and anti-

patterns. Key objectives include data integration and centralization to unify diverse datasets from

educational institutions, including student records, faculty information, academic performance, and

administrative data. The paper also emphasizes real-time data processing to enable near-instant
analysis

and support quick decision-making for educational administrators and policymakers. Security and

compliance are highlighted, focusing on protecting sensitive information in line with AICTE regulatory

standards.

A scalable infrastructure is recommended to accommodate growing data volumes from expanding

technical institutions. Additionally, the whitepaper advocates for an analytics and reporting
framework to

generate actionable insights, fostering data-driven decisions at institutional and regulatory levels.
Finally,

a collaborative data ecosystem is proposed to encourage secure data exchange and interoperability

between educational institutions within the AWS environment, ensuring an integrated and
cooperative

approach to data management. The paper concludes with scenarios demonstrating AWS analytics in

action and offers resources for initiating big data analytics on AW

COURSE MODULES

OVERVIEW OF AWS ACADEMY DATA ENGINEERING

Course objectives:

This course prepares you to do the following:

Summarize the role and value of data science in a data-driven organization.

Recognize how the elements ofdata influence decisions about the

infrastructure of a data pipeline.

Illustrate a data pipeline by using AWS services to meet a generalized use case.

Identify the risks and approaches to secure and govern data at each step

andeach transition of the data pipeline

Identify scaling considerations and best practices for building pipelines that

handle large-scale datasets.

Design and build a data collection process while considering constraints such

as scalability, cost, fault tolerance, and latency.

Code Generation

Open Code Reference Log

Code Whisperer learns from open-source projects and the code it suggests might occasionally

resemble code samples from the training data. With the reference log, you can view references

to code suggestions that are similar to the training data. When such occurrences happen, Code

Whisperer notifies you and provides repository and licensing information.

Benefits of Amazon Code Whisperer

Code Whisperer code generation offers many benefits for software development organizations.

It accelerates application development for faster delivery of software solutions.By automating

repetitive tasks, it optimizes the use of developer time, so developers can focus on more critical

aspects of the project. Additionally, code generation helps mitigate security vulnerabilities,
safeguarding the integrity of the codebase. Code Whisperer also protects open source

intellectual property by providing the open source reference tracker. Code Whisperer enhances

code quality and reliability, leading to robust and efficient applications. And it supports an

efficient response to evolving software threats, keeping the codebase up to date with the latest

security practices. Code Whisperer has the potential to increase development speed, security,

and the quality yof software.

How do organizations decide...

Which of these customer transactions should be flagged as fraud?

Which webpage design leads to the most completed sales?

Which patients are most likely to have a relapse?

Which type of online activity represents a security issue?

When is the optimum time to harvest this year's crop?

The data pipeline infrastructure for data-driven decisions:

data Pipeline

Another key characteristic of deriving insights by using your data pipeline is that the process

will almost always be iterative. You have a hypothesis about what you expect to find in the

data, and you need to experiment and see where it takes you. You might develop your

hypothesis by using BI tools to do initial discovery and analysis of data that has already been

collected. You might iterate within a pipeline segment, or you might iterate across the entire

pipeline. For example, in this illustration, the initial iteration (number 1) yielded a result that

wasn't as defined as was desired. Therefore, the data scientist refined the model and

reprocessed the data to get a better result (number 2). After reviewing those results, they

determined that additional data could improve the detail available in their result, so an

additional data source was tapped and ingested through the pipeline to produce the desired

result (number 3). A pipeline often has iterations of storage and processing. For example, after

the external data is ingested into pipeline storage, iterative processing transforms the data into

different levels of refinement for different needs.

Code Whisperer code generation offers many benefits for software development organizations.

It accelerates application development for faster delivery of software solutions.By automating

repetitive tasks, it optimizes the use of developer time, so developers can focus on more critical

aspects of the project. Additionally, code generation helps mitigate security vulnerabilities,

safeguarding the integrity of the codebase. Code Whisperer also protects open source

intellectual property by providing the open source reference tracker. Code Whisperer enhances

code quality and reliability, leading to robust and efficient applications. And it supports an
efficient response to evolving software threats, keeping the codebase up to date with the latest

security practices. Code Whisperer has the potential to increase development speed, security,

and the quality software

Chapter- 3:

THE ELEMENTS OF DATA, DESIGN PRINCIPLES &

PATTERNS FOR DATA PIPELINES

The five Vs of data- volume, velocity, variety, veracity& value:

Data Characteristics

The evolution of data architectures:

So, which of these data stores or data architectures is the best one for your data pipeline?

The reality is that a modern architecture might include all of these elements. The key to a

modern data architecture is to apply the three-pronged strategy that you learned about earlier.

Modernize the technology that you are using. Unify your data sources to create a single source

of truth that can be accessed and used across the organization. And innovate to get higher

value analysis from the data that you have.

Variety data types, Modern data architecture on AWS:

The architecture illustrates the following other AWS purpose-built services that integrate

with Amazon S3 and map to each component that was described on the previous slide:

Amazon Redshift is a fully managed data warehouse service.

•Amazon OpenSearch Service is a purpose-built data store and search engine that is

optimized for real-time analytics, including log analytics.

•Amazon EMR provides big data processing and simplifies some of the most complex

elements of setting up big data processing.

•Amazon DynamoDB is a fully managed nonrelational database that is designed to run

high-performance applications.

•Amazon Sage Maker is an AI/ML service that democratizes access to ML process

Modern data architecture

pipeline: Ingestion and

storage:

Data being ingested into the Amazon S3 data lake arrives at the landing zone, where it is first

cleaned and stored into the raw zone for permanent storage. Because data that is destined for

the data warehouse needs to be highly trusted and conformed to a schema, the data needs to

be processed further additional transformations would include applying the schema and

partitioning (structuring) as well as other transformations that are required to make the data
conform to requirements that are established for the trusted zone. Finally, the processing layer

prepares the data for the curated zone by modeling and augmenting it to be joined with other

datasets (enrichment) and then stores the transformed, validated data in the curated layer.

Datasets from the curated layer are ready to be ingested into the data warehouse to make them

available for low-latency access or complex SQL querying.

Streaming analytics pipeline:

Producers ingest records onto the stream. Producers are integrations that collect data from

a source and load it onto the stream. Consumers process records. Consumers read data from

the stream and perform their own processing on it. The stream itself provides a temporary but

durable storage layer for the streaming solution. In the pipeline that is depicted in this slide,

Amazon CloudWatch Events is the producer that puts CloudWatch Events event data onto the

stream. Kinesis Data Streams provides the storage. The data is then available to multiple

consumers.
Chapter-4:

SECURING & SCALING DATA PIPELINE

to describe and provision all of the infrastructure resources in your cloud environment. Cloud

Formation creates, updates, and deletes the resources for your applications in environments

called stacks. A stack is a collection of AWS resources that are managed as a single unit.

CloudFormation is all about automated resource provisioning—

it simplifies the task of repeatedly and predictably creating groups of related resources that

power your applications. Resources are written in text files by using JSON or YAML format.

Chapter 5 INGESTING & PREPARING DATA

Data wrangling: Transforming large amounts of unstructured or structured raw data from multiple
sources with different schemas into a meaningful set of data that has value for downstream
processes or users. Data Structuring: For the scenario that was described previously, the structuring
step includes exporting a .json file from the customer support ticket system, loading the .json file into
Excel, and letting Excel parse the file. For the mapping step for the supp2 data, the data engineer
would modify the cust num field to match the customer id field in the data warehouse. Forthis
example, you would performadditionaldata wrangling steps before compressing the file for upload
to the S3 bucket. Data Cleaning: It includes; • Remove unwanted data. • Fill in missing data values. •
Validate or modify the data types. • Fix outliers

Chapter 6 INGESTING BY BATCH OR BY STREAM

To generalize the characteristics of batch processing, batch ingestion involves running batch

jobs that query a source, move the resulting dataset or datasets to durable storage in the

pipeline, and then perform whatever transformations are required for the use case. As noted in

the Ingesting and Preparing Data module, this could be just cleaning and minimally formatting

data to put it into the lake. Or, it could be more complex enrichment, augmentation, and

processing to support complex querying or big data and machine learning (ML) applications.

Batch processing might be started on demand, run on a schedule, or initiated by an event.

Traditional extract, transform, and load (ETL) uses batch processing, but extract, load, and

transform(LT) processing might also be done by batch.

The process of transporting data from one or more sources to a target site for further processing

and analysis. This data can originate from a range of sources, including data lakes,IoT devices,

on-premises databases, and SaaS apps, and end up in different target environments, such as

cloud data warehouses or data marts

chapter 7 STORING AND ORGANIZING DATA

Data in cloud object storage is handled as objects. Each object is assigned a key, which is a

unique identifier. When the key is paired with metadata that is attached to the objects, other

AWS services can use the information to unlock a multitude of capabilities. Thanks to

economies of scale, cloud object storage comes at a lower cost than traditional storage.

Data Warehouse Storage:

•Provide a centralized repository

.Store structured and semi-structured data

•Store data in one of two ways:

•Frequently accessed data in fast storage

•In frequently accessed data in cheap storage

•Might contain multiple databases that are organized into tables and columns

•Separate analytics processing from transactional databases

• Example: Amazon Redshift

Purpose-Built Data Bases:

•ETL pipelines transform data in buffered memory prior to loading data into a data lake or

data warehouse for storage.

•ELT pipelines extract and load data into a data lake or data warehouse for storage without

transformation. Here are a few key points to summarize this section. Storage plays an integral

part in ELT and ETL pipelines. Data often moves in and out of storage numerous times, based
on pipeline type and workload type.

ETL pipelines transform data in buffered memory prior to loading data into a data lake or

data warehouse for storage. Levels of buffered memory vary by service.

ELT pipelines extract and load data into data lake or data warehouse storage without

transformation. The transformation of the data is part of the target system’s workload

Securing Storage

Security for a data warehouse in Amazon Redshift

•Amazon Redshift database securityis distinct from the security of the service itself.

•Amazon Redshift provides additional features to manage database security.

•Due to third-party auditing, Amazon Redshift can help to support applications that are

required to meet international compliance standards

chapter 8 PROCESSING BIG DATA & DATA FOR ML

Apache Spark characteristics,

•Is an open-source, distributed processing framework

•Uses in-memorycaching and optimized query processing

•Supports coder use across multiple workloads

•Clusters consist of leader and worker nodes

Amazon EMR Characteristics:

•Managed cluster platform

•Big data solution for petabyte-scale data processing, interactive analytics, and machine

learning

chapter 9 ANALYZING & VISUALIZING DATA

•How much data is there?

•How frequently is it updated?

•How quickly is it processed?

• What type of data is it?

9.2 Comparing AWS tools and Services:

For accessibility: Data from multiple sources is put in Amazon S3, where Athena can be used

for one-time queries. Amazon EMR aggregates the data and stores the aggregates in S3.

Athena can be used to query the aggregated datasets. From S3, the data can be used in Amazon

Redshift, where Quick Sight can access the data to create visualizations. End of accessibility

description.

Chapter 10 AUTOMATING THE PIPELINE

you build infrastructure with code, you gain the benefits of repeatability and reusability

while you build your environments. In the example shown, a single template is used to deploy

Network Load Balancers and Auto Scaling groups that contain Amazon Elastic Compute

Cloud (Amazon EC2) instances. Network Load Balancers distribute traffic evenly across

targets.

CI/CD:

CI/CD can be pictured as a pipeline, where new code is submitted on one end, tested over a

series of stages (source, build, test, staging, and production), and then published as

production-ready cod
CONCLUSION

Data engineering is a critical component in the modern data landscape, playing a crucial role

in the success of data-driven decision-making and analytics. As we draw conclusions about

data engineering, several key points come to the forefront:

Foundation for Data-Driven Insights:

Data engineering serves as the foundation for extracting, transforming, and loading (ETL) data

from diverse sources into a format suitable for analysis. This process is essential for generating

meaningful insights and facilitating informed decision-making within organizations.

Data Quality and Integrity:

Maintaining data quality and integrity is paramount in data engineering. Data engineers are

responsible for cleaning, validating, and ensuring the accuracy of data, contributing to the

reliability ofdownstream analyses and business processes.

Scalability and Performance:

With the increasing volume, velocity, and variety of data, data engineering solutions must be

scalable and performant. Scalability ensures that systems can handle growing amounts of data,

while performance optimization ensures timely processing and availability of data for analytics.

Integrationof Diverse Data Sources:

Data engineering enables the integration of data from various sources, whether structured or

unstructured, providing a unified view of information. This integration is crucial for a

comprehensive understanding ofbusiness operations