DATA ENGINEERING

Data collection & storage -> data preparation -> data exploration & visualization -> data
experimentation & prediction

Data Engineer is a professional who builds and maintains data infrastructure, including data
pipelines, databases, and data warehouses.

Data Pipeline -> collecting the data from various resources than processing it as per requirement and
transferring it to the destination by following some sequential activities. It is a set of manner that
first extracts data from various resources and transforms it to a destination means it processes it as
well as moves it from one system to another system.

Data Warehouse -> A data warehouse is a centralized system used for storing and managing large
volumes of data from various sources. It is designed to help businesses analyze historical data and
make informed decisions. Data from different operational systems is collected, cleaned, and stored in
a structured way, enabling efficient querying and reporting.

PYTHON

Python uses indentation to indicate a block of code.

Variables are containers for storing data values.

Casting ->

If you want to specify the data type of a variable, this can be done with casting. Examples ->

x = str(3) # x will be '3'

y = int(3) # y will be 3

z = float(3) # z will be 3.0

You can get the data type of a variable with the type() function.

x=5

y = "John"

print(type(x))
print(type(y))

A variable name must start with a letter or the underscore character

A variable name cannot start with a number

A variable name can only contain alpha-numeric characters and underscores (A-z, 0-9, and _ )

Variable names are case-sensitive (age, Age and AGE are three different variables)

A variable name cannot be any of the Python keywords.

Legal variable names:

myvar = "John"

my_var = "John"

_my_var = "John"

myVar = "John"

MYVAR = "John"

myvar2 = "John"

Variables that are created outside of a function are known as global variables. Global variables can
be used by everyone, both inside of functions and outside.

x = "awesome"

def myfunc():

print("Python is " + x)

myfunc()

To create a global variable inside a function, you can use the global keyword.

Normally, when you create a variable inside a function, that variable is local, and can only be used
inside that function.

def myfunc():

global x

x = "fantastic"
myfunc()

print("Python is " + x)

You can get the data type of any object by using the type() function:

UNPACKING ->

If you have a collection of values in a list, tuple etc. Python allows you to extract the values into
variables. This is called unpacking.

fruits = ["apple", "banana", "cherry"]

x, y, z = fruits

print(x)

print(y)

print(z)

PYTHON DATA TYPES->

Type conversion

x = 1 # int

y = 2.8 # float

z = 1j # complex

#convert from int to float:

a = float(x)

#convert from float to int:

b = int(y)

#convert from int to complex:

c = complex(x)

print(a)
print(b)

print(c)

print(type(a))

print(type(b))

print(type(c))

Random number-> Python does not have a random() function to make a random number, but
Python has a built-in module called random that can be used to make random numbers:

import random

print(random.randrange(1, 10))

PYHTON STRINGS -> Like many other popular programming languages, strings in Python are arrays of
bytes representing unicode characters.

Square brackets can be used to access elements of the string.

Get the character at position 1 (remember that the first character has the position 0):

a = "Hello, World!"

print(a[1])

Looping through a string ->Since strings are arrays, we can loop through the characters in a string,
with a for loop.

for x in "banana":

print(x)

String length-> To get the length of a string, use the len() function.

The len() function returns the length of a string:

a = "Hello, World!"

print(len(a))
Check string ->To check if a certain phrase or character is present in a string, we can use the
keyword in.

Check if "free" is present in the following text:

txt = "The best things in life are free!"

print("free" in txt)

txt = "The best things in life are free!"

if "free" in txt:

print("Yes, 'free' is present.")

-> Check if "expensive" is NOT present in the following text:

txt = "The best things in life are free!"

print("expensive" not in txt)

SLICING STRINGS->

Slicing: You can return a range of characters by using the slice syntax.

Get the characters from position 2 to position 5 (not included):

b = "Hello, World!"

print(b[2:5])

Slice from start: By leaving out the start index, the range will start at the first character:

Get the characters from the start to position 5 (not included):

b = "Hello, World!"

print(b[:5])

Slice to the end: By leaving out the end index, the range will go to the end:

Get the characters from position 2, and all the way to the end:

b = "Hello, World!"

print(b[2:])

Negative indexing: Use negative indexes to start the slice from the end of the string:
Get the characters:

From: "o" in "World!" (position -5)

To, but not included: "d" in "World!" (position -2):

b = "Hello, World!"

print(b[-5:-2])

MODIFY STRINGS->

Python has a set of built-in methods that you can use on strings.

UpperCase: The upper() method returns the string in upper case:

a = "Hello, World!"

print(a.upper())

LowerCase: The lower() method returns the string in lower case:

a = "Hello, World!"

print(a.lower())

Remove whitespaces: The strip() method removes any whitespace from the beginning or the end:

a = " Hello, World! "

print(a.strip()) # returns "Hello, World!"

Replace String: The replace() method replaces a string with another string:

a = "Hello, World!"

print(a.replace("H", "J"))

Split string:The split() method splits the string into substrings if it finds instances of the separator:

a = "Hello, World!"

print(a.split(",")) # returns ['Hello', ' World!']

String Concatenation: To concatenate, or combine, two strings you can use the + operator.

Example:

Merge variable a with variable b into variable c:

a = "Hello"
b = "World"
c=a+b
print(c)

Example

To add a space between them, add a " ":

a = "Hello"
b = "World"
c=a+""+b
print(c)

String format: we cannot combine strings and numbers like this:

Example

age = 36
txt = "My name is John, I am " + age
print(txt)

But we can combine strings and numbers by using f-strings or the format() method!

To specify a string as an f-string, simply put an f in front of the string literal, and add curly
brackets {} as placeholders for variables and other operations.

Example

Create an f-string:

age = 36
txt = f"My name is John, I am {age}"
print(txt)

Escape Characters ->To insert characters that are illegal in a string, use an escape character.

An escape character is a backslash \ followed by the character you want to insert.

Example: You will get an error if you use double quotes inside a string that is surrounded by double
quotes: txt = "We are the so-called "Vikings" from the north."

To fix this problem, use the escape character \":

Example: The escape character allows you to use double quotes when you normally would not be
allowed:

txt = "We are the so-called \"Vikings\" from the north."

String Methods -> Python has a set of built-in methods that you can use on strings.

Note: All string methods return new values. They do not change the original string.

ljust(), rjust(), count(), partition(), replace(), rfind(), split(), strip(), zfill() ….etc refer this ->
https://www.w3schools.com/python/python_strings_methods.asp

Pyhton Booleans -> The bool() function allows you to evaluate any value, and give
you True or False in return,

Example

Evaluate a string and a number:

print(bool("Hello"))
print(bool(15))

Most Values are True

Almost any value is evaluated to True if it has some sort of content.

Any string is True, except empty strings.

Any number is True, except 0.

Any list, tuple, set, and dictionary are True, except empty ones.

Example

The following will return True:

bool("abc")
bool(123)
bool(["apple", "cherry", "banana"])

Some Values are false

In fact, there are not many values that evaluate to False, except empty values, such as (), [], {}, "", the
number 0, and the value None. And of course the value False evaluates to False.

Example

The following will return False:

bool(False)
bool(None)
bool(0)
bool("")
bool(())
bool([])
bool({})
Python also has many built-in functions that return a boolean value, like the isinstance() function,
which can be used to determine if an object is of a certain data type:

Example

Check if an object is an integer or not:

x = 200
print(isinstance(x, int))

Python Operators -> Python divides the operators in the following groups:

 Arithmetic operators

 Assignment operators

 Comparison operators

 Logical operators

 Identity operators

 Membership operators

 Bitwise operators

Arthimetic Operators ->

+, -, *, /, %, **, //

Assignment operators ->