Python Programming

Day 1
Learning Outcomes
A. To be able to gain the confidence to use Python as a programming language
B. To do basic operations in Numpy and Pandas
1. Indexing, Slicing, and Querying
2. Vector and matrix computations

C. To be able to do basic preprocessing using Python, Numpy and Pandas:

1. Type conversion
2. Missing values and standardization
3. String cleaning
4. Cleaning outliers

D. To be able to visualize the data with Matplotlib, Pandas and Seaborn

Jupyter Notebook
Familiarization
Python Programming
Familiarization
Save and Checkpoint

Insert Cell Below

Cut, Copy, or Paste Cells

Move Selected Cell Up or Down

Run Cell, Interrupt Kernel, or Restart the Kernel

Shortcuts:
Shortcuts:
Tips
1.) CTRL-M then H to see help
2.) CTRL-M then S to save the notebook
3.) CTRL-Enter to Run the Code but stay in the same cell
4.) SHIFT-Enter to Run the Code and advance to the next cell

5.) Using %pylab inline preceding everything else in the notebook, imports matplotlib and
numpy. It also enables graphics to be part of the notebook.

6.) You can use TAB to see available functions and SHIFT-TAB repeatedly for the
documentation
Variables and Data Types
Python Programming
Variables and Data Types
Python uses five standard data types:
Numbers Tuples

Strings Dictionaries

Lists
Arithmetic
Addition Exponent

Subtraction Division

Multiplication
Arithmetic
Increment/Decrement String Concatenation

Decrement Complex Expressions

Challenge 1!
Python Programming
Challenge 1: Convert the following formula into a code

1.) When z = 8 and e = 2.718; g(z) should be equal to 0.0003

2.) When z = 2 and e = 2.718; g(z) should be equal to 0.1192246961081721

Control Statements and
Data Structures
Python Programming
Conditional Statements
Boolean Conditions Numerical Conditions

String Conditions
Loops
For Loops Nested Loops

While Loops

*Note: The loop declaration ends with a

colon (:) while its contents inside
are indented
Lists
Can contain a number of values comprised of different datatypes.

*Note: Lists starts with an index of 0.

The remove() method erases the
first value that matches the
parameter.
Dictionaries
List Generators and Comprehension
Generators can be used to build list in the memory as objects

*Note: This creates an object

*Note: Python’s built in range function

range([start], stop, [step])
Slicing

*Note: list( start : )

list( : end )
list( start : end )
list( start : end : step )
Functions
Python Programming
Functions
Python functions use the following notation:

def function_name:
commands *Note: Functions are defined with the keyword
def before the function_name. Similar to
loops, the function definition ends with a
colon (:) while its contents inside are
indented
Challenge 2!
Python Programming
Challenge 2: Coin Flip
Create a function coin_flip() that simulates coin flips repeated n times, with 0 representing
tails and 1 representing heads. Make the function a generator with the parameter n as the
number of coin flips.
Use the numpy library as np and the function of np.random.randint() in order to simulate a
coin flip. After defining this function, return the result as a List using List Comprehension
discussed previously.

*Note: numpy.random.randint([start], stop)

Vectors, Matrices and
Computation
Python Programming
Array Range
Declaring an Array containing a Range Using the coin_flip() from Challenge 2
Vector Computations
Vector to Scalar Computation Vector to Vector: Element-wise Multiplication

Vector to Vector: Dot Product *Note: Dot product multiples the elements on
the same index and sums the result

[ 0 1 2 3 4 ]
x [ 5 6 7 8 9 ]
0 + 6 + 14 + 24 + 36 = 80
Challenge 3
Python Programming
Challenge 3: Compute the mean of 1000 coin flips
Using the coin_flip() function from Challenge 2, compute for the result of 1000 coin flips.
Recall that the mean is computed thus:

*Note: Since a coin flip would randomize from a value of 0 and 1, as the number
of flips increase, the mean of the result should get closer to about 0.5
Matrix Computations
Matrix Initialization Matrix to Matrix: Matrix Multiplication

Matrix to Scalar: Element-wise Multiplication

Advanced Matrix Operations
The SciPy Library of Python contains easy-to-use function for formulas and computations
used in mathematics, Science, and Engineering. The core packages used are not limited to
NumPy, SciPy Library, Matplotlob, Ipython, Sympy, and Pandas.

*Note: The following SciPy code on the left

solves for an ordinary or generalized
eigenvalue problem of a square
matrix varMat

Visit the SciPy home page from the

url below for more information:
https://scipy.org/
Pandas: Python Data
Analysis Library
Python Programming
Introducing Pandas
Reading CSV files and storing them in a variable

*Note: Pandas is a Python library which

Row and column count allows the use of data-structures
and analysis tools.

Please use the movie_metadata.csv

Enumerating the column names dataset provided. You may also
obtain the dataset from:

https://www.kaggle.com/deepmatri
x/imdb-5000-movie-dataset
Slicing Data Frames
Slicing Data
Indexing Columns and Rows
Indexing Columns Indexing Rows

Indexing Columns from a List

*Note: data.loc[var] uses labels as parameters

as opposed to using data.iloc[var] or
data[var] which uses indexes
Finding Movies by James Cameron

*Note: data['director_name’] is also the same

as data.director_name

The code tries to obtain all movies

within the dataset where the
director_name is James Cameron
Sort Films by Gross Earnings

*Note: Sorting values are helpful in being

able to arrange which ones go first.
With the inclusion of slicing, there
are plenty of applications and
queries one could implement with
the use of these two functions
together in one code
Challenge 4!
Python Programming
Challenge 4: Get the Top 5 films of Michael Bay
Create a variable named sorted_data which would contain the sorted values by gross of
each film of Michael Bay. Output the top 5 films into a table. Your answer should be similar
to that of the output below.

*Note: You should be able to obtain an output

similar to that of the table on the left.

You would need to implement a form of

sorting as well as slicing in order to
complete this challenge
Multiple Conditions
Find Canadian films that have Hugh Jackman as the actor_1_name

*Note: There is a difference between

the syntax and as well as the
symbol &, it Is that the syntax
and does a Boolean operation
and checks if both statements
are True then the result would
be True, while the symbol &,
does a Bit-wise AND operator
Challenge 5!
Python Programming
Challenge 5: Actor with the movie that grossed an exact value
Find the actor who is the actor_1_name for the movie that grossed exactly 67344392. The
resulting output should be similar to the one below.

Once completed, try to find the films with whose actor_3_name is Piolo Pascual and the
actor_1_name is the person from Armageddon

*Note: You may check the previously discussed

topics on Multiple Conditions for Pandas
Preprocessing
Python Programming
Datatypes of the Columns

*Note: Since the datatypes

are automatically
labelled as they are
read from the dataset,
numbers are typically
assigned to have a
float64 value even if
they do not possess
any fractions
Filling up Blank values
Converting actor_1_facebook from a datatype of float64 to int64
*Note: An error has occurred due to
the reason that conversion
cannot be done when at least
one of the values are NA or
Infinite. If this happens, fill up
or initialize the NA values
before converting

The parameter used by fillna()

replaces all NA values to the
specified value
Lambda or the Nameless Function
Applying a Lambda

*Note: Lambda is commonly used in the

event that a function would only
be used once as opposed to it
being continuously reused.
This better utilized the memory
being used by the application.
Cleaning the Dataset

*Note: The \xc2\xa0 after

each title are unicode
Escape Characters

*Note: The problem with this

statement is that it removes the ñ
Cleaning the Dataset

*Note: The problem with this

statement is that it removes the ñ
Data Summaries
Referencing the original data Saving the output to a CSV file

*Note: When adding a new reference to the

original data, only new cells are allocated
memory, while the unchanged cells are
referenced to the original

The describe function allows an

easy-to-use summarization of aggregated
date
Data Correlations

*Note: When using a dataframe from

Pandas, such as that of the
variable cleaned_data, you
may also look for a number of
functions that are readily
available. What is shown here
is how to easily obtain the
correlation of one column to
the other
Outliers: Clipping to the nth percentile
Removing outliers may be adjusted based on the percentile.

*Note: Clipping to the 99th percentile is just

one of the many rules-of-thumb used in
practice. However it does not always work,
especially if there are too many outliers.
The percentile should be adjusted based on
the dataset being used
Aggregation: Dataframe Grouping by title_year

*Note:
*Note: There are also readily
available aggregation
functions present such
as size(), mean(), and
Group together the title_year as the index the like
Data Visualization
Python Programming
Matplotlib: Line Plot of Average Facebook Likes per Year
Importing Seaborn

Average Facebook Likes per Year

*Note: Plot takes in the x-axis as the first

parameter and the second, the y-axis values

figsize=(15,8) is the image size for the

plot
Matplotlib: Scatterplot of Gross vs Budget

*Note: plt.figure(figsize=(15,8)) is
used to render the size of
the graph. You may try to
play with the parameters
in the size in order to
create a better fit for your
visualization
Matplotlib: Histogram of IMDB Scores

*Note: When using Matplotlib, you may

modify a number of parameters in
your graphs. In this example, the bin
is set to bin=20 however if you set
that to bin=50, you would get a
different data summarization for your
visualization. You may press SHIFT +
TAB in order to see other parameters
you can modify
Pandas: Overlay Barplot and Line Graph
Barplot of the gross earning of the 10 movies with the highest budget superimposed with
their budget as a line graph
Histograms of imdb_scored
Histograms of imdb_scored according to the
different content rating types

*Note: Using overlay graphs allow better

differentiation and comparison of columns.
Take note that some data may only be
comparable when normalized.
Scatter Matrix

*Note: Scatter Matrix are good to use when trying

to determine the correlation of two columns
in the dataset
Seaborn: Conditional Formatting

*Note: Seaborn allows better visualization of

graphs and one of its useful functions
is to render the visualization with
conditional formatting to easily
determine the difference in values of
your dataset
Advanced Graphs: Plotting more than 2 variables

*Note: This visualization shows 4

variables. The red nodes
designate Fantasy while the
green nodes are movies which
are not classified as Fantasy. The
size of each node is proportional
to the gross of the movie. The x-
axis shows the number of
Facebook Likes. Lastly, the y-axis
shows the IMDB Score
Seaborn: Line Plot with Regression
Voted Users and Reviews

*Note: This visualization shows 4 variables. The red

nodes designate Fantasy while the green
nodes are movies which are not classified
as Fantasy. The size of each node is
proportional to the gross of the movie. The
x-axis shows the number of Facebook Likes.
Lastly, the y-axis shows the IMDB Score
Challenge 6!
Python Programming
Challenge 6: Fix the plot by clipping with the Tukey's Test (k=1)

Answer :

*Note: The original output of the code would

result in a graph that is You may have to
research about the function np.clip() which
limits the value in an array.
Seaborn: Barplot Gross earnings of the last 10 years

*Note: The list of years

were obtained first
and saved in a list as
last_10_years then it
was used as the y-axis
labels
Challenge 7!
Python Programming
Challenge 7: Get the top 10 directors with most movies
directed and use a boxplot for their gross earnings
The result of your code should be similar to this:
*Note: You may have to use groupings,
sortings, and slicing in order to be
able to complete this task.

Try to obtain each parameter

needed one at a time in order to
build your data. Understanding
which one is aggregated, sorted,
and sliced to get the top 10.