Unit-I

Big Data Analytics – An Introduction

 Big Data - Definition
• Big Data is data whose scale, distribution,
diversity, and/or timeliness require the use of
new technical architectures and analytics to
enable insights that unlock new sources of
business value.
 Three attributes of Big data
Characteristics

• Huge volume of data (for instance, tools that can

manage billions of rows and billions of columns)

• Complexity of data types and structures, with an

increasing volume of unstructured data (80-90%
of the data in existence is unstructured)

• Speed or velocity of new data creation

 Key characteristics of Big Data
• Data Volume
44x increase from 2010 to 2020
(1.2zb to 35.2 zb)
• Processing Complexity
 changing data structures
Use cases warranting additional transformations
and analytical techniques
• Data Structure
Greater variety of data structures to mine and
analyze
Big Data Characteristics : Data Structures
Data growth is increasingly Unstructured
Structured
• Data containing defined data type ,
format, structure
• Eg : Transaction data , OLAP
Semi-structured
Structured • Textual data files with a discernable
pattern, enabling parsing
• Eg: XML data files that are self
describing and defined by an xml
schema
Semi - Structured “Quasi” structured
• Texual data with erratic data formats,
can be formatted with effort, tools
and time
• Eg: Web click stream data
Unstructured
“Quasi” Structured • Data that has no inherent structure
and is usually stored as different
types of files
• Text docs, PDFs, images and videos
Unstructured
 Examples of data structures
• Structured data: Data containing a defined data type, format, and
structure (that is,transaction data, online analytical processing
[OLAP] data cubes, traditional RDBMS, CSV files, and even simple
spreadsheets).

• Semi-structured data: Textual data files with a discernible pattern

that enables parsing (such as Extensible Markup Language [XML]
data files that are self-describing and defined by an XML schema).

• Quasi-structured data: Textual data with erratic data formats that

can be formatted with effort, tools, and time (for instance, web
click stream data that may contain inconsistencies in data values
and formats).

• Unstructured data: Data that has no inherent structure, which may

include text documents, PDFs, images, and video.
Introduction to R
 Programming features of R
• Almost everything is a object
• Almost everything is a vector
• All commands are functions
Eg : quit() or q(), not q
• Some commands produce different output
much like virtual functions in OO languages
• Have large number of default arguments
 Getting help in R
• apropos returns a character vector giving the
names of all objects in the search list that
match your(potentially partial) enquiry
• help.start() command displays the html help
manual
• To see worked examples, type say
example(lm), in which you can see graphical
output produced by lm.
Basic R commands
Basic R commands
R GUI
 R GUI
• RStudio provides both a desktop and a Web
browser interface
• RStudio offers three panes that are fairly
common to all R GUIs The upper left pane is
for script editing. The lower left pane is the R
console itself, where all commands are
executed.
• The lower right pane is the help screen,
invoked by the help(<topic>) command
 Getting Data Into and Out of R
• Getting data into R
Type it in!
Read from a data file
Read from a database

• Getting Data out of R

 Save in a workspace
 Write a text file
 Save an object to the file system(can save plots as
well)
 Getting data into R
• R supports multiple file formats
 read.table() is the main function
• File name can be a URL
read. table(“http://localhost/file.csv”, sep=“ ,“) is
same as the read.csv(…)
• Can read directly from a database via ODBC
interface
Mydb<- odbcConnect(“My HadoopR”, …..)
• R pakages exist to read data from Hadoop and
HDFS
 Getting out of R
Options R Code

Save it as a part of your work save.image(file=“dfm.Rdata”)

space save.image()
load.image(“dfm.Rdata”)`
Save it as a data file write.csv(dfm,file=“dfm.csv”)

Save it as an R object save (UCBAdmissions, file =“UCBadmin.Rdata”)

Load(file=“UCBadmin.Rdata”)

Plots can be saved as images saveplot(filename=“filename.ext”,type=“type”)

Importing Excel data into R
 R

Attributes and Data types

 Attributes
• Categorized into four types(NOIR):
Nominal Catergorical attributes

Ordinal
Interval Numeric atrributes

Ratio
 NOIR Attributes types
Nominal Ordinal Interval Ratio

Definition The values Attributes imply The difference Both the

represent a sequence. between two difference and
labels that values is the ratio of two
distinguish meaningful values are
one from another. meaningful.

Examples ZIP codes, Quality of Temperature in Age, temperature

nationality, diamonds, Celsius or in Kelvin,
street names, academic grades, Fahrenheit, counts, length,
gender, magnitude of calendar dates, weight
employee ID earthquakes latitudes
numbers,
TRUE
or
FALSE

Operations =, ≠ =,≠,<,≤,>,≥ =,≠,<,≤,>,≥,+,- =,≠,<,≤,>,≥,+,-,×,÷

 Numeric, character, and logical data
types
• i <- 1 # create a numeric variable
• sport <- “football” # create a character variable
• flag <- TRUE # create a logical variable
• class(i) # returns “numeric”
• typeof(i) # returns “double”
• class(sport) # returns “character”
• typeof(sport) # returns “character”
• class(flag) # returns “logical”
• typeof(flag) # returns “logical”
• is.integer(i) # returns FALSE
• j <- as.integer(i) # coerces contents of i into an integer
• is.integer(j) # returns TRUE
• length(i) # returns 1
• length(flag) # returns 1
• length(sport) # returns 1 (not 8 for “football”)
 Vectors
• Vectors are a basic building block for data in R.
As seen previously, simple R variables are
actually vectors. A vector can only consist of
values in the same class. The tests for vectors
can be conducted using the is.vector()
function.
• is.vector(i) # returns TRUE
• is.vector(flag) # returns TRUE
• is.vector(sport) # returns TRUE
 Vectors
• The following R code illustrates how a vector
can be created using the combine function, c()
or the colon operator, to build a vector from
the sequence of integers from 1 to 5.
• Furthermore, the code shows how the values
of an existing vector can be easily modified or
accessed.
 Vectors
• u <- c(“red”, “yellow”, “blue”) # create a vector “red” “yellow” “blue”
• u # returns “red” “yellow” “blue”
• u[1] # returns “red” (1st element in u)
• v <- 1:5 # create a vector 1 2 3 4 5
• v # returns 1 2 3 4 5
• sum(v) # returns 15
• w <- v * 2 # create a vector 2 4 6 8 10
• w # returns 2 4 6 8 10
• w[3] # returns 6 (the 3rd element of w)
• z <- v + w # sums two vectors element by element
• z # returns 3 6 9 12 15
• z > 8 # returns FALSE FALSE TRUE TRUE TRUE
• z[z > 8] # returns 9 12 15
• z[z > 8 | z < 5] # returns 3 9 12 15 (“|” denotes “or”)
 Arrays and Matrices
• The array() function can be used to restructure a vector as an array.
# the dimensions are 3 regions, 4 quarters, and 2 years
>quarterly_sales <- array(0, dim=c(3,4,2))
>quarterly_sales[2,1,1] <- 158000
>quarterly_sales
,,1
[,1] [,2] [,3] [,4]
[1,] 0 0 0 0
[2,] 158000 0 0 0
[3,] 0 0 0 0
,,2
[,1] [,2] [,3] [,4]
[1,] 0 0 0 0
[2,] 0 0 0 0
[3,] 0 0 0 0

>sales_matrix <- matrix(0, nrow = 3, ncol = 4)

sales_matrix
[,1] [,2] [,3] [,4]
[1,] 0 0 0 0
[2,] 0 0 0 0
[3,] 0 0 0 0
 Data Frames
• Data frames provide a structure for storing
and accessing several variables of possibly
different data types. In fact, as the
is.data.frame() function indicates, a data
frame was created by the read.csv() function
 List and factors
• Lists can contain any type of objects, including
other lists.
• Factors were briefly introduced during the
discussion of the gender variable in the data
frame sales . In this case, gender could assume
one of two levels: F or M. Factors can be ordered
or not ordered. In the case of Gender, the levels
are not ordered.
>class(sales$gender) # returns “factor”
>is.ordered(sales$gender) # returns FALSE
 Contigency table
• In R, table refers to a class of objects used to
store the observed counts across the factors
for a given dataset.
• Such a table is commonly referred to as a
contingency table and is the basis for
performing a statistical test on the
independence of the factors used to build the
table.
 Descriptive Statistics
• The following code provides some common R functions
that include descriptive statistics.
# to simplify the function calls, assign
x <- sales$sales_total
y <- sales$num_of_orders
cor(x,y) # returns 0.7508015 (correlation)
cov(x,y) # returns 345.2111 (covariance)
IQR(x) # returns 215.21 (interquartile range)
mean(x) # returns 249.4557 (mean)
median(x) # returns 151.65 (median)
range(x) # returns 30.02 7606.09 (min max)
sd(x) # returns 319.0508 (std. dev.)
var(x) # returns 101793.4 (variance)
• The IQR() function provides the difference between the
third and the first quartiles.
 Exploratory Data Analysis
• Functions such as summary() can help analysts easily get an idea of the
magnitude and range of the data, but other aspects such as linear
relationships and distributions are more difficult to see from descriptive
statistics.
• For example, the following code shows a summary view of a data frame
data with two columns x and y . The output shows the range of x and y ,
but it’s not clear what the relationship may be between these two
variables.
summary(data)
xy
Min. :-1.90483 Min. :-2.16545
1st Qu.:-0.66321 1st Qu.:-0.71451
Median : 0.09367 Median :-0.03797
Mean : 0.02522 Mean :-0.02153
3rd Qu.: 0.65414 3rd Qu.: 0.55738
Max. : 2.18471 Max. : 1.70199
 Exploratory Data Analysis
• A useful way to detect patterns and anomalies in the
data is through the exploratory data analysis with
visualization.
• Visualization gives a succinct, holistic view of the data
that may be difficult to grasp from the numbers and
summaries alone.
• Variables x and y of the data frame data can instead be
visualized in a scatter plot which easily depicts the
relationship between two variables.
• An important facet of the initial data exploration,
visualization assesses data cleanliness and suggests
potentially important relationships in the data prior to
the model planning and building phases.
 Exploratory Data Analysis
• Exploratory data analysis is a data analysis
approach to reveal the important
characteristics of a dataset, mainly through
visualization.
 Visualization Before Analysis
• To illustrate the importance of visualizing data, consider
Anscombe’s quartet. Anscombe’s quartet consists of four
datasets, to demonstrate the importance of graphs in
statistical analyses.
• Each dataset is plotted as a scatterplot, and the fitted lines
are the result of applying linear regression models.
• The estimated regression line fits Dataset 1 reasonably
well. Dataset 2 is definitely nonlinear.
• Dataset 3 exhibits a linear trend, with one apparent outlier
at . For Dataset 4, the regression line fits the dataset quite
well.
Visualization Before Analysis
 Dirty Data
• Dirty data can be detected in the data
exploration phase with visualizations. In
general, analysts should look for anomalies,
verify the data with domain knowledge, and
decide the most appropriate approach to
clean the data
• Consider a scenario in which a bank is
conducting data analyses of its account
holders to gauge customer retention.
Dirty Data
 Dirty Data
• The left side of the graph shows a huge spike of
customers who are zero years old or have
negative ages. This is likely to be evidence of
missing data.
• Or it might be caused by transferring data among
several systems that have different definitions for
null values (such as NULL, NA, 0, –1, or –2).
Therefore, data cleansing needs to be performed
over the accounts with abnormal age values.
 Dirty Data
• Data analysts should perform sanity checks
against domain knowledge and decide if the
dirty data needs to be eliminated.
• Dirty data can occur due to acts of omission.
 Visualizing a Single Variable
• Using visual representations of data is a hallmark of
exploratory data analyses: letting the data speak to
its audience rather than imposing an interpretation
on the data a priori.
• R has many functions available to examine a single
variable. Some of these functions are
Dotchart and Barplot
Histogram and Density Plot
Density plots of (a) diamond prices and (b)
the logarithm of diamond prices
 Examining Multiple variables
• A scatterplot is a simple and widely used
visualization for finding the relationship among
multiple variables.
• A scatterplot can represent data with up to five
variables using x-axis, y-axis, size, color, and
shape. But usually only two to four variables are
portrayed in a scatterplot to minimize confusion.
• When examining a scatterplot, one needs to pay
close attention to the possible relationship
between the variables.
 Examining Multiple variables
Box-and-Whisker Plot
• Box-and-whisker plots show the distribution of a continuous
variable for each value of adiscrete variable.
 Examining Multiple variables
Hexbinplot for Large Datasets
• The scatterplot as a popular visualization can visualize data
containing one or more variables. But one should be careful about
using it on high-volume data.
• If there is too much data, the structure of the data may become
difficult to see in a scatterplot. Consider a case to compare the
logarithm of household income against the years of education.
• The cluster in the scatterplot on the left (a) suggests a somewhat
linear relationship of the two variables. However, one cannot really
see the structure of how the data is distributed inside the cluster.
This is a Big Data type of problem. Millions or billions of data points
would require different approaches for exploration, visualization,
and analysis.
Examining Multiple variables – (a)
Scatterplot and (b) Hexabinplot
 Data Exploration vs Presentation
• Using visualization for data exploration is
different from presenting results to
stakeholders.
• Density plots are fairly technical, and they
contain so much information that they would
be difficult to explain to less technical
stakeholders.
Density Plot
Histograms based on Density plot
 Statistical Methods For Evaluation
• Visualization is useful for data exploration and
presentation
• Statistical techniques are used during the
initial data exploration and data preparation,
model building, evaluation of the final models,
and assessment of how the new models
improve the situation when deployed in the
field.
 Statistical Methods For Evaluation
• In particular, statistics can help answer the following
questions for data analytics:
• Model Building and Planning
• What are the best input variables for the model?
• Can the model predict the outcome given the input?
• Model Evaluation
• Is the model accurate?
• Does the model perform better than an obvious guess?
• Does the model perform better than another candidate model?
• Model Deployment
• Is the prediction sound?
• Does the model have the desired effect (such as reducing the
cost)?
 Hypothesis Testing
• When comparing populations, such as testing or
evaluating the difference of the means from two
samples of data, a common technique to assess
the difference or the significance of the
difference is hypothesis testing.
• The basic concept of hypothesis testing is to form
an assertion and test it with data.
• When performing hypothesis tests, the common
assumption is that there is no difference between
two samples.
 Hypothesis Testing
• This assumption is used as the default position for building the test
or conducting a scientific experiment. Statisticians refer to this as
the null hypothesis(H0).
• The alternative hypothesis (HA ) is that there is a difference
between two samples.
• For eg., if the task is to identify the effect of drug A compared to
drug B on patients, the null hypothesis and alternative hypothesis
would be this.
H0: Drug A and drug B have the same effect on patients.
HA : Drug A has a greater effect than drug B on patients.
• If the task is to identify whether advertising Campaign C is effective
on reducing customer churn, the null hypothesis and alternative
hypothesis would be as follows.
H0: Campaign C does not reduce customer churn better than the current
campaign
method.
HA : Campaign C does reduce customer churn better than the current
campaign.
 Hypothesis Testing
Ex. Null Hypotheses and Alternative Hypotheses
 Hypothesis Testing
• The null hypothesis is that the proposed model
does not predict better than the existing model.
• The alternative hypothesis is that the proposed
model indeed predicts better than the existing
model.
• The null hypothesis could be that the new
algorithm does not produce better
recommendations than the current algorithm
being deployed.
• The alternative hypothesis is that the new
algorithm produces better recommendations
than the old algorithm.
 Hypothesis Testing
• The null hypothesis is that the coefficient is zero,
which means the variable does not have an
impact on the outcome.
• The alternative hypothesis is that the coefficient
is nonzero, which means the variable does have
an impact on the outcome.
• A common hypothesis test is to compare the
means of two populations. Two such hypothesis
tests are
• Difference of Means
 Student’s t-test
 Welch’s t-test
 Difference of Means
• Hypothesis testing is a common approach to
draw inferences on whether or not the two
populations, are different from each other.
• Specifically, the two hypothesis tests in the
following null and alternative hypotheses.
• H0 : µ1 = µ2
• HA : µ1 ≠ µ2
 Difference of Means
• The difference in means can be tested using Student’s t-
test or the Welch’s t-test.
Student’s t-test
• Student’s t-test assumes that distributions of the two
populations have equal but unknown variances.
Suppose n1 and n2 samples are randomly and
independently selected from two populations, pop1 and
pop2, respectively.
• If each population is normally distributed with the same
mean ( µ1 = µ2 ) and with the same variance, then T (the
t-statistic), follows a t-distribution n1 + n2 – 2 with
degrees of freedom (df).
T= X1’ – X2’
sp√(1 /n1+ 1/n2)
 Difference of Means
Welch’s t-test
• When the equal population variance assumption is
not justified in performing Student’s t-test for the
difference of means, Welch’s t-test can be used
based on T expressed in
Twelch = X1 ’ – X2 ’
√(s12 /n1+ s22/n2)
Where X1’ , X2’, s1, s2, n1, n2 correspond to the i-th
sample mean, sample variance, and sample size.
 Wilcoxon Ranksum Test
• A t-test represents a parametric test in that it
makes assumptions about the population
distributions from which the samples are drawn.
If the populations cannot be assumed or
transformed to follow a normal distribution, a
nonparametric test can be used.
• The Wilcoxon rank-sum test is a nonparametric
hypothesis test that checks whether two
populations are identically distributed.
 Type I and Type II Errors
• A type I error is the rejection of the null hypothesis
when the null hypothesis is TRUE. The probability of
the type I error is denoted by the Greek letter α.
• A type II error is the acceptance of a null hypothesis
when the null hypothesis is FALSE. The probability of
the type II error is denoted by the Greek letter β .
 ANOVA
• The hypothesis tests presented in the previous sections are
good for analyzing means between two populations. But
what if there are more than two populations?
• Consider an example of testing the impact of nutrition and
exercise on 60 candidates between age 18 and 50. The
candidates are randomly split into six groups, each assigned
with a different weight loss strategy, and the goal is to
determine which strategy is the most effective.
 ANOVA
• Group 1 only eats junk food.
• Group 2 only eats healthy food.
• Group 3 eats junk food and does cardio exercise every other
day.
• Group 4 eats healthy food and does cardio exercise every
other day.
• Group 5 eats junk food and does both cardio and strength
training every other day.
• Group 6 eats healthy food and does both cardio and strength
training every other day.
 ANOVA
• Multiple t-tests could be applied to each pair of weight loss
strategies. In this example, the weight loss of Group 1 is
compared with the weight loss of Group 2, 3, 4, 5, or 6.
• Similarly, the weight loss of Group 2 is compared with that of the
next 4 groups. Therefore, a total of 15 t-tests would be
performed.
• multiple t-tests may not perform well on several populations for
two reasons.
• First, because the number of t-tests increases as the number of
groups increases, analysis using the multiple t-tests becomes
cognitively more difficult. Second, by doing a greater number of
analyses, the probability of committing at least one type I error
somewhere in the analysis greatly increases.
 ANOVA
• Analysis of Variance (ANOVA) is designed to
address these issues. ANOVA is a
generalization of the hypothesis testing of the
difference of two population means.
• ANOVA tests if any of the population means
differ from the other population means. The
null hypothesis of ANOVA is that all the
population means are equal.