Unit 1
Unit 1
Ans;
• At its core, data science aims to discover and extract actionable knowledge
from data that can be used to make sound business decisions and predictions.
• Data science uses advanced analytical theory and various methods such as
time series analysis for predicting future.
Ans. Structured data is arranged in rows and column format. It helps for
application to retrieve and process data easily. Database management system is
used for storing structured data. The term structured data refers to data that is
identifiable because it is organized in a structure.
Ans. Unstructured data is data that does not follow a specified format. Row and
columns are not used for unstructured data. Therefore it is difficult to retrieve
required information. Unstructured data has no identifiable structure.
2. Retrieving data
3. Data preparation
4. Data exploration
5. Data modeling
ii. Data isolation allows for easier and faster data reporting.
Ans. Data cleaning means removing the inconsistent data or noise and
collecting necessary information of a collection of interrelated data.
Q.13 What are the three challenges to data mining regarding data mining
methodology?
Ans. Challenges to data mining regarding data mining methodology include the
following:
Ans. Predictive mining tasks perform inference on the current data in order to
make predictions. Predictive analysis provides answers of the future queries that
move across using historical data as the chief principle for decisions.
Ans. Data cleaning means removing the inconsistent data or noise and
collecting necessary information of a collection of interrelated data.
Q.16 List the five primitives for specifying a data mining task.
Ans. :
Ans. Data repository is also known as a data library or data archive. This is a
general term to refer to a data set isolated to be mined for data reporting and
analysis. The data repository is a large database infrastructure, several databases
that collect, manage and store data sets for data analysis, sharing and reporting.
Syllabus
Types of Data - Types of Variables - Describing Data with Tables and Graphs -
Describing Data with Averages - Describing Variability - Normal Distributions
and Standard (z) Scores.
Types of Data
• Data is collection of facts and figures which relay something specific, but
which are not organized in any way. It can be numbers, words, measurements,
observations or even just descriptions of things. We can say, data is raw
material in the production of information.
• Collection of data objects and their attributes. Attributes captures the basic
characteristics of an object
• Each row of a data set is called a record. Each data set also has multiple
attributes, each of which gives information on a specific characteristic.
• Data can broadly be divided into following two types: Qualitative data and
quantitative data.
Qualitative data:
1. Nominal data
2. Ordinal data
Qualitative data:
• There are two types of qualitative data: Interval data and ratio data.
1. Advantages:
customers.
• Avoid pre-judgments
2. Disadvantages:
• Time consuming
2. Disadvantages:
Ranked Data
• Ranked data is a variable in which the value of the data is captured from an
ordered set, which is recorded in the order of magnitude. Ranked data is also
called as Ordinal data.
c) Along with the information provided by the nominal scale, ordinal scales
give the rankings of those variables
e) The surveyors can quickly analyze the degree of agreement concerning the
identified order of variables
• Examples:
Scale of Measurement
• There are four different scales of measurement. The data can be defined as
being one of the four scales. The four types of scales are: Nominal, ordinal,
interval and ratio.
Nominal
• A nominal data usually deals with the non-numeric variables or the numbers
that do not have any value. While developing statistical models, nominal data
are usually transformed before building the model.
3. The numbers don't define the object characteristics. The only permissible
aspect of numbers in the nominal scale is "counting".
• Example:
Interval
a) The interval data is quantitative as it can quantify the difference between the
values.
c) To understand the difference between the variables, you can subtract the
values between the variables.
d) The interval scale is the preferred scale in statistics as it helps to assign any
numerical values to arbitrary assessment such as feelings, calender types, etc.
• Examples:
1. Celsius temperature
2. Fahrenheit temperature
Ratio
• Any variable for which the ratios can be computed and are meaningful is
called ratio data.
d) Ratio data has unique and useful properties. One such feature is that it allows
unit conversions like kilogram - calories, gram - calories, etc.
(c) age
(f) temperature
(i) IQ score
(j) gender
Solution :
Types of Variables
Discrete variables:
• The word discrete means countable. For example, the number of students in a
class is countable or discrete. The value could be 2, 24, 34 or 135 students, but
it cannot be 23/32 or 12.23 students.
• Number of page in the book is a discrete variable. Discrete data can only take
on certain individual values.
Continuous variables:
• Continuous variables are a variable which can take all values within a given
interval or range. A continuous variable consists of numbers whose values, at
least in theory, have no restrictions.
• Example of continuous variables is Blood pressure, weight, high and income.
• Continuous data can take on any value in a certain range. Length of a file is a
continuous variable.
Approximate Numbers
• For example, 2, 4, 9 are exact numbers as they do not need any approximation.
• The two main variables in an experiment are the independent and dependent
variable. An experiment is a study in which the investigator decides who
receives the special treatment.
1. Independent variables
• The independent variable is the one that the researcher intentionally changes
or controls.
2. Dependent variables
• The dependent variable is the factor that the research measures. It changes in
response to the independent variable or depends upon it.
Observational Study
• These studies are often qualitative in nature and can be used for both
exploratory and explanatory research purposes. While quantitative observational
studies exist, they are less common.
• Observational studies are generally used in hard science, medical and social
science fields. This is often due to ethical or practical concerns that prevent the
researcher from conducting a traditional experiment. However, the lack of
control and treatment groups means that forming inferences is difficult and
there is a risk of confounding variables impacting user analysis.
Confounding Variable
• Confounding variables are those that affect other variables in a way that
produces spurious or distorted associations between two variables. They
confound the "true" relationship between two variables. Confounding refers to
differences in outcomes that occur because of differences in the baseline risks of
the comparison groups.
• A difference between groups might be due not to the independent variable but
to a confounding variable.
• Consider the example, in order to conduct research that has the objective that
alcohol drinkers can have more heart disease than non-alcohol drinkers such
that they can be influenced by another factor. For instance, alcohol drinkers
might consume cigarettes more than non drinkers that act as a confounding
variable (consuming cigarettes in this case) to study an association amidst
drinking alcohol and heart disease.
• For example, suppose a researcher collects data on ice cream sales and shark
attacks and finds that the two variables are highly correlated. Does this mean
that increased ice cream sales cause more shark attacks? That's unlikely. The
more likely cause is the confounding variable temperature. When it is warmer
outside, more people buy ice cream and more people go in the ocean.
• In order to find the frequency distribution of quantitative data, we can use the
following table that gives information about "the number of smartphones owned
per family."
• When observations are sorted into classes of single values, the result is
referred to as a frequency distribution for ungrouped data. It is the
representation of ungrouped data and is typically used when we have a smaller
data set.
1. Grouped data:
• Grouped data refers to the data which is bundled together in different classes
or categories.
• Data are grouped when the variable stretches over a wide range and there are a
large number of observations and it is not possible to arrange the data in any
order, as it consumes a lot of time. Hence, it is pertinent to convert frequency
into a class group called a class interval.
• Suppose we conduct a survey in which we ask 15 familys how many pets they
have in their home. The results are as follows:
1, 1, 1, 1, 2, 2, 2, 3, 3, 4, 5, 5, 6, 7, 8
• The real limits are located at the midpoint of the gap between adjacent tabled
boundaries; that is, one-half of one unit of measurement below the lower tabled
boundary and one-half of one unit of measurement above the upper tabled
boundary.
• Table 2.3.4 gives a frequency distribution of the IQ test scores for 75 adults.
• If the lower class limit for the second class, 95, is added to the upper class
limit for the first class,94 and the sum divided by 2, the upper boundary for
the first class and the lower boundary for the second class is determined. Table
2.3.5 gives all the boundaries for Table 2.3.5.
• If the lower class limit is added to the upper class limit for any class and the
sum divided by 2, the class mark for that class is obtained. The class mark for a
class is the midpoint of the class and is sometimes called the class midpoint
rather than the class mark.
Example 2.3.1: Following table gives the frequency distribution for the
cholesterol values of 45 patients in a cardiac rehabilitation study. Give the
lower and upper class limits and boundaries as well as the class marks for
each class.
• Solution: Below table gives the limits, boundaries and marks for the classes.
Example 2.3.2: The IQ scores for a group of 35 school dropouts are as
follows:
b) Specify the real limits for the lowest class interval in this frequency
distribution.
(123-69)/ 10=54/10=5.4≈ 5
Example 2.3.3: Given below are the weekly pocket expenses (in Rupees) of
a group of 25 students selected at random.
37, 41, 39, 34, 41, 26, 46, 31, 48, 32, 44, 39, 35, 39, 37, 49, 27, 37, 33, 38, 49,
45, 44, 37, 36
Solution:
• In the given data, the smallest value is 26 and the largest value is 49. So, the
range of the weekly pocket expenses = 49-26=23.
Outliers
• 'In statistics, an Outlier is an observation point that is distant from other
observations.'
• An outlier is a value that escapes normality and can cause anomalies in the
results obtained through algorithms and analytical systems. There, they always
need some degrees of attention.
• Understanding the outliers is critical in analyzing data for at least two aspects:
• The simplest way to find outliers in data is to look directly at the data table,
the dataset, as data scientists call it. The case of the following table clearly
exemplifies a typing error, that is, input of the data.
• The field of the individual's age Antony Smith certainly does not represent the
age of 470 years. Looking at the table it is possible to identify the outlier, but it
is difficult to say which would be the correct age. There are several possibilities
that can refer to the right age, such as: 47, 70 or even 40 years.
• A relative frequency distribution lists the data values along with the percent
of all observations belonging to each group. These relative frequencies are
calculated by dividing the frequencies for each group by the total number of
observations.
• Example: Suppose we take a sample of 200 India family's and record the
number of people living there. We obtain the following:
Cumulative frequency:
• A cumulative frequency distribution can be useful for ordered data (e.g. data
arranged in intervals, measurement data, etc.). Instead of reporting frequencies,
the recorded values are the sum of all frequencies for values less than and
including the current value.
• Example: Suppose we take a sample of 200 India family's and record the
number of people living there. We obtain the following:
• To convert a frequency distribution into a cumulative frequency distribution,
add to the frequency of each class the sum of the frequencies of all classes
ranked below it.
1. Histogram
• Here the data values only take on integer values, but we still split the range of
values into intervals. In this case, the intervals are [1,2), [2,3), [3,4), etc. Notice
that this graph is also close to being bell-shaped. A symmetric, bell-shaped
distribution is called a normal distribution.
• Notice that all the rectangles are adjacent and they have no gaps between them
unlike a bar graph.
• If we had used the percentage to make the histogram, we would call the graph
a percentage histogram.
• A relative frequency histogram is the same as a regular histogram, except
instead of the bar height representing frequency, it now represents the relative
frequency (so the y-axis runs from 0 to 1, which is 0% to 100%).
2. Frequency polygon
• We can say that frequency polygon depicts the shapes and trends of data. It
can be drawn with or without a histogram.
• Suppose we are given frequency and bins of the ages from another survey as
shown in Table 2.4.1.
• The midpoints will be used for the position on the horizontal axis and the
frequency for the vertical axis. From Table 2.4.1 we can then create the
frequency polygon as shown in Fig. 2.4.2.
• A line indicates that there is a continuous movement. A frequency polygon
should therefore be used for scale variables that are binned, but sometimes a
frequency polygon is also used for ordinal variables.
(i) What is the frequency of the class interval whose class mark is 15?
• Solution:
(iii) As the class marks of consecutive overlapping class intervals are 5, 15, 25,
35, 45, 55 we find the class intervals are 0 - 10, 10-20, 20 - 30, 30 - 40, 40 - 50,
50 - 60. Therefore, the frequency table is constructed as below.
3. Steam and Leaf diagram:
• Stem and leaf diagrams allow to display raw data visually. Each raw score is
divided into a stem and a leaf. The leaf is typically the last digit of the raw
value. The stem is the remaining digits of the raw value.
• Data points are split into a leaf (usually the ones digit) and a stem (the other
digits)
• To generate a stem and leaf diagram, first create a vertical column that
contains all of the stems. Then list each leaf next to the corresponding stem. In
these diagrams, all of the scores are represented in the diagram without the loss
of any information.
• A stem-and-leaf plot retains the original data. The leaves are usually the last
digit in each data value and the stems are the remaining digits.
• Create a stem-and-leaf plot of the following test scores from a group of college
freshmen.
• There are a couple of graphs that are appropriate for qualitative data that has
no natural ordering.
1. Bar graphs
• Bar Graphs are like histograms, but the horizontal axis has the name of each
category and there are spaces between the bars.
• Usually, the bars are ordered with the categories in alphabetical order. One
variant of a bar graph is called a Pareto Chart. These are bar graphs with the
categories ordered by frequency, from largest to smallest.
Misleading Graph
• It is a well known fact that statistics can be misleading. They are often used to
prove a point and can easily be twisted in favour of that point.
• Good graphs are extremely powerful tools for displaying large quantities of
complex data; they help turn the realms of information available today into
knowledge. But, unfortunately, some graphs deceive or mislead.
• This may happen because the designer chooses to give readers the impression
of better performance or results than is actually the situation. In other cases, the
person who prepares the graph may want to be accurate and honest, but may
mislead the reader by a poor choice of a graph form or poor graph construction.
1. Title
2. Labels on both axes of a line or bar chart and on all sections of a pie chart
4. Key to a pictograph
• A graph can be altered by changing the scale of the graph. For example, data
in the two graphs of Fig. 2.6.1 are identical, but scaling of the Y-axis changes
the impression of the magnitude of differences.
Example 2.6.1: Construct a frequency distribution for the number of
different residences occupied by graduating seniors during their college
career, namely: 1, 4, 2, 3, 3, 1, 6, 7, 4, 3, 3, 9, 2, 4, 2, 2, 3, 2, 3, 4, 4, 2, 3, 3, 5.
What is the shape of this distribution?
Solution: