UNIT-1:

WHAT IS DATA?

 Data is a set of values of subjects with respect to qualitative or

quantitative variables.
 Data is raw, unorganized facts that need to be processed. Data can be
something simple and seemingly random and useless until it is
organized.
 When data is processed, organized, structured or presented in a given
context so as to make it useful, it is called information.
 Information, necessary for research activities are achieved in different
forms.

TYPES OF DATA:

Structured Data:
Unstructured Data:
Semi-structured Data:

The data which is to the point, factual, and highly organized is referred
to as structured data. It is quantitative in nature, i.e., it is related to
quantities that means it contains measurable numerical values like
numbers, dates, and times.
It is easy to search and analyze structured data. Structured data exists in
a predefined format. Relational database consisting of tables with rows
and columns is one of the best examples of structured data. Structured
data generally exist in tables like excel files and Google Docs
spreadsheets. Structured data is highly organized and understandable for
machine language.

Unstructured Data-

All the unstructured files, log files, audio files, and image files are
included in the unstructured data. Some organizations have much data
available, but they did not know how to derive data value since the data
is raw. Unstructured data is the data that lacks any predefined model or
format. It requires a lot of storage space, and it is hard to maintain
security in it. It cannot be presented in a data model or schema. That's
why managing, analyzing, or searching for unstructured data is hard. It
 resides in various different formats like text, images, audio and video
files, etc. It is qualitative in nature and sometimes stored in a non-
relational database or NO-SQL.

Semi-structured Data-
Semi-structured data refers to data that is not captured or formatted in
conventional ways. Semi-structured data does not follow the format of a tabular data
model or relational databases because it does not have a fixed schema. However, the
data is not completely raw or unstructured, and does contain some structural elements
such as tags and organizational metadata that make it easier to analyze. The advantages
of semi-structured data is that it is more flexible and simpler to scale compared to
Structured Data.
HTML code, graphs and tables, e-mails, XML documents are examples of semi-
structured data.

Structured Data

When we talk about structured data, we are often talking about tabular
data(rectangular data) i.e. rows and columns from a database.These tables further
contain mainly two types of structured data:

1.Numerical Data Data that is expressed on a numerical scale. It is further

represented in two forms:

Continuous — Data that can undertake any value in an interval. For example,
the speed of a car, heart rate, etc.

Discrete — Data that can undertake only integer values, such as counts. For
example, the number of heads in 20 flips of a coin.

2. Categorical Data Data that can undertake only a specific set of values
representing possible categories. These are also called enums, enumerated,
factors, or nominal.
  Binary — A special case of categorical data where the features are
dichotomous i.e. can accept only 0/1 or True/False.

 Ordinal — Categorical data that has an explicit ordering. For

example, five-star rating of a restaurant(1,2,3,4,5)

 The next step is to dive deeper into structured data and how we can
use third party packages and libraries to manipulate such structures.
We have mainly two types of structures or data storage models:

1. Rectangular

2. Non-Rectangular

Rectangular Data Mostly all analyses in data science are done with a rectangular
two-dimensional data object like a dataframe, spreadsheet, CSV file, or a
database table.This mainly consists of rows that represent records(observations)
and columns(features/variables). Dataframe on the other hand is a special data
structure with a tabular format that offers super-efficient operations to
manipulate the data. Dataframes are the most commonly used data structures and
it’s important to cover a few definitions here:

Data frame Rectangular data structure (like a spreadsheet) for efficient

manipulation and application of statistical and machine learning models.

Feature A column within a dataframe is commonly referred to as a feature.

Synonyms — attribute, input, predictor, variable Outcome Many data science
projects involve predicting an outcome — often a yes/no outcome. Synonyms —
dependent variable, response, target, output.
Records A row within a dataframe is commonly referred to as a record.
Synonyms — case, example, instance, observation, pattern, sample.

Non-rectangular Data -Besides rectangular data, we have several other data

structures which come under the umbrella of non-rectangular data. Spatial data
structures, which are used in geolocation analytics, are more complex and
different from rectangular data structures. In the object representation, the focus
of the data is an object (e.g., a park) and its spatial coordinates. The field view,
by contrast, focuses on small units of space and the value of a relevant metric
(pixel intensities, for example).

Graph data structures are used to represent relationships — physical, social, and
abstract. For example, Facebook or Twitter represents connections between
people on the network as a graph of social relationships. Graph structures are
useful for certain types of problems, such as network optimization and
recommender systems. Each of these data types has a specific set of methods in
data science. The focus of this series is on rectangular data which forms the
foundational building blocks of predictive modeling.

Classification Of Data:
1.Based on Observation-
Cross Sectional Data: Cross-section data is collected in a single time period and
is characterized by individual units - people, companies, countries, etc. Some
examples include:

 Student grades at the end of the current semester;

 Household data of the previous year - expenditure on food,
unemployment, income, etc.
  Car data - average speed, horsepower, color, etc.

With cross-sectional data the ordering of the data does not matter. In other
words, we can order the data by ascending, descending or even randomized
order and this will not affect out modeling results.
Time Series Data: Data collected at a number of specific points in time is called
time series data. Such examples include stock prices, interest rates, exchange
rates as well as product prices, GDP, etc. Time series data can be observed at
many different frequencies (hourly, daily, weekly, monthly, quarterly, anually,
etc.).Unlike cross-sectional data, the ordering of the data is important in time-
series data. Each point represents the values at specific points in time. As
such, time series data are typically presented in chronological order. Changing
the order of the data ignores the time-dimensionality of the data.
Panel Data:Panel data combines cross-sectional and time series data:
the same individuals (persons, firms, cities, etc.) are observed at several
points in time (days, years, before and after treatment etc.). Panel data allows
you to control for variables you cannot observe or measure like:

 cultural (like country or region specific) factors;

 difference in business practices across companies;

2.Based on Measurement-
Nominal: If the values in a variable does not follow any particular order, we can
call it as nominal. Taking a mean or median is meaningless here. Note that
sorting the values of a nominal data type does not make any difference.

Ordinal: If the values in a variable follows a particular order, then we can call it
as ordinal. This means a lower value present in the feature holds lesser weight
compared to a higher value. Hence, sorting the values of an ordinal data makes
sense. For example,

 Ratings on a scale of 10: 1 being lowest and 10 being highest

Interval:
In interval type, 0 doesn’t have a true meaning. In the case of temperature, 0
doesn’t mean no temperature. Instead, it is a valid value. A classic example for
interval data is temperature.
Ratio: If there is a true meaning for 0, then we can call it ratio data type. For
example, in the case of length or income, a value 0 means no length or no
income. They are of type ratio.
3.Based on Availability:

Primary Sources These sources are records of events or evidence as they

are first described or actually happened without any interpretation or
commentary. It is information that is shown for the first time or original
materials on which other research is based. Primary sources display
original thinking, report on new discoveries, or share fresh information.

Examples of primary sources:

Theses, dissertations, scholarly journal articles (research based), some
government reports, symposia and conference proceedings, original
artwork, poems, photographs, speeches, letters, memos, personal
narratives, diaries, interviews, autobiographies, and correspondence.

Secondary Sources These sources offer an analysis or restatement of

primary sources. They often try to describe or explain primary sources.
They tend to be works which summarize, interpret, reorganize, or otherwise
provide an added value to a primary source.

Examples of Secondary Sources:

Textbooks, edited works, books and articles that interpret or review
research works, histories, biographies, literary criticism and interpretation,
reviews of law and legislation, political analyses and commentaries .
Tertiary Sources These are sources that index, abstract, organize, compile,
or digest other sources. Some reference materials and textbooks are
considered tertiary sources when their chief purpose is to list, summarize or
simply repackage ideas or other information. Tertiary sources are usually
not credited to a particular author.

Examples of Tertiary Sources:

Dictionaries/encyclopedias (may also be secondary), almanacs, fact books,
Wikipedia, bibliographies (may also be secondary), directories,
guidebooks, manuals, handbooks, and textbooks (may be secondary),
indexing and abstracting sources.

4.Based On Structural Form:

Structured-
Unstructured-
Semi structured-
5. Based on Inherent:
Qualitative Analysis:
data are expressed in numbers and involve statistical methods. They are concise
and measurable. They help to bring conclusions to the research.
Quantitative Analysis:
data is based on properties or characteristics. They are exploratory and may lead
to further evaluations.

Concept Of Sample Data & Population:

A population is the entire group that you want to draw conclusions about.

A sample is the specific group that you will collect data from. The size of the
sample is always less than the total size of the population.

In research, a population doesn’t always refer to people. It can mean a group

containing elements of anything you want to study, such as objects, events,
organizations, countries, species, organisms, etc.

Collecting data from a population

Populations are used when your research question requires, or when you have
access to, data from every member of the population.
Usually, it is only straightforward to collect data from a whole population when
it is small, accessible and cooperative.

Example: Collecting data from a population. A high school administrator wants

to analyze the final exam scores of all graduating seniors to see if there is a
trend. Since they are only interested in applying their findings to the graduating
seniors in this high school, they use the whole population dataset.

Collecting data from a sample

When your population is large in size, geographically dispersed, or difficult to
contact, it’s necessary to use a sample. With statistical analysis, you can use
sample data to make estimates or test hypotheses about population data.

Example: Collecting data from a sample ,You want to study political attitudes in
young people. Your population is the 300,000 undergraduate students in the
Netherlands. Because it’s not practical to collect data from all of them, you use
a sample of 300 undergraduate volunteers from three Dutch universities – this
is the group who will complete your online survey.

Reasons for sampling

 Necessity: Sometimes it’s simply not possible to study the whole

population due to its size or inaccessibility.
 Practicality: It’s easier and more efficient to collect data from a sample.
 Cost-effectiveness: There are fewer participant, laboratory, equipment,
and researcher costs involved.
 Manageability: Storing and running statistical analyses on smaller
datasets is easier and reliable.

Population parameter vs. sample statistic

When you collect data from a population or a sample, there are various
measurements and numbers you can calculate from the data. A parameter is a
measure that describes the whole population. A statistic is a measure that
describes the sample.

You can use estimation or hypothesis testing to estimate how likely it is that a
sample statistic differs from the population parameter.

Sampling error
A sampling error is the difference between a population parameter and a sample
statistic.
Statistics & Its Types:
Statistics simply means numerical data, and is field of math that generally
deals with collection of data, tabulation, and interpretation of numerical data.
It is actually a form of mathematical analysis that uses different quantitative
models to produce a set of experimental data or studies of real life. It is an area
of applied mathematics concern with data collection analysis, interpretation,
and presentation. Statistics deals with how data can be used to solve complex
problems.
1. Descriptive Statistics :
Descriptive statistics uses data that provides a description of the population
either through numerical calculation or graph or table. It provides a graphical
summary of data. It is simply used for summarizing objects, etc.
2. Inferential Statistics :
Inferential Statistics makes inference and prediction about population based on
a sample of data taken from population. It generalizes a large dataset and
applies probabilities to draw a conclusion. It is simply used for explaining
meaning of descriptive stats. It is simply used to analyze, interpret result, and
draw conclusion. Inferential Statistics is mainly related to and associated with
hypothesis testing whose main target is to reject null hypothesis.
Hypothesis testing is a type of inferential procedure that takes help of sample
data to evaluate and assess credibility of a hypothesis about a population.
Inferential statistics are generally used to determine how strong relationship is
within sample. But it is very difficult to obtain a population list and draw a
random sample.
Application of Statistics:
Statistics is indispensable for decision-making in various sectors and verticals.
It is applied in marketing, e-commerce, banking, finance, human resource,
production, and information technology.