Chapter 2

Introduction to Data Science

 Outline

 An Overview of Data Science

 What are data and information?

Data types and their representations

Data value Chain

Basic concepts of big data

2 Introduction to Data Science

 Objective
After completing this chapter, the students will be able
to:
 Describe what data science is and the role of data scientists.
 Differentiate data and information.
 Describe data processing life cycle
 Understand different data types from diverse perspectives
 Describe data value chain in emerging era of big data.
 Understand the basics of Big Data.
 Describe the purpose of the Hadoop ecosystem
components.

3 Introduction to Data Science

 Overview of Data Science
 Data science is a multi-disciplinary field that uses scientific
methods, processes, algorithms and systems to extract
knowledge and insights from structured, semi
structured and unstructured data.
 Data science continues to evolve as one of the most
promising and in-demand career paths for skilled
professionals.
 Today, successful data professionals understand that they
must advance past traditional skills of analyzing large
amounts of data, data mining, and programming skills.

4 Introduction to Data Science

 Overview of Data Science
 In order to uncover useful intelligence for an organization,
data scientists must master the full spectrum of the data
science life cycle and possess a level of flexibility and
understanding to maximize returns at each phase of the
process.
 Data scientists need to:
 Be curious and result-oriented
 Be good at communication skills that allow them to explain
highly technical results to their non-technical counterparts.
 Have a strong quantitative background in statistics and linear
algebra as well as programming knowledge with focuses in
data warehousing, mining, and modeling to build and analyze
algorithms.
5 Introduction to Data Science
 What are Data and Information?
What is Data?
 Data can be defined as a representation of facts,
concepts, or instructions in a formalized manner,
which should be suitable for communication,
interpretation, or processing by human or electronic
machine.
 It can be described as unprocessed facts and figures.
 Data is represented with the help of characters such as
alphabets (A-Z, a-z), digits (0-9) or special characters
(+,-,/,*,<,>,= etc.)
6 Introduction to Data Science
 What is Information?
 Information is the processed data on which decisions
and actions are based.
 It is data that has been processed into a form that is
meaningful to the recipient and is of real or perceived
value in the current or the prospective action or
decision of recipient.
 Furtherer more, information is interpreted data;
created from organized, structured, and processed
data in a particular context.

7 Introduction to Data Science

 Data Vs. Information

Data Information
Described as unprocessed or raw facts and Described as processed data
figures

Cannot help in decision making Can help in decision making

Raw material that can be organized, Interpreted data; created from organized,
structured, and interpreted to create useful structured, and processed data in a particular
information systems. context.

‘groups of non-random’ symbols in the Processed data in the form of text, images, and
form of text, images, and voice voice representing quantities, action and objects'.
representing quantities, action and
objects'.

8 Introduction to Data Science

 Data Processing Cycle
 Data processing is the re-structuring or re-ordering of
data by people or machine to increase their usefulness
and add values for a particular purpose.
 Data processing cycle is consisted of three basic steps-
Input, Processing, and Output.

9 Introduction to Data Science

 Data Processing Cycle
 Input step− the input data is prepared in some convenient
form for processing. The form depends on the processing
machine.
 For example - when electronic computers are used, the input data can be
recorded on any one of the several types of input medium, such as
magnetic disks, tapes,
 Processing step − the input data is changed to produce data
in a more useful form.
 For example - pay-checks can be calculated from the time cards, or a summary of
sales for the month can be calculated from the sales orders.
 Output step − the result of the proceeding processing step
is collected.
 The particular form of the output data depends on the use of the data.
 For example - output data may be pay-checks for employees.
10 Introduction to Data Science
 Data types and their representation
 Data types can be described from diverse
perspectives.
 In computer science and computer programming, for
instance, a data type is simply an attribute of data that
tells the compiler or interpreter how the programmer
intends to use the data.

11 Introduction to Data Science

 Data types from Computer
programming perspective
 All programming languages explicitly include the notion
of data type
 Common data types include:
• Integers(int)- is used to store whole numbers, mathematically
known as integers
• Booleans(bool)- is used to represent restricted to one of two
values: true or false
• Characters(char)- is used to store a single character
• Floating-point numbers(float)- is used to store real
numbers
• Alphanumeric strings(string)- used to store a combination
of characters and numbers
12 Introduction to Data Science
 Data types from Computer
programming perspective

 A data type makes the values that expression,

such as a variable or a function, might take.
 This data type defines the operations that can be
done on:
 the data
 the meaning of the data, and
 the way values of that type can be stored.

13 Introduction to Data Science

 Data types from Data Analytics
perspective
 From a data analytics point of view, it is important to
understand that there are three common data types or
structures:
 Structured data
 Semi-structured data and
 Unstructured data types.

14 Introduction to Data Science

 Structured Data
 Structured data is data that adheres to a pre-defined
data model and is therefore straightforward to analyze.
 Structured data conforms to a tabular format with a
relationship between the different rows and columns.
 Common examples of structured data are Excel files
or SQL databases.
 Each of these has structured rows and columns that can
be sorted.

15 Introduction to Data Science

 Unstructured Data
 Unstructured data is information that either does not have
a predefined data model or is not organized in a pre-
defined manner.
 Unstructured information is typically text-heavy but may
contain data such as dates, numbers, and facts as well.
 This results in irregularities and ambiguities that make it
difficult to understand using traditional programs as
compared to data stored in structured databases.
 Common examples of unstructured data include audio,
video files or No-SQL databases.
16 Introduction to Data Science
17 Introduction to Data Science
 Metadata – Data about Data
 The last category of data type is metadata.
 From a technical point of view, this is not a
separate data structure, but it is one of the most
important elements for Big Data analysis and big
data solutions.
 Metadata is data about data.
 It provides additional information about a
specific set of data.
18 Introduction to Data Science
 Metadata – Data about Data
 In a set of photographs, for example, metadata
could describe when and where the photos were
taken.
 The metadata then provides fields for dates and
locations which, by themselves, can be
considered structured data.
 Because of this reason, metadata is frequently
used by Big Data solutions for initial analysis.
19 Introduction to Data Science
 Data Value Chain (DVC)
 The Data Value Chain describes the information flow
within a big data system that aims to generate values and
useful insights from data.
 The Big Data Value Chain identifies the following key high-
level activities:
 Data Acquisition
 Data analysis
 Data curation
 Data Storage
 Data Usage
20 Introduction to Data Science
 Data Value Chain (DVC)

21 Introduction to Data Science

 Data Acquisition
 It is the process of gathering, filtering, and cleaning data
before it is put in a data warehouse or any other storage
solution on which data analysis can be carried out.
 Data acquisition is one of the major big data challenges in
terms of infrastructure requirements.
 The infrastructure required to support the acquisition of
big data must provide
 low latency in both capturing data and in executing
queries.
 high volumes of transaction, often in a distributed
environment;
 flexible and dynamic data structures

22 Introduction to Data Science

 Data Analysis
 It is concerned with making the raw data acquired
amenable to use in decision-making as well as domain-
specific usage.
 Data analysis involves exploring, transforming, and
modeling data with the goal of highlighting relevant
data, synthesizing and extracting useful hidden
information with high potential from a business point of
view.
 Related areas include data mining, business intelligence,
and machine learning.
23 Introduction to Data Science
 Data Curation
 It is the active management of data over its life cycle to ensure it
meets the necessary data quality requirements for its effective usage.
 Data curation processes can be categorized into different activities
such as Content creation, Selection, Classification, transformation,
validation and preservation
 Data curation is performed by expert curators (Data curators,
scientific curators, or data annotators) that are responsible for improving
the:
 Accessibility
 Quality
 Trustworthy
 Discoverable,
 Accessible, and
 Reusable of data
24 Introduction to Data Science
 Data Storage
 It is the persistence and management of data in a
scalable way that satisfies the needs of applications that
require fast access to the data.
 Relational Database Management Systems (RDBMS)
have been the main, and almost unique, a solution to
the storage paradigm for nearly 40 years.

25 Introduction to Data Science

 Data Storage
 However, the ACID (Atomicity, Consistency, Isolation,
and Durability) properties that guarantee database
transactions lack flexibility with regard to schema
changes and the performance and fault tolerance when
data volumes and complexity grow, making them
unsuitable for big data scenarios.
 NoSQL technologies have been designed with the
scalability goal in mind and present a wide range of
solutions based on alternative data models.

26 Introduction to Data Science

 Data Usage
 It covers the data-driven business activities that need
access to data, its analysis, and the tools needed to
integrate the data analysis within the business activity.
 Data usage in business decision-making can enhance
competitiveness through the reduction of costs,
increased added value, or any other parameter that can
be measured against existing performance criteria.

27 Introduction to Data Science

 Basic Concepts of Big Data
 Big data is a blanket term for the non-traditional
strategies and technologies needed to gather, organize,
process, and gather insights from large datasets.
 While the problem of working with data that exceeds
the computing power or storage of a single computer
is not new, the pervasiveness, scale, and value of this
type of computing have greatly expanded in recent
years.

28 Introduction to Data Science

 Basic Concepts of Big Data
What Is Big Data?
 Big data is the term for a collection of data sets so large and
complex that it becomes difficult to process using on-hand
database management tools or traditional data processing
applications.
 In this context, a “large dataset” means a dataset too large to
reasonably process or store with traditional tooling or on a
single computer.
 This means that the common scale of big datasets is constantly
shifting and may vary significantly from organization to
organization.

29 Introduction to Data Science

 Basic Concepts of Big Data
Big data is characterized by 3V and more:
Volume: large amounts of data Zeta bytes/Massive datasets
Velocity: Data is live streaming or in motion
Variety: data comes in many different forms from diverse
sources
Veracity: can we trust the data? How accurate is it? etc.

30 Introduction to Data Science

 Clustered Computing
and
Hadoop Ecosystem

31 Introduction to Data Science

 Clustered Computing
 Because of the qualities of big data, individual
computers are often inadequate for handling the data
at most stages.
 To better address the high storage and computational
needs of big data, computer clusters are a better fit.
 Big data clustering software combines the resources of
many smaller machines, seeking to provide a number
of benefits.

32 Introduction to Data Science

 Clustered Computing
Benefits of Clustered Computing
 Resource Pooling: Combining the available storage
space to hold data is a clear benefit, but CPU and
memory pooling are also extremely important.
Processing large datasets requires large amounts of all
three of these resources.
 High Availability: Clusters can provide varying
levels of fault tolerance and availability guarantees to
prevent hardware or software failures from affecting
access to data and processing. This becomes increasingly
important as we continue to emphasize the importance
of real-time analytics.

33 Introduction to Data Science

 Clustered Computing
Benefits of Clustered Computing
 Easy Scalability: Clusters make it easy to scale
horizontally by adding additional machines to the
group. This means the system can react to changes
in resource requirements without expanding the
physical resources on a machine.

34 Introduction to Data Science

 Hadoop Ecosystem
 Using clusters requires a solution for managing cluster
membership, coordinating resource sharing, and scheduling
actual work on individual nodes. Cluster membership and
resource allocation can be handled by software like Hadoop’s
YARN (which stands for Yet Another Resource Negotiator).
 The assembled computing cluster often acts as a foundation
that other software interfaces with to process the data. The
machines involved in the computing cluster are also typically
involved with the management of a distributed storage system,
which we will talk about when we discuss data persistence.

35 Introduction to Data Science

 Hadoop Ecosystem
 Hadoop is an open-source framework intended to make
interaction with big data easier.
 It is a framework that allows for the distributed
processing of large datasets across clusters of
computers using simple programming models.
 It is inspired by a technical document published by
Google.

36 Introduction to Data Science

 Hadoop Ecosystem
The four key characteristics of Hadoop are:
 Economical: Its systems are highly economical as ordinary
computers can be used for data processing.
 Reliable: It is reliable as it stores copies of the data on
different machines and is resistant to hardware failure.
 Scalable: It is easily scalable both, horizontally and
vertically. A few extra nodes help in scaling up the
framework.
 Flexible: It is flexible and you can store as much structured
and unstructured data as you need to and decide to use them
later.
37 Introduction to Data Science
 Hadoop Ecosystem
 Hadoop has an ecosystem that has evolved from its
four core components:
 Data Management
 Access
 Processing
 Storage
 It is continuously growing to meet the needs of Big
Data.

38 Introduction to Data Science

 Hadoop Ecosystem
Hadoop comprises the following components and many others:
 HDFS: Hadoop Distributed File System
 YARN:Yet Another Resource Negotiator
 MapReduce: Programming based Data Processing
 Spark: In-Memory data processing
 PIG, HIVE: Query-based processing of data services
 HBase: NoSQL Database
 Mahout, Spark MLLib: Machine Learning algorithm libraries
 Solar, Lucene: Searching and Indexing
 Zookeeper: Managing cluster
 Oozie: Job Scheduling
39 Introduction to Data Science
 Hadoop Ecosystem

40 Introduction to Data Science

 Big Data Life Cycle with Hadoop

 The general categories of activities or life cycle

involved with big data processing are:
I. Ingesting data into the system
II. Processing the data in storage
III. Computing and analyzing data
IV. Visualizing the results

41 Introduction to Data Science

 Ingesting data into the system
 The first stage of Big Data processing is Ingest.
 The data is ingested or transferred to Hadoop from
various sources such as relational databases, systems,
or local files.
 Sqoop transfers data from RDBMS to HDFS, whereas
Flume transfers event data.

42 Introduction to Data Science

 Processing the data in storage
 The second stage is Processing.
 In this stage, the data is stored and processed.
 The data is stored in the distributed file system, HDFS,
and the NoSQL distributed data, HBase.
 Spark and MapReduce perform data processing.

43 Introduction to Data Science

 Computing and analyzing data
 The third stage is to Analyze.
 Here, the data is analyzed by processing frameworks
such as Pig, Hive, and Impala.
 Pig converts the data using a map and reduce and then
analyzes it.
 Hive is also based on the map and reduce
programming and is most suitable for structured data.

44 Introduction to Data Science

 Visualizing the results
 The fourth stage is Access, which is performed by tools
such as Hue and Cloudera Search.
 In this stage, the analyzed data can be accessed by
users.

45 Introduction to Data Science