Chapter Two

Data Science
Prepared by : Surafiel H.
Department of Computer Science,
Addis Ababa University
November, 2021
 Overview of Data Science
• Data science is a multidisciplinary field that uses scientific methods, processes
and algorithm systems to extract knowledge and insights from structured, semi-
structured and unstructured data.

• Data science is much more than simply analyzing data.

• It offers a range of roles and requires a range of skills.

• Data science continues to evolve as one of the most promising and in-demand
career paths for skilled professionals.
2
 What is Data?
• A representation of facts, concepts, or instructions in a formalized manner,
which should be suitable for communication, interpretation, or processing by
human or electronic machine.

• Data can be described as unprocessed facts and figures.

Or
• Data are streams of raw facts representing events occurring in an
organization or in the physical environment before they have been organized
and arranged in to a form that people understand or use it.

3
• Data can also defined as groups of non-random symbols in the form of text, images,
and voice representing quantities, action and objects.

• Data is represented with the help of characters such as alphabets (A-Z, a-z) or special
characters (+, -,/ , *, <, >, = etc.).

4
 What is Information?
• Organized or classified data, which has some meaningful values for the
receiver.

• Processed data on which decisions and actions are based.

• Plain collected data as raw facts cannot help much in decision-making.

• Interpreted data; created from organized, structured, and processed data in a

particular context.

5
 Summary: Data Vs. Information

Data Information
Described as unprocessed or raw facts and figures. Described as processed data.

Can’t help in decision making. Can help in decision making

Raw material that can be organized, structured, and Interpreted data; created from organized, structured, and
interpreted to create useful information systems. processed data in a particular context.

Groups of non-random symbols in the form of text, Processed data; in the form of text, images, and voice
images, and voice representing quantities, action and representing quantities, action and objects'.
objects.

6
 Data VS. Information - Examples
• Data:
 The number of cars sold by a dealership in the past month: 100
 The number of customers who visited the dealership in the past month: 500
• Information:
 The dealership's sales have increased by 10% in the past month.
 The dealership's conversion rate is 20%.
• Data:
 The temperature in Addis Ababa on October 21, 2021, at 6:00 PM was 23 degrees Celsius.
• Information:
 The temperature in Addis Ababa on October 21, 2021, at 6:00 PM was above average for that time of
year.

7
 Data Processing Cycle
• It’s a sequence of steps or operations for processing raw data to be the usable form.
• It is the re-structuring or re-ordering of data by people or machine to increase their
usefulness and add values for a particular purpose.
• Simply put, it’s the process of converting the raw data into information—the
transformation of raw data into meaningful information.
• It is a cyclical process; it starts and ends with data, and the output of one step is the input
for the next step.
• The value of data is often realized when it’s processed and turned into actionable
information.
• Data processing can be used for various purposes, such as business intelligence, research
or decision support.
8
• Data processing cycle typically consists of four main stages:

 Input

 Processing,

 Output, and

 Storage

9
Input:
 The input data is prepared in some convenient form for
processing.

 The form will depend on the processing machine.

 For example, when electronic computers are used, the input data can be
recorded on any one of the several types of input medium, such as flash disks,
hard disk, and so on.

10
Processing:
 In this step, the input data is changed to produce data in a
more useful form (The data is transformed into meaningful
information).
 The raw data is processed by a suitable or selected processing
method.
 For example, a summary of sales for a month can be calculated from the
sales orders data.
11
Output:
 At this stage, the result of the processing step is collected.
 Present processed data in a human readable format, including
reports, charts, graphs and dashboards.
 The particular form of the output data depends on the use of
the data.
 For example, output data can be total sale in a month.

12
Storage:
 Refers to how and where the output of the data processing is
stored for future use.
 The processed data can be stored in databases or file systems,
and it can be kept on various storage devices such as hard
drives, solid-state drives, and cloud storage.

13
 Data Types and its Representation
• In Computer Science and/or Computer Programming, a data type is an
attribute of data which tells the compiler or interpreter how the programmer
intends to use the data.

• The data types defines:-

 The operations that can be done on the data,

 The meaning of the data, and

 The way values of that type can be stored.

14
• Common data types in Computer Programming includes:

 Integers (int) are used to store whole numbers and all the negatives (or opposites) of
the natural numbers, mathematically known as integers.

 Booleans (bool) are used to represent values restricted to one of two values: true (1)
or false (0).

 Characters (char) are used to store a single character.

 Floating-point numbers (float) are used to store real numbers.

 Alphanumeric strings (strings) are used to store a combination of characters and

numbers.
15
 Data Types from Data Analytics Perspective
• From a data analytics point of view, there are three common types of data types or
structures:
1. Structured,
2. Semi Structured, and
3. Unstructured data types.

16
 Structured Data
• Data that adhere to a predefined data model and is therefore
straightforward to analyze.

• Data that resides in a fixed field within a file or record.

• Conforms to a tabular format with relationships between

different rows and columns.

17
• It depends on the creation of data model, defining what types of
data to include and how to store and process it.
 Data model: a visual representation of a database structure.

 Database: an organized collection of structured data typically stored in a computer

system.

• Common examples of structured data are Excel files or SQL databases.

18
 Semi-structured Data
• A form of structured data that does not conform with the formal structure of

data models associated with relational databases or other forms of data tables.

• i.e. doesn’t conform to the formal structure of data model.

• But contain tags or other markers to separate semantic elements and enforce
hierarchies of records and fields within the data.

• Therefore, it is also known as self-describing structure.

19
• Examples of semi-structured data: JSON (JavaScript Object Notation) and XML (Extensible Markup
Language).

 JSON 

 XML 

20
 Unstructured Data
• Data that either doesn’t have a predefined data model or isn’t organized in a
predefined manner.

• There is no data model; the data is stored in its native format.

• It is typically text-heavy, but may contain data such as dates, numbers, and
facts as well.

• Common examples:

 Audio, video files, NoSQL, pictures, pdfs ...

21
 Metadata – Data about Data
• It provides additional information about a specific set of data.

• For example:

 Metadata of a photo could describe when and where the photos were taken.

• The metadata then provides fields for dates and locations which, by
themselves, can be considered structured data.

22
 What is Big Data?
• Generally speaking, Big Data is:
 Large datasets,
 The category of computing strategies and technologies that are used to handle large datasets.
• A data set is an ordered collection of data.
• Big data is a collection of data sets so large and complex that it becomes difficult to
process using on-hand database management tools or traditional data processing
applications.
• The common scale of big data sets is constantly shifting and may vary significantly
from organization to organization.
• Big Data is mainly characterized by 4V’s.
23
24
• Volume: refers the amount of the data.
 Large amounts of data - Zeta bytes/Massive datasets.

• Velocity: refers the speed of data processing

 Data is live streaming or in motion.

• Variety: refers to the number of types of data.

 Data comes in many different forms from diverse sources.

• Veracity: which in this context is equivalent to quality.

 Can we trust the data?

 Are the data “clean” and accurate?

 Do they really have something to offer?

25
 Other V's of Big Data
• Value: refers to the usefulness of gathered data for the business.

• Variability: refers the number of inconsistencies in the data and the inconsistent speed
at which big data is loaded into the database.

• Validity: data quality, governance, master data management on massive.

• Venue: distributed multiple data from heterogeneous data from multiple platforms.

• Vocabulary: data models, semantics that describes data structure.

26
• Vulnerability: big data brings new security concerns.

 After all, a data breach with big data is a big breach.

• Volatility: due to the velocity and volume of big data, however, its volatility needs to
be carefully considered.

 How long does data need to be kept for?

• Visualization: different ways of representing data such as data clustering or using

tree maps, sunbursts, parallel coordinates, circular network diagrams, cone trees.

• Vagueness: confusion over meaning of big data and tools used.

27
28
 Data Value Chain
• Describe the information flow within a big data system as a series of steps
needed to generate value and useful insights from data.

• The Big Data Value Chain identifies the following key high-level activities:

 Data Acquisition,
 Data Analysis,
 Data Curation,
 Data Storage, and
 Data Usage
29
 Data Acquisition
• It is the process of gathering, filtering, and cleaning data before it is put in a data

warehouse or any other storage solution on which data analysis can be carried out.

 A data warehouse is a system that aggregates, stores, and processes information

from diverse data sources.

• Data acquisition is one of the major big data challenges in terms of infrastructure

requirements.

30
• The infrastructure required for data acquisition must:

 Deliver low, predictable latency in both capturing data and in executing queries.

 Be able to handle very high transaction volumes, often in a distributed

environment.

 support flexible and dynamic data structures.

31
 Data Analysis
• A process of cleaning, transforming, and modeling data to discover useful information
for business decision-making.

• Involves exploring, transforming, and modelling data with the goal of highlighting
relevant data, synthesising and extracting useful hidden information with high potential
from a business point of view.

• Related areas include data mining, business intelligence, and machine learning.

32
 Data Curation
• Active management of data over its life cycle to ensure it meets the necessary data
quality requirements for its effective usage.

• Data curation processes can be categorized into different activities such as content :
 Creation,
 Selection,
 Classification,
 Transformation,
 Validation, and
 Preservation
33
• Data curation is performed by expert curators that are responsible for improving the
accessibility and quality of data.
• Data curators (also known as scientific curators, or data annotators) hold the
responsibility of ensuring that data are trustworthy, discoverable, accessible, reusable,
and fit their purpose.

• A key trend for the curation of big data utilizes community and crowd sourcing
approaches.

34
 Data Storage
• It is the persistence and management of data in a scalable way that satisfies the needs of

applications that require fast access to the data.

• Relational Database Management Systems (RDBMS) have been the main, and almost

unique, solution to the storage paradigm for nearly 40 years.

35
• Relational database that guarantee database transactions, lack flexibility with regard to

schema changes, performance and fault tolerance when data volumes and complexity

grow, making them unsuitable for big data scenarios.

• NoSQL technologies have been designed with the scalability goal in mind and present a

wide range of solutions based on alternative data models.

36
 Data Usage
• Covers the data-driven business activities that need access to data, its analysis, and the

tools needed to integrate the data analysis within the business activity.

• In business decision-making, it can enhance competitiveness through reduction of costs,

increased added value, or any other parameter that can be measured against existing

performance criteria.

37
 Cluster Computing
• Computing refers to the process of using computers to perform various tasks, including

calculations, data processing, and problem-solving.

• It involves the manipulation and transformation of data using software applications and

algorithms.

• Computing can be done on a single computer or distributed across multiple computers

connected through a network.

38
• Cluster computing is a specific type of computing that involves the use of a cluster.

• In general, cluster means “small group”.

• Cluster computing is a group of interconnected computers or servers working together to

perform a task or solve a problem.

• It refers to multiple computers connected to a network that function as a single entity

• It allows for the distribution of computational load across multiple machines, enabling
faster processing and increased computational power.
39
• In a cluster computing setup, each computer in the cluster, also known as a node, works
in parallel with other nodes to handle different parts of a larger problem or workload.

• The nodes are connected through a high-speed network and communicate with each
other to coordinate their tasks.

• Each node is performing a dedicated task.

• Many nodes (each node) connected with a single node which is called Head node.

• Accessing a cluster system typically means accessing a head node or gateway node.

• A head node or gateway node is set up to be the launching point for jobs running on the
cluster and the main point of access to the cluster.
40
• A classic cluster essentially allows the nodes to
share infrastructure, such as disk space, and
collaborate by sharing program data while those
programs are running.
• Cluster computing offers solutions to solve
complicated problems by:
 Providing faster computational speed, and
 Enhanced data integrity.
 Data integrity refers to the overall
accuracy, completeness, and
consistency of data.

41
 Big Data Cluster System
• In big data, individual computers are often inadequate for handling data at most stages.

• Therefore, the high storage and computational needs of big data are addressed through
computer clusters.

• A big data cluster system is a specialized type of cluster computing system designed to
manage and process large volumes of data.

• The primary goal of a big data cluster system is to enable scalable and distributed
processing of big data across multiple nodes within the cluster.

42
• Big Data clustering software that combines the resources of many smaller machines

offers several benefits.

• Some examples of big data clustering software/tools include Hadoop's YARN (Yet

Another Resource Negotiator), Qubole, HPCC, Cassandra, MongoDB, Apache Storm,

CouchDB, and Statwing.

43
• Using big data cluster provides solution for :
 Managing cluster membership,
 Coordinating resource sharing, and
 Scheduling actual work on individual nodes.
• Cluster membership & resource allocation can be handled by software like Hadoop’s
YARN (Yet Another Resource Negotiator).
• The assembled computing cluster often acts as a foundation which other software
interfaces with to process data.
• Additionally, the machines in the computing cluster typically manage a distributed
storage system.

44
Benefits of Big Data Clustering Software
1. Resource Pooling:

 Involves combining available storage space to hold data.

 Encompasses CPU and memory pooling, which are crucial for processing large datasets that

require substantial amounts of these two resources.

2. High Availability:
 Clusters offer varying levels of fault tolerance and availability.

 Guarantee to prevent hardware/software failures from affecting access to data and

processing.

 This becomes increasingly important as real-time analytics continue to be emphasized.

45
3. Easy Scalability:

 Clusters facilitate easy horizontal scaling by adding more machines to the group.

 This allows the system to adapt to changes in resource demands without needing
to increase the physical resources of individual machines.

46
 Hadoop
• It’s an open-source framework designed to simplify interaction with big data.

• Hadoop is an open-source framework designed for distributed storage and processing of large
datasets across clusters of computers.

• It is inspired by a technical document published by Google.

 Open-source software allows anyone to inspect, modify, and enhance its source code.
 This development and distribution model provide the public with access to the underlying
(source) code of a program or application.
 Source code refers to the part of software that typical computer users don’t see; it's the code
that programmers can modify to alter how a piece of software, such as a program or
application, functions.
 A software framework is an abstraction that provides generic functionality, allowing users to
47
extend it with additional code to create application-specific software.
 Characteristics of Hadoop
• Economical:
 Hadoop systems are highly economical because they can utilize ordinary computers for data processing.

• Reliable:
 Hadoop is reliable due to its ability to store data copies on different machines, making it resistant to hardware
failures.

• Scalable:
 Hadoop is easily scalable both horizontally and vertically, allowing the framework to expand with the addition of
extra nodes.

• Flexible:
 Hadoop is flexible, enabling storage of both structured and unstructured data for future use as needed.
48
 Hadoop Ecosystem
• Hadoop has an ecosystem that has evolved from its four core components:
1. Data management: Involves handling the storage, organization, and retrieval of data.

2. Data access: Enables users to interact with and retrieve data stored in Hadoop.

3. Data processing: Enables the execution of computations and analytics on large datasets.

4. Data storage: Focuses on efficient and scalable ways to store and manage data.

• The Hadoop ecosystem is continuously expanding to meet the needs of big data.

• These components work together to provide a comprehensive ecosystem for managing,

accessing, processing, and storing big data.

• The ecosystem offers a range of tools and technologies to address different aspects of the
data lifecycle and cater to various use cases in big data analytics.
49
 Components of Hadoop Ecosystem
• Hadoop Distributed File System (HDFS): A distributed file system that provides reliable
and scalable storage for big data across multiple machines.

• Yet Another Resource Negotiator (YARN): The resource management framework in

Hadoop that manages resources and schedules tasks across the cluster, enabling multiple
processing engines to run on Hadoop.

• MapReduce: A programming model and processing engine in Hadoop for parallel

processing of large datasets by dividing tasks into map and reduce phases.

• Spark: A fast and general-purpose cluster computing system with in-memory processing
capabilities. It seamlessly integrates with Hadoop and offers higher performance for
certain workloads. 50
• Pig: A high-level scripting platform in Hadoop that simplifies data processing tasks using
a language called Pig Latin, abstracting away the complexities of MapReduce.

• Hive: A data warehouse infrastructure built on Hadoop, providing a high-level query

language (HiveQL) for querying and analyzing data stored in Hadoop.

• HBase: A distributed and scalable NoSQL database that runs on top of Hadoop. It offers
random, real-time access to big data and is suitable for low-latency read/write operations.

• Mahout: A library of machine learning algorithms for Hadoop, providing scalable

implementations for tasks like clustering, classification, and recommendation.

51
• MLlib: A machine learning library in Spark that offers a rich set of algorithms and tools for scalable
machine learning tasks, including data preprocessing, feature extraction, model training, and
evaluation.

• Solr: An open-source search platform based on Apache Lucene, providing powerful search capabilities
such as full-text search, faceted search, and real-time indexing.

• Lucene: A Java library for full-text search, providing indexing and searching functionalities and
serving as the core technology behind search-related applications like Solr and Elasticsearch.

• ZooKeeper: A centralized coordination service for distributed systems, offering reliable infrastructure
for maintaining configuration information, synchronizing processes, and managing distributed locks.

• Oozie: A workflow scheduling system for Hadoop, enabling users to define and manage workflows that
coordinate the execution of multiple Hadoop jobs, automating complex data processing pipelines.
52
53
 Big Data Life Cycle with Hadoop
1. Ingesting data into the system:

 The first stage of big data processing with Hadoop is ingestion.

 Data is transferred to Hadoop from various sources such as relational

databases, systems, or local files.

 Sqoop facilitates data transfer from RDBMS to HDFS, while Flume

handles event data transfer.

54
2. Processing the data in storage:

 The second stage involves processing and storing.

 Data is stored in the distributed file system, HDFS, and in the NoSQL
distributed database, HBase.

 Data processing is carried out using Spark and MapReduce.

55
3. Computing and analyzing data:

 The third stage is analysis.

 Data is analyzed using processing frameworks like Pig, Hive, and

Impala.

 Pig employs map and reduce techniques for data conversion and
analysis, while Hive, based on map and reduce programming, is well-
suited for structured data.

56
4. Visualizing the results:

 The final stage, access, involves tools such as Hue and Cloudera Search.

 Here, users can access the analyzed data and visualize the results.

57
58