Chapter 1 Big Data Overview

Foreword
⚫ This chapter describes big data basic concepts, characteristics, key
technologies, and use cases.

2
 Objectives
⚫ Upon completion of this course, you will understand:
 Big data concepts
 Four V's of big data
 Key technologies of big data
 Big data use cases

3
 Contents
1. Big Data Concepts
1.1 Database

1.2 Data Warehouse

1.3 Big Data

2. Key Technologies of Big Data

3. Big Data Use Cases

4
Database
⚫ Databases are usually controlled by a database management
system (DBMS). A database is an organized collection of data
stored into multiple datasets, and these datasets (which are
independent of applications) are stored in a computer system
which allows easy access and unified management for multiple
users.

5
Database Characteristics
⚫ Organized: Data is organized, described, and stored based on a
certain data model.
⚫ Shared: Data can be shared by multiple users.
⚫ Low redundancy
⚫ High data independency
⚫ Easy scalability

6
 Contents
1. Big Data Concepts
1.1 Database
1.2 Data Warehouse

1.3 Big Data

2. Key Technologies of Big Data

3. Big Data Use Cases

7
Data Warehouse
⚫ A data warehouse is a subject-oriented, integrated, non-volatile,
and time-variant collection of data, and this data helps analysts to
take informed decisions in an organization by providing a set of
strategies supporting all types of data. The storage structure of
data warehouses is conducive to analytical reporting and decision
making. In addition, data warehouses help enterprises that require
business intelligence improve their business processes, guarantee
service quality, and control monitoring time and cost.
8
Data Warehouse Features

Subject-
Integrated
oriented

Non- Time-
volatile variant

9
 Contents
1. Big Data Concepts
1.1 Database
1.2 Data Warehouse
1.3 Big Data

2. Key Technologies of Big Data

3. Big Data Application Scenarios

10
Big Data
⚫ Currently, there is no universally accepted definition of big data.
Different definitions are all based on characteristics of big data.
 Wikipedia defines big data as datasets with sizes beyond the ability of
commonly used software tools to capture, manage, and process data
within a tolerable elapsed time.
 McKinsey defines big data as datasets whose size is beyond the ability of
typical database software tools to capture, store, manage, and analyze.
Also note that they don't define big data in terms of being larger than a
certain number of terabytes (thousands of gigabytes).
11
The Four V's of Big Data
⚫ Big data is often differentiated by the four V's: volume, velocity,
variety, and value.

Volume Velocity
Huge volume Rapid processing

Four
V's
Variety Value
Various types Low value density

12
 Contents
1. Big Data Concepts

2. Key Technologies of Big Data

1.1 Big Data Collection

1.2 Big Data Storage

1.3 Big Data Analysis and Processing
1.4 Big Data Visualization

3. Big Data Use Cases

13
Big Data Collection
⚫ Collect different types of data in large volume from various sources, such as
website data, mobile device data, spatial data, unstructured texts, as well as
digital data such as voice, images, videos, and image data. Common tools
include Flume, Sqoop, and Kafka.

14
 Contents
1. Big Data Concepts

2. Key Technologies of Big Data

1.1 Big Data Collection
1.2 Big Data Storage

1.3 Big Data Analytics

1.4 Big Data Visualization

3. Big Data Use Cases

15
Big Data Storage
⚫ A distributed file system, such as HDFS, HBase, and Hive, is used for big data
storage and management to support high-throughput data ingestion. For
data scattered on different data nodes, big data storage and management
tools are used to ensure data security and quick ingestion requests through
data replicas and images.

16
 Contents
1. Big Data Concepts

2. Key Technologies of Big Data

1.1 Big Data Collection
1.2 Big Data Storage
1.3 Big Data Analytics

1.4 Big Data Visualization

3. Big Data Use Cases

17
Big Data Analytics
⚫ After sorting, cleaning, and integrating big data, data engineers
use different algorithms and methods to analyze data based on
data analytics tools and service requirements. Big data storage is
the basis of big data analytics, and big data analytics mines data
value and rebuilds these data. Common tools include MapReduce,
Spark, and Flink.

18
 Contents
1. Big Data Concepts

2. Key Technologies of Big Data

1.1 Big Data Collection
1.2 Big Data Storage
1.3 Big Data Analytics
1.4 Big Data Visualization

3. Big Data Use Cases

19
Big Data Visualization
⚫ Big data visualization can visualize both scientific and data
information. Scientific information is mainly oriented to natural
sciences, such as geography, medicine, biology, meteorology, and
aerospace. Data information visualization focuses more on data at
the application layer. Common tools include Echarts, Tableau, and
DLV.

20
 Contents
1. Big Data Concepts

2. Key Technologies of Big Data

3. Big Data Use Cases

21
Big Data Use Cases — Government and Public
Security
⚫ Automatic warning and system linkage
Head counting threshold in an area > 10,000

The supervision
department
immediately locates
the problem.

Automatic warning
system:
Abnormal gathering
alarm on the right side
Head counting of Lakou Cafe Mass gathering at
threshold in an
Send a group fight event
area > 2,000
verification
information.

Warning of abnormal gatherings Report to the

upper-level
departments.

City/Community monitoring Departments of

system verification transaction processing

22
Big Data Use Cases — Transportation Planning
⚫ Multi-dimensional transportation suggestions for different age groups
⚫ Park PI: > 500/hour
Area where the historical people ⚫ Utica Avenue: > 800/hour
flow exceeds the threshold ⚫ McDonald`s: > 1500/hour

Age analysis Travel mode analysis

35% 35% 40%

30%
20% 15% 20%
10%

< 20 20-30 30-40 > 50 Bus Subway Car Other

Traffic prediction suggestions

Road network planning Bus route planning
for different age groups

23
Big Data Use Cases — Education
⚫ Nowadays, big data analytics has been applied to education at all levels.
Average time spent on each question

The Sequencing of Learning performance

Examination Questions 12
1
11
Frequency and
percentage of Enrollment rate
10 2
Teacher-student
interaction
Big data in education
Frequency of and
correctness rate of 9 3 Dropout rate
question
answering

Question 8 4
answering 5 Enrollment rate
frequency
7 5
Word reading accuracy
Hand-raising 6
frequency
Homework accuracy

24
Big Data Use Cases— Finance
Importance of data mining
Obtain service analysis and create
data anytime and anywhere.
have better experience
Review the details
Participate in creating content, Operations
product, and experience New
customers

Receive services in a fixed All channels

Conventional location and at a fixed time
customers Passively receive data
Trust market information Focused
Passively receive propagation scenarios

Offer standard industrial services Marketing

Focus on processes and procedures
Passively receive information from a
single source
Contact customers by customer managers Personalized and
Interact in fixed channels flexible services

Traditional New financial

finance institutions

26
 Section Summary
⚫ This chapter described big data concepts, four V's, key processes,
key technologies, and use cases.

27
 Q&A
1. Describe one or two big data use cases.

2. Briefly discuss the definition of big data.

28
 Recommendations
⚫ Huawei Cloud websites
 Official website: https://www.huaweicloud.com/intl/en-us/
 Developer Institute: https://edu.huaweicloud.com/intl/en-us/

Huawei Cloud
Developer Institute

29
 Thank You.
Copyright© 2023 Huawei Technologies Co., Ltd. All Rights Reserved.
The information in this document may contain predictive statements including,
without limitation, statements regarding the future financial and operating results,
future product portfolio, new technology, etc. There are a number of factors that
could cause actual results and developments to differ materially from those
expressed or implied in the predictive statements. Therefore, such information is
provided for reference purpose only and constitutes neither an offer nor an
acceptance. Huawei may change the information at any time without notice.

30