Big Data Industry and

Technological Trends

www.huawei.com

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved.

 Objectives
 After completing this course, you will be able to understand:
 What big data is
 Big data technological trends and applications
 Huawei big data solution

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 2
 Contents
1. Big Data Era

2. Big Data Application Scope

3. Opportunities and Challenges in the Big Data Era

4. Huawei Big Data Solution

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 3
Big Data As a Country Strategy for All
Countries
USA
 Group of Eight (G8) has released the G8 Open Data Charter and proposed
to accelerate the implementation of data openness and usability.
 The European Union (EU) promotes the Data Value Chain to transform
traditional governance model, reduce common department cost, and
G8
accelerate economic growth and employment growth with big data.
 The Abe Cabinet announced the Declaration to be the World's Most
Advanced IT Nation, which plans to develop Japan's national IT strategy
with open public data and big data as its core.
UK  The UK Government released the Capacity Development Strategy, which
aims to use data to generate business value and boost economic growth,
as well as undertakes to open the core databases in the transportation,
weather, medical treatment fields.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 4
Implementing the National Big Data
Strategy
 Implementing the national big data strategy to
accelerate the construction of a "Digital China"
involves five tasks, which are summarized as follows:
 Promote the innovation and development of big
data technology.
 Build a digital economy with data as a key
enabler.
 Improve the country's capability in governance
with big data.
 Improve people's livelihood by applying big data.
 Protect national data security.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 5
Big Data Era

Definition from Wikipedia:

Big data is data sets that are so voluminous and complex that traditional data-
processing application software is inadequate to deal with them.

4 V's

Huge amount of data (Volume) Data processing speed

(Velocity)
Various types of data (Variety) Low data value density
(Value)

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 6
Source of Big Data

There are more

than 200 million There are more than 300
messages every million active users every There are 2.8 billion
day. day. smartphone users
worldwide.

Hundreds of
millions of devices
that support the
Global Positioning
System (GPS) are
sold each year.

Facebook: 50 TB log data is generated each day, CERN: Experiments at CERN are generating
with over 100 TB analysis data derived. an entire petabyte (PB) of data every second
as particles fired around the Large Hadron
Collider (LHC).

Social data Machine data

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 7
All Businesses Are Data Businesses

Your business is Data about your

a data business customers is as
now. valuable as your
customers.

Data as a
Platform
Streaming data (DaaP)
is business
opportunity.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 8
Differences Between Data Processing in the Big
Data Era and the Traditional Data Processing
From databases (DBs) to big data (BD)
"Pond fishing" vs. "Sea fishing". "Fishes" represent the data to be processed.

Database Big Data

Data scale Small (in MB) Large (in GB, TB, or PB)

Diversified (structured, semi-structured,

Data type Single (mostly structured)
or unstructured)
Relationship
Modes come ahead of data. Data comes ahead of modes. Modes
between modes and
(Ponds come ahead of fishes.) evolve constantly as data increases.
data
Object to be (Fishes, determine whether other types of
Data (fishes in ponds)
processed fishes exist by certain fishes.)

Processing tool One size fits all. No size fits all.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 10
Big Data Era
China's netizens rank the first in the world, and the data volume generated each
day also surpasses others in the world.

 More than 50 thousand GB data generated per day

Taobao website  40 million GB data storage volume

 1 billion GB data in total

Baidu  1 trillion web pages stored
 About 6 billion search requests to be processed each day

A camera working  3.6 GB data can be generated per hour.

at a rate of 8  Tens of millions GB data can be generated each month in one city
Mbit/s

 The amount of CT image data generated for a patient reaches dozens of

Hospitals GB.
 Tens of billions GB data in a country needs to be stored each year.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 11
Big Data Era
 Decrease of hardware costs
 Acceleration of network
bandwidth
 Emergence of cloud computing
 Popularization of intelligent
terminals
 E-commerce and social
networks
 Comprehensive application of
electronic maps
 Internet of Things (IoT)

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 12
Relationship Between Big Data and
People
If all you have is a hammer, everything looks like a nail.

Today, it seems that big data is miraculous and

omnipotent. However, do not take big data as an
all-round way to solve all problems in the world.
Human thought, personal cultural and behavior
modes, as well as the existence and development of
different nations and the society are complicated,
intricate, and unique. Computers cannot be used to
let the numbers speak for themselves. No matter
when, it is people who are speaking and thinking.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 13
What Big Data Cannot Do?
 Substitute managers' decision-making capabilities
 Big data is not only a technical issue, but also a decision-making issue.
 Data onboarding must be pushed and determined by top leaders.

 Substitute effective business models

 Balance cannot always be obtained with big data.
 Business models are paramount. We must figure out how to obtain profits in advance.

 Discover knowledge aimlessly

 Data mining must be worked out with restrictions and targets.
Otherwise, it is futile.

 Substitute the role of experts

 Experts contribute greatly to product modeling, such as IBM Deep Blue and AlphaGo.
 The role of experts may decrease as time goes on. However, experts play a major role in the
initial stage.

 Build a model for permanent benefits

 Big data requires 'live' data (with feedback).
 Models need to be updated through lifelong machine learning.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 14
 Contents
1. Big Data Era

2. Big Data Application Scope

3. Opportunities and Challenges in the Big Data Era

4. Huawei Big Data Solution

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 15
Big Data Era Leading the Future
Data has penetrated into every industry and business domain.
Discerning essences (services), forecasting trends, and guiding the
future are the core of the big data era.
Guide the efforts we make now with a clear future target and
make due efforts now to secure future success.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 16
Big Data Application Scope
Proportion of Top 100 industries using big data

Finance
17% City 14%

Medical
Retail treatment
24% 8%
Sports
6%
Education
Telecom 4%
Others 23% 4%

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 17
Big Data Application: Politics
Big data psychological analysis helped Trump win
the America's presidential election.
 Donald Trump employed Cambridge Analytica Ltd (CA) to make
personality and requirement analysis on American voters, which
acquired personalities of 220 million Americans.
 CA uses the behavior of giving likes by voters on Facebook to analyze
the personality traits and political orientation of voters, classifies
voters into three types (Republican supporters, Democratic
supporters, and swing voters), and focuses on attracting swing
voters.
 Trump has never sent emails before. He bought his first smartphone
after the presidential election and was fascinated with Twitter. The
messages sent by him on Twitter are data-driven and vary for
different voters.
The cave  For African Americans, they can see the video in which the black is
(data analysis center) called predators by Hillary Clinton, and thereby go away from Hillary's
ballot box. These dark posts are visible for only specified users.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 18
Big Data Application: Finance
Importance of
Obtain services anytime
and anywhere.
data mining
Analyze and create data.
Seek for meaningful
experience.
Review details. Operate
Involve in creating customers
Obtain services at fixed content, products, and New
times and places. experience. customers
Passively receive data.
Trust market information. Omni-
Traditional
Passively receive channel
customers information propagation.

Scenario-
focused
Offer standard industrial services.
Focus on processes and procedures. Efficiency
Passively receive information from a single Merchandise
source. customers
Contact customers by customer managers.
Interact with each other in fixed channels
and in inflexible ways.
Flexible
personalized
services
Serve
New financial customers
Traditional
finance institutions

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 19
Big Data Application: Finance Case

Four-hour time difference

between the east and west
coasts of the USA

Walmart
Walmart uses the sales
analysis result of the east
coast to guide the goods
arrangement of the west
coast on the same day.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 20
Big Data Application: Education
Now, big data analysis has been applied to American public education and become an
important force of education reform.
Average time for answering
each question
Sequence of question-
answering 12
in exams 11 Academic performance
1
Duration and
frequency of
interaction with
10 Big data in 2 Enrollment rate

teachers school education

Duration and correctness
of 9 3 Dropout rate
answering questions

Question-answering Rate of admission into

8 4 higher schools
times
5
Hand-raising 7 5 Literacy accuracy
times in class
6 Homework correctness

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 21
Big Data Application: Transportation
Most people may choose railway for a distance less than 500 km, but...

Example mode of transportation for a distance Mode of transportation

less than 500 km: Beijing–Taiyuan during the Chinese Spring
Festival in 2018 Price
Time
Has the highest More cost-effective
performance- because the performance-
price ratio but price ratio of vehicle rental
Beijing with difficulties in is inferior to entraining
500 km scrambling for
tickets

Shanghai
Chengdu
500 km
500 km

Guangzhou Plane Train Vehicle Long-distance

rental coach
500 km

For a 500 km or 6-hours driving distance, railway has the highest performance-price
ratio, but the chance of buying tickets depends upon luck. The performance-price
ratio of vehicle rental is inferior to entraining. According to a survey, in the event of
failing in scrambling for train tickets, more than 70% of people will rent a vehicle to
go home.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 22
Big Data Application: Tourism
Island Travel Preference During China’s National
Day Holiday
3% 3%

5%
5%
29%
5%

9%

11%
18%
12%

Phuket Koh Samui Bali Kuala Lumpur Okinawa

Manila Colombo Jakarta Jeju Honolulu

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 23
Big Data Application: Tourism
Honolulu

Colombo

Bali

Okinawa

Jeju

Phuket

Jakarta

Manila

Koh Samui

Kuala Lumpur

0 1000 2000 3000 4000 5000 6000

Air ticket price forecast

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 24
Big Data Application: Government and
Public Security
Public security scenario: automatic warning and response
Area-based people flow threshold > 10,000 people

Supervision
department
performs real-
time locating of
Automatic warning issues at the
system: initial stage.
The number of
people in right side of
The crowd gathers
Beijing Olympic
to watch an affray.
Forest Park exceeds Delivers the
the threshold. issue for
confirmation.
Area-based people
flow threshold >
2000 people Reports to
upper-level
departments
Warning for abnormal
increase of people flow

City or community monitoring Transaction processing

Big data analysis can monitor and
system confirmation departments
analyze population flow into
cities.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 25
Big Data Application: Traffic Planning
Traffic planning scenarios: multi-dimensional analysis of the traffic crowd

 North gate of Beijing Workers' Gymnasium: The number of

Areas with people flow people per hour exceeds 500.
exceeding the threshold  Sanlitun: The number of people per hour exceeds 800.
 Beijing Workers' Gymnasium: The number of people per
once
hour exceeds 1500.

Analyze by people flow Analyze by travel mode proportion

35% 40% 30%

35% 15%
20% 20% 10%

Younger 20-30 30-40 Older than Bus Metro Auto Others

than 20 50

Road Network
Traffic forecast based on the crowd Bus line planning
Planning

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 26
Big Data Applications: Sports

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 27
 Contents
1. Big Data Era

2. Big Data Application Scope

3. Opportunities and Challenges in the Big Data Era

4. Huawei Big Data Solution

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 28
Challenges of Traditional Data
Processing Technologies
Scalability
required for There is a gap between data
big data scalability requirements and
processing hardware performance.

 High cost for storing massive

data
Portal  Insufficient batch data
processing performance
Special Self-service Data Data
OLAP
application Report Select KPI analysis mining management  Lack of streaming data
Appframe Spring processing
Application Middleware (Weblogic8.2 and Apache Tomcat 5.2)
 Limited scalability
 Single data source
 External value-added data
assets

Minicomputer Resource: P595, P570, and P690 Storage Network

Traditional framework: midrange computer + disk array + commercial data warehouse

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 29
Application Scenarios of the Enterprise
Big Data Platform
Operation Management Supervision Profession
 Operation analysis  Performance  Public security network  Audio and video
 Telecom signaling management monitoring  Seismic prospecting
 Financial subledger  Report analysis  Public opinion monitoring  Weather nephogram
 Financial bill  History analysis  China Banking Regulatory  Satellite remote sensing
Social security analysis Commission (CBRC) Radar data
 Electricity distribution  

Tax analysis inspection IoT

 Smart grid  
 Food source tracing
 Decision-making support
 Environment monitoring
and prediction

Telecom and Finance Government Government

finance
Structured Structured + semi- Structured + semi- Non-structured
structured structured

With strong appeals for data analysis in telecom carriers, financial institutions, and
governments, new technologies have been adopted on the Internet to process big
data of low value density.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 30
Challenges Faced by Enterprises (1)
 Challenge 1: Business departments do not have clear
requirements on big data.

Many enterprises' business departments are not

familiar with big data as well as its application
scenarios and benefits. Therefore, it is difficult for
them to provide accurate requirements on big data.
The requirements of business departments are not
clear. The big data departments are non-profit
departments. Therefore, enterprises' decision-makers
worry about low input-output ratio and hesitate to
construct a big data department. Moreover, they
even delete lots of valuable historical data because
there is no application scenario.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 31
Challenges Faced by Enterprises (2)
 Challenge 2: Serious data silo problems within
enterprises

The most important challenge faced by

enterprises in implementing big data is data
fragments. In large-scale enterprises, different
types of data are often scattered in different
departments, so that the same data within an
enterprise cannot be shared and the value of big
data cannot be fully used.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 32
Challenges Faced by Enterprises (3)
 Challenge 3: Low data availability and poor
quality
The problem
locating time is
decreased Many large and medium enterprises generate a
by 50%.
Manual checks are
decreased due to large amount of data each day. However, some
self-service on
problem
detection.
enterprises pay no attention to big data
Availability is The service revenue
improved is improved more preprocessing, resulting in nonstandard data
Manual by 10%. than 20%.
participation is not
required due to
processing. During big data preprocessing, data
proactive problem
detection. needs to be extracted and converted into data
The time spent in that is easy to be processed, cleaned, and
identifying
problems
is reduced by
denoised to obtain valid data. According to data
90%.
from Sybase, if high-quality data availability
improves by 10%, enterprise revenue will
improve more than 10%.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 33
Challenges Faced by Enterprises (4)
 Challenge 4: Data-related management technology and architecture
 Traditional databases cannot process hundreds of TB-scale data
or above.
 Data diversities are not considered in traditional databases. In
particular, the compatibility of structured data, semi-structured
data, and non-structured data is not considered.
 Traditional databases do not have high requirements on the
data processing time. However, big data needs to be processed
in real time.
 O&M of massive data needs to ensure data stability, supports
high concurrency, and reduces the server load.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 34
Challenges Faced by Enterprises (5)
 Challenge 5: Data security

Network-based lives let criminals obtain

personal information easily, and also lead
to more crime methods that are difficult to
be tracked and prevented.

How to ensure personal information security becomes an important subject in the big data era. In
addition, with the continuous increase of big data, requirements on the security of physical devices
for storing data as well as on the multi-copy and disaster recovery mechanism of data will become
higher and higher.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 35
Challenges Faced by Enterprises (6)
 Challenge 6: Insufficient big data talents

Each step of big data construction must be completed by professionals. Therefore, it is necessary
to develop and build a professional team that understands big data, knows much about
administration, and has experience in big data applications. Hundreds of thousands of big data-
related jobs are increased globally ever year. More than 1 million talent gaps will appear in the
future. Therefore, universities and enterprises make joint efforts to develop and mining talents.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 36
Challenges Faced by Enterprises (7)
 Challenge 7: Tradeoff between data openness and
Legislative
privacy protection

Today, with the increased importance of big data

Electronic personal
applications, opening and sharing of data resources information of citizens
has become a key factor in maintaining advantages
during data wars. However, the data opening will
inevitably infringe the privacy of some users. How
to effectively protect the privacy of citizens and
enterprises and gradually strengthen the privacy
legislation while promoting all-round data opening,
applications, and sharing will be a major challenge
in the big data era.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 37
From Batch Processing to Real-Time
Analysis
Hadoop is a basis for batch processing of big data, but Hadoop cannot provide real-
time analysis.

Real-time intelligentization of highly integrated high-value information and knowledge

is a main commercial requirement.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 38
Hadoop Reference Practice in the
Industry

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 39
In-Memory Computing Reference
Practice in the Industry
Google PowerDrill

 Based on column-oriented storage,

PowerDrill uses the in-memory computing
technology to deliver the performance of
querying trillions of cell data per second,
which is 10 to 100 times of the traditional
column-oriented storage performance.

 PowerDrill can quickly skip unnecessary

data blocks. Compared with the full
scanning performance, the PowerDrill
performance is improved by 100 times.

 Data memory usage space can be optimized

and reduced to 1/16 using the compression
and encoding technologies.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 40
Stream Computing Reference Practice
in the Industry

IBM InfoSphere Streams is one of the core components of IBM's big data strategy,
supports high-speed processing of structured and unstructured data, processing data
in motion, throughput of millions of events per second, high expansibility, and the
streams processing language (SPL).
HStreaming conducted a streaming reconstruction on the Hadoop MapReduce framework.
The reconstructed Hadoop MapReduce framework is compatible with the existing
mainstream Hadoop infrastructures. The Hadoop MapReduce framework processes data in
streaming MapReduce mode under the premise of making no/tiny changes on the
framework. Gartner rated HStreaming as the coolest ESP vendor. Now, the reconstructed
framework supports text and video processing using the Apache Pig language (that is, Pig
Latin) and provides the high scalability of Hadoop, throughput of millions of events per
second, and millisecond-level delay.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 41
Opportunities in the Big Data Era
Opportunity: The big data blue ocean strategy becomes a new focus
of enterprise competition.

The huge commercial value brought by big data

will lead a great transformation that is equal in
force with the computer revolution in the
twentieth century. Big data is affecting each
field, including commercial, economic, and
other fields. Big data is promoting the
generation of a new blue ocean, bringing new
growth point of economy, and is becoming a
new focus of enterprise competition.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 42
Talents Required During the
Development of Big Data
 Big data system R&D engineers
 Big data application development
engineers
 Big data analysts
 Data visualization engineers
 Data security R&D engineers
 Data science research talents

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 43
 Contents
1. Big Data Era

2. Big Data Application Scope

3. Opportunities and Challenges in the Big Data Era

4. Huawei Big Data Solution

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 44
Huawei Big Data Platform
Architecture
Application service layer
OpenAPI/SDK REST/SNMP/Syslog

Data Information Knowledge Wisdom

DataFarm Porter Miner Farmer Manager
System
management
Hadoop API Plugin API
Service
governance
Hive MapReduce Spark Storm Flink
Hadoop LibrA
Yarn/ ZooKeeper Security
management
HDFS/HBase

 The Hadoop layer provides real-time data processing environment, which is enhanced based on the community open
source software.
 The DataFarm layer provides end-to-end data insight and builds the data supply chain from data to information,
knowledge, and wisdom, including Porter for data integration services, Miner for data mining services, and Farmer for
data service frameworks.
 Manager is a distributed management architecture. The administrator can control distributed clusters from a single
access point, including system management, data security management, and data governance.

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 45
Core Capabilities of Huawei Big Data
Team Be able to
establish top-level
projects that are
adaptable to the
Be able to take the eco-system in the
Be able to lead in the
independently communities
communities and
complete develop future-
kernel-level oriented kernel
development features
for critical
service features
Be able to resolve
kernel-level
Be able to problems by team
resolve kernel-
level problems
(outstanding Large number of
Be able to individuals)
locate components and code
peripheral Frequent component
problems updates
Be able Apache open-source
to use community ecosystem Efficient feature
Hadoop integration

Outstanding product development and delivery capabilities as well as carrier-class operation support capabilities empowered by
the Hadoop kernel team

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 46
Big Data Platform Partners from Finance
and Carrier Sectors

Industrial and China Pacific China China

Commercial Merchants Insurance Mobile Unicom
Bank of China Bank (CMB) Co., Ltd.
(ICBC) (CPIC)

Top 3 50%
China Telco Top 10 Customers in China's
Financial Industry

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 47
 Summary
These slides introduce:
 The big data era
 Applications of big data in all walks of life
 Opportunities and challenges brought by big data
 Huawei big data solution

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 48
 Quiz
1. Where is big data from? What are the features of big data?

2. Which social fields can big data be applied to?

3. What is Huawei big data solution called?

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 49
 More Information
 Training materials:
 http://support.huawei.com/learning/Certificate!showCertificate?lang=en&pbiPath=term100002
5450&id=Node1000011796
 Exam outline:
 http://support.huawei.com/learning/Certificate!toExamOutlineDetail?lang=en&nodeId=Node10
00011797
 Mock exam:
 http://support.huawei.com/learning/Certificate!toSimExamDetail?lang=en&nodeId=Node10000
11798
 Authentication process:
 http://support.huawei.com/learning/NavigationAction!createNavi#navi[id]=_40

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 50
 Thank You
www.huawei.com

Copyright © 2018 Huawei Technologies Co., Ltd. All rights reserved. Page 51