BIG DATA

IMPLEMENTATION
BIG DATA SECURITY
 Introduction

 The development of Big data is still facing many

challenges, in which the risks of security and
privacy are recognized as one of t he most severe
problems.
 For example, shopping record is monitored by
Shopping Website; our mind and private photos are
recorded by social network; private property
information is under the control of Bank.
 Introduction

 Obviously, the leaking of this sensitive information

is serious. It seems that all your information and
actions are in the hands of others, which is very
horrible.
 Actually, sensitive data can easily be leaked if there
is no effective protection in its lifetime, including
data collection, storage and management, transport,
analysis, and data destruction
 Big Data Security
Definition
 Big Data security is the processing of guarding data
and analytics processes, both in the cloud and on-
premise, from any number of factors that could
compromise their confidentiality.
Big Data Security Concerns
 STORAGE AND
MANAGEMENT
 Since the birth of Big data, the scale of Big data has been changing
constantly.
 According to the concept of 5Vs proposed by IBM, Big data has the
characteristics of large scale and fast growth.
 Specifically, the size of data has increased from PB level to EB level, and
still expands according to the needs of the actual applications and the
development of enterprises, running toward ZB level at a rapid pace.
 STORAGE AND
MANAGEMENT
 We take Taobao, the biggest e-commerce company in China as an
example, until the end of 2011, the number of registered users is more
than 400 million
 Taobao generates more than 400 million product information every day,
to be more precise, daily active data has been more than 50 TB.
 Such a large amount of private data once leaked, the consequences will
be unimaginable.
 STORAGE AND
MANAGEMENT
 However, most of the existing Big data storage schemes are
stored in a distributed, horizontal scalable manner, such as
HDFS, Tachyon, QFS, Ceph, GlusterFS, and so on.
 Although these schemes meet the storable and scalable that
cloud environment require, sometimes they do not
guarantee concurrency and security needs of cloud
applications.
 Furthermore, the security of distributed data stored in
cloud server is challenging. Users are not convenient to
operate data directly in the cloud server, and they cannot
ensure the data is unavailable for unauthorized person.
 STORAGE AND
MANAGEMENT
 For ensuring cloud server secure, password of
account is employed for data security in
management.
 However, administrators of cloud server with high
authority may collude to share valuable Big data.
 How to control some of the managers with high
authority for data and passwords is also a challenge.
Meanwhile, there is a dramatic increasing need of
safer distributed schemes with trusted storage and
robustness
 TRANSMIT AND
SHARING
 The usage and sharing of user private data are lacking in
specifications, and lack of supervision, mainly rely on the self-
discipline of enterprises, which leads to the failure of client to
determine the purpose of their private information.
 In addition, the huge amount of Big data discussed before adds to the
transmission time and increases the risk of privacy data. In order to
reduce data risk during transmission, 2 solutions solve themselves.
 First, put the Big data “in place” and only transmit the analysis result.
In other words, client brings code to data rather than the traditional
ways that bring data into code. Second, classify data and transmit
only the data that is important to downstream analysis. In these 2
cases, integral and originated metadata are supposed to transmit
together with the actual data.
 TRANSMIT AND
SHARING
 Moreover, for superiority and
competitiveness in the business,
different companies will cooperate in
the form of customer and business data
sharing, which also adds to the risk of
client privacy disclosure.
 Still, most private companies and other
institutions are reluctant to share their
operating and user data. Obstacles may
include reputational considerations, as
well as a conservative culture
 ANALYTICS

 The analysis of Big data depends on the structure of the data, which
includes structured data, semi-structured data and unstructured
data.
 However, regardless of the structure of Big data has volume
characteristics. Hence, in data analysis, the biggest challenge is the
scalability of Big data
 In recent decades, researchers paid more and more attentions to
processor performance and accelerate analysis algorithms.
 ANALYTICS

 For the former, CPU (central processing unit) is embedded with more
and more cores, which lead to the progress in processor parallel
computing. For the later, it is important to develop online and multi-
resolution analysis methods for all machine learning technologies. These
2 aspects determine the long-term development and success of Big data.
 For Big data analysis, data encryption is an effective way to protect
privacy. However, the huge amount of data increases the security load.
 Moreover, distributed storage of Big data also increases the risk in data
analysis process.
 Specifically, for real-time Big data applications, such as social networks,
intelligent transportation, navigation, stock analysis, and so on, the
requirement of timeliness for data stream analysis is a greater challenge
to encryption methods and security protection.
 SOCIAL MEDIA

 Social network is one of the most important sources of

privacy data. Different from aforementioned schemes, user
data in social networks is typically stored and managed in a
graphical scheme.
 Hence, its anonymous protection in social network is very
different from structured data. Typical anonymous
protection needs in social networks are anonymous of user
identities and attributes.
 Hence, the latest popular anonymous social networks, such
as Whisper, Tinder, 4Chan, and Yik Yak, hide information
and relationship between the users when messages are send.
 Big Data Security Use
Cases
 Cloud Security Monitoring

 Cloud computing generally offers more efficient

communication and increased profitability for all
businesses.
 This communication needs to be secure. Big data
security offers cloud application monitoring. This
provides host sensitive data and also monitors
cloud-hosted infrastructure. Solutions also offer
support across several relevant cloud platforms.
 Big Data Security Use
Cases
 Network Traffic Analysis

 Traffic continually moves in and out of your network. Due to the

high volume of data over the network, it is difficult to maintain
transactional visibility over the network traffic.
 Security analytics allow your enterprise to watch over this network
traffic. It is used to establish baselines and detect anomalies.
 This also helps in cloud security monitoring. It is used to analyze
traffic in and out of cloud infrastructure.
 It also illuminates dark spaces that are hidden in infrastructures
and analyze encrypted sensitive data. Thus, ensuring the proper
working of channels.
 Big Data Security Use
Cases
 Insider Threat Detection

 Insider threats are as much as a danger to your enterprise as external

threats.
 An active malicious user can do as much damage as any malware
attack. But it is only in some rare cases that an insider threat can
destroy a network.
 With the help of security analytics, organizations can easily detect the
insider threats.
 This is anticipated through behaviors such as abnormal login times,
unusual email usage, and unauthorized database access requests.
Sometimes it also looks for indicators that ask for visibility to third-
party actors.
 Big Data Security Use
Cases
 Threat Hunting

 Generally, the IT Security team mostly engage in threat

hunting. They search for potential indicators of dwelling
threats and breaches that try to attack the IT infrastructure.
 Security analytics helps to automate this threat of hunting.
It acts as an extra set of eyes for your threat hunting efforts.
 Threats hunting automation can help in detecting malware
beaconing activity and thus alerts for its stoppage as soon
as possible.
 Big Data Security Use
Cases
 Incident Investigation

 Generally, the sheer number of security alerts from

certain software solutions would overwhelm your IT
security team.
 These continuous alerts can cause more fostering burnout
and frustration. Thus to minimize this issue, security
analytics automates the incident investigation by
providing contextualizations to alerts.
 Thus your team has more time to prioritize incidents and
can deal with potential breach incidents first.
 Big Data Security Use
Cases
 User Behaviour Analysis

 Organization’s users generally interact with your IT infrastructure

all the time. Mainly it is the user’s behavior that decides the success
or failure of your cybersecurity.
 Therefore there is a need for tracking user’s behavior. The security
analytics monitor the unusual behavior of employees. Thus it helps
to detect an insider threat or a malicious account. It can also detect
suspicious patterns by correlating malicious activities.
 An example of one such renowned security analytics use case is
UEBA. It helps to provide visibility into the IT environment. Thus
compiling user activities from multiple datasets into complete
profiles.
 Big Data Security Use
Cases
 Data Exfiltration Detection

 Data exfiltration is termed as any unauthorized

movement of data moving in and out of any
network. Unauthorized data movements can cause
theft and leakage of data.
 Thus there is a need to protect data from such
unauthorized access. The security analytics helps to
detect the data exfiltration over a network. It is
generally used to detect data leakage in encrypted
communications.
Big Data Security
Technologies

0
5

Intrusion
Detection &
Prevention
 Big Data Security
Technologies
 Encryption

 Encryption of data is generally done to secure a

massive volume of data, different types of data. It can
be user-generated or machine-generated code.
Encryption tools along with different analytics
toolsets format or code the data.
 They also get applied to data from different sources
like relational database management system
(RDBMS), specialized file systems like Hadoop
Distributed File System (HDFS), etc.
 Big Data Security
Technologies
 User Access Control

 It is the most basic network security tool. But few

companies practice this because it involves high
management overhead, this can be dangerous at the
network level and not good for the Big data
platforms.
 Automated strong user access control is a must for
organizations. Automation control manages complex
user control levels that protects the Big data platform
against the inside attack.
 Big Data Security
Technologies
 Physical Security

 Physical security should not be ignored. It is

generally built in when you deploy the Big data
platform in your own center.
 It can also be built around your cloud provider’s
data center security. They are important as they can
deny data center access to strangers or suspicious
visitors. Video surveillance and security logs are
also used for the same purpose.
 Big Data Security
Technologies
 Centralized Key Management

 It is one of the best security practices for many

years. It is applied in Big data environments,
especially on those having wide geographical
distribution.
 Best practices under centralized key management
include policy-driven automation, on-demand key
delivery, logging, and abstracting key management
from key usage.
 Big Data Security
Technologies
 Intrusion Detection and Prevention:

 Intrusion detection and prevention systems are

security workhorses. This does not make them any
less valuable to the big data platform. Big data’s
value and distributed architecture lends itself to
intrusion attempts. IPS enables security admins to
protect the big data platform from intrusion, and
should an intrusion succeed, IDS quarantine the
intrusion before it does significant damage.