Big Data Architecture

BIG DATA ADVANTAGES

• Understanding and Targeting Customers
• big data is used to better understand
customers and their behaviors and
preferences
• Understanding and Optimizing Business
Process
– big data tool.
• Improving Science and Research
• Improving Healthcare and Public Health
• Optimizing Machine and Device Performance
 BIG DATA ADVANTAGES
Financial Trading
Improving Security and Law Enforcement
Big data technologies and
management
 Evolution of big data
• first Big Data challenge came in the form of the
1880 U.S. census, when the information
concerning approximately 50 million people
had to be gathered, classified, and reported on
with—particular elements, such as age, sex,
occupation, education level.
• 1880 census, so it took over seven years to
manually tabulate and report on the data.
• In 1890, introduction of the first Big Data
platform: a mechanical device called the
 Evolution of big data
• Hollerith Tabulating System, which worked
with punch cards that could hold about 80
variables.
• Analysis now took six weeks instead of
seven years.
• For the U.S. government, the ability to analyze
the 1890 census led to an improved
understanding of the populace, which the
government could use to shape economic and
social policies ranging from taxation to
education to military conscription.
 Evolution of big data
• The next giant leap for Big Data analytics came
with the Manhattan Project, the U.S.
development of the atomic bomb during
World War II.
• The next largest Big Science project began in
the late 1950s with the launch of the U.S.
space program.
 Evolution of big data
• As the term Big Science gained currency in the
1960s, the Manhattan Project and the space
program became paradigmatic examples.
• International Geophysical Year, an
international scientific project that lasted from
July 1, 1957, to December 31, 1958,provided
scientists with an alternative model: a
synoptic collection of observational data on a
global scale
 Evolution of big data
• International Geophysical Year project,
International Biological Program and later the
Long-Term Ecological Research Network
• 1974, when the program ended, many
participants viewed it as a failure.
• The lessons learned from the birth of Big
Science spawned new Big Data projects:
weather prediction, physics research
 Evolution of big data
• (supercollider data analytics), astronomy images
(planet detection), medical research (drug
interaction), and many others.
• Big Business uses Big Data to discover new
opportunities, measure efficiencies, or uncover
relationships among what was thought to be
unrelated data sets.
 Evolution of big data
• Perhaps Google CEO Erik Schmidt said it best: “Every
two days now we create as much information as we
did from the dawn of civilization up until 2003. That’s
something like five exabytes of data.” An exabyte is
an incredibly large, almost unimaginable amount of
information: 10 to the 18th power. Think of an
exabyte as the number 1 followed by 18 zeros.
 Evolution of big data
• Farnam Jahanian, the assistant director for computer
and information science and engineering for the
National Science Foundation (NSF), kicked off a May 1,
2012, briefing about Big Data on Capitol Hill by calling
data “a transformative new currency for science,
engineering, education, and commerce.”
 Evolution of big data
TechAmerica, brought together a panel of
leaders from government and industry to
discuss the opportunities for innovation
arising from the collection, storage, analysis,
and visualization of large, heterogeneous data
sets, all the while taking into consideration the
significant security and privacy implications.
 Evolution of big data
• Jahanian further explained the implications of
the modern era ofBig Data with three specific
points:
• First, insights and more accurate predictions
from large and complex collections of data
have important implications for the economy.
Access to information is transforming
traditional businesses and is creating
opportunities in new markets.
 Evolution of big data
• Big Data is driving the creation of new IT products and services
based on business intelligence and data analytics and is
boosting the productivity of firms that use it to make better
decisions and identify new business trends.
• Second, advances in Big Data are critical to accelerate
the pace of discovery in almost every science and engineering
discipline. From new insights about protein structure,
biomedical research and clinical decision making, and climate
modeling to new ways to mitigate and respond to natural
disasters and new strategies for effective learning and
education, there are enormous opportunities for datadriven
discovery.
 Evolution of big data
• Third, Big Data also has the potential to solve
some of the nation’s most pressing
challenges—in science, education,
environment and sustainability, medicine,
• commerce, and cyber and national security—
with enormous societal benefit and laying the
foundations for U.S. competitiveness for many
decades to come
 Evolution of big data
• White House Office of Science and Technology
Policy, together with other agencies,
announced a $200 million Big Data R&D
initiative to advance core techniques and
technologies.
• Big Data science and engineering
• involves the following:
• Advances in foundational techniques and
technologies (i.e., new methods) to derive
knowledge from data.
 Evolution of big data
• Nurturance of new types of collaborations—
multidisciplinary teams and communities
enabled by new data access policies—to
• make advances in the grand challenges of the
computation- and data-intensive world today.
 Evolution of big data
• IBM is now working with financial giant Citi to
explore how the Watson technology could
improve and simplify the banking experience.
Watson’s applicability doesn’t end with
banking, however; IBM has also teamed up
with health insurer WellPoint to turn Watson
into a machine that can support the doctors of
the world.
• IBM has stated that 90 percent of the world’s
data was created in the last two years, and 80
 Evolution of big data
• Watson-as-a-Service— which will be delivered
as a private or hybrid cloud service
• Examples abound for the benefits of Big Data
and the medical field; Health care (or in this
context, “Big Medicine”) has some specific
challenges to overcome and some specific
goals to achieve to realize the potential of Big
Data
• Technologies
 Evolution of big data
• Decision support needs to be easier to access.
• Information needs to flow more easily.
• Quality of care needs to be increased while
driving costs down
• The physician–patient relationship needs to
improve.
• Most approaches to dealing with large data
sets within a classification learning paradigm
• attempt to increase computational efficiency.
 Evolution of big data
• Agriculture
• In 30 years, the world’s population is
estimated to grow ahead of what our food
supply can support.
IBM Watson IoT analyzes a variety of data like
temperature, soil pH and other agricultural
and environmental factors to give farmers
insights that can help them make better
decisions — and harvest greater yields.
Classification Algorithms
 Support and confidence
• Support :
• GiventheassociationruleX1,...,Xn=>Y,thesuppo
rtisthepercentageofrecordsforwhichX1,...,Xn
andY both hold.
The statistical significance of the association
rule.
• Confidence:
• Given the association rule
 Support and confidence
• X1,...,Xn=>Y,the confidence is the percentage
of records for which Y holds,within the group
of records for whichX1,...,Xn hold.
• The degree of correlationin the dataset
between X and Y.
• A measure of the rule's strength.