Introduction to Data Science
Data All Around
• Lots of data is being collected
and warehoused
• Web data, e-commerce
• Financial transactions, bank/credit transactions
• Online trading and purchasing
• Social Network
How Much Data Do We have?
• Google processes 160 PB a day (2013)
• Google process more than 40000 searches/sec (2018)
• Facebook has 60 TB of daily logs
• Cost of 1 TB of disk: $35
• Time to read 1 TB disk: 3 hrs
(100 MB/s)
Big Data
Big Data is any data that is expensive to manage and
hard to extract value from
• Volume
• The size of the data
• Velocity
• The latency of data processing relative to the growing demand for
interactivity
• Variety and Complexity
• the diversity of sources, formats, quality, structures.
Big Data
CMS: Content Management System
SMS: Short Message Service
Types of Data We Have
• Relational Data (Tables/Transaction/Legacy Data)
• Text Data (Web)
• Semi-structured Data (XML)
• Graph Data
• Social Network, Semantic Web , …
• Streaming Data
What To Do With These Data?
• Aggregation and Statistics
• Data warehousing and OLAP
• Indexing, Searching, and Querying
• Keyword based search
• Pattern matching
• Knowledge discovery
• Data Mining
• Statistical Modeling
What is Data Science?
• A field that manages, manipulates, extracts, and interprets
knowledge from tremendous amount of data
• Data science (DS) is a multidisciplinary field of study with goal to
address the challenges in big data
• Data science principles apply to all data – big and small
What is Data Science?
• Theories and techniques from many fields and
disciplines are used to investigate and analyze a
large amount of data to help decision makers in
many industries such as science, engineering,
economics, politics, finance, and education
• Computer Science
• Pattern recognition, visualization, data warehousing, High
performance computing, Databases, AI
• Mathematics
• Mathematical Modeling
• Statistics
• Statistical and Stochastic modeling, Probability.
• Gartner’s 2014 Hype Cycle
Data Science
Data Science
Real Life Examples
• Companies learn your secrets, shopping patterns,
and preferences
• Data Science and election (2008, 2012)
• 1 million people installed the Obama Facebook app that
gave access to info on “friends”
Data Scientists
• Data Scientist
• May be the most paid Job of the 21st Century
• They find stories, extract knowledge. They are not reporters
Data Scientists
• Data scientists are the key to realizing the
opportunities presented by big data. They bring
structure to it, find compelling patterns in it, and
advise executives on the implications for products,
processes, and decisions
What do Data Scientists do?
• National Security
• Cyber Security
• Business Analytics
• Engineering
• Healthcare
• And more ….
Concentration in Data Science
• Mathematics and Applied Mathematics
• Applied Statistics/Data Analysis
• Solid Programming Skills (R, Python, Julia, SQL)
• Data Mining
• Data Base Storage and Management
• Machine Learning and discovery
Goal of Data Science
Turn data into data products.
Data Science – A Visual Definition
Contrast: Databases
Databases Data Science
Data Value “Precious” “Cheap”
Data Volume Modest Massive
Examples Bank records, Online clicks,
Personnel records, GPS logs,
Census, Tweets,
Medical records Building sensor readings
Priorities Consistency, Speed,
Error recovery, Availability,
Auditability Query richness
Structured Strongly (Schema) Weakly or none (Text)
Properties Transactions, ACID* CAP*,
eventual consistency
Realizations SQL NoSQL:
MongoDB, CouchDB,
Hbase, Cassandra, Riak, Memcached,
Apache River, …
ACID = Atomicity, Consistency, Isolation and Durability
CAP = Consistency, Availability, Partition
Tolerance
Contrast: Business Intelligence
Business Intelligence Data Science
Querying the past Querying the past
present and future
Contrast: Machine Learning
Machine Learning Data Science
Develop new (individual) models Explore many models, build and tune
hybrids
Prove mathematical properties of
models
Understand empirical properties of
models
Improve/validate on a few, relatively
Develop/use tools that can handle
clean, small datasets
massive datasets
Publish a paper Take action!
