Data Visualization

Unit-1
Chapter-1

Data Extraction, Cleaning, and Annotation:

1. Data Extraction:
o The process of gathering data from various sources such as databases, web
scraping, APIs, or flat files (e.g., CSV, Excel).
o Tools like SQL, Python libraries (e.g., pandas, BeautifulSoup), and ETL
(Extract, Transform, Load) frameworks are commonly used.
o Goal: Collect relevant, high-quality data required for analysis.
2. Data Cleaning:
o Involves identifying and correcting errors or inconsistencies in the dataset to
improve quality.
o Steps include handling missing data, removing duplicates, correcting data types,
and fixing inconsistencies.
o Tools: Python (pandas, NumPy), R, and specialized tools like OpenRefine.
o Example: Replacing null values with averages or medians, removing outliers,
or standardizing text formats.
3. Data Annotation:
o Process of labeling or tagging data to make it usable for machine learning
models or analysis.
o Examples include tagging images, annotating sentiment in text, or marking key
phrases.
o Tools: Label Studio, AWS SageMaker Ground Truth.
o Often used in supervised machine learning to create training datasets.

Data Integration, Reduction, and Transformation:

1. Data Integration:
o Combining data from multiple sources into a unified format or repository.
o Ensures consistency and removes redundancy.
o Examples: Merging sales and marketing data, integrating data from different
APIs.
o Tools: ETL tools like Talend, Informatica, or Python-based solutions.
2. Data Reduction:
o Reducing the volume of data while preserving its integrity and key features.
o Techniques:
▪ Feature selection: Choosing the most relevant attributes for analysis.
▪ Dimensionality reduction: Techniques like Principal Component
Analysis (PCA) or t-SNE.
▪ Sampling: Selecting a subset of the data.
o Goal: Improve computational efficiency and focus on significant patterns.
3. Data Transformation:
 o Converting data into a suitable format or structure for analysis.
o Includes normalization, standardization, encoding categorical variables, and
aggregating data.
o Example: Converting timestamps into day-of-week features or one-hot
encoding categorical data.

Role of Visualization in Data Processing:

1. Exploratory Data Analysis (EDA):

o Visualizations help identify patterns, trends, outliers, and relationships in the
data.
o Examples: Scatter plots to find correlations, box plots to detect outliers.
2. Improved Understanding:
o Translating complex datasets into easy-to-understand visual formats like bar
charts, line graphs, and heatmaps.
3. Decision Support:
o Enables stakeholders to make informed decisions based on clear visual
evidence.
4. Validation:
o Visualizations can confirm the effectiveness of data cleaning, transformation,
or integration.

Definitions and Basic Concepts:

• Dataset: A collection of data, often structured in rows and columns.

• Feature: An individual measurable property of data.
• Outlier: A data point significantly different from others in the dataset.
• Missing Data: Data that is not recorded or available in the dataset.
• Normalization: Scaling data to fall within a specific range, often [0, 1].
• Standardization: Scaling data to have a mean of 0 and a standard deviation of 1.

Overview of Basic Charts and Plots:

1. Line Chart:
o Used for displaying trends over time.
o Example: Stock prices over a year.
2. Bar Chart:
o Compares categorical data.
o Example: Sales across regions.
3. Histogram:
o Shows the distribution of a numerical variable.
o Example: Age distribution of customers.
4. Pie Chart:
o Displays proportions of a whole.
 o Example: Market share distribution.
5. Scatter Plot:
o Depicts relationships or correlations between two variables.
o Example: Age vs. income.
6. Box Plot:
o Summarizes data distribution and identifies outliers.
o Example: Exam scores distribution across classes.
7. Heatmap:
o Visualizes data intensity or density using color gradients.
o Example: Correlation matrix for variables.

Chapter-2

Multivariate Data Visualization

Multivariate data involves multiple variables or dimensions. Visualization helps to explore

relationships, patterns, and trends among these variables. Common techniques include:

1. Scatterplot Matrix:
o Displays scatterplots for all pairs of variables in a grid format.
o Useful for identifying correlations or clusters.
2. Parallel Coordinates:
o Represents each data point as a line passing through multiple parallel axes,
where each axis represents a variable.
o Helps in detecting patterns or outliers across multiple dimensions.
3. Heatmaps:
o Uses a matrix format where values are represented by varying colors.
o Example: Correlation matrix for identifying relationships between variables.
4. 3D Scatter Plots:
o Extends 2D scatter plots by adding a third dimension, often with interactive
rotation.
o Tools: Matplotlib (Python), Tableau.
5. Bubble Charts:
o Similar to scatter plots but adds a third variable through the size of the bubbles.

Pixel-Oriented Visualization Techniques

These techniques map each data value to a pixel or small graphical element. They are
particularly useful for large datasets.

1. Principle:
o Pixels are arranged in a way that maintains spatial relationships or highlights
patterns.
2. Examples:
o Recursive Patterns: Display data hierarchically, where each pixel represents a
data point, and clusters are recursively divided.
o Color-Coded Pixels: Each pixel’s color intensity represents the magnitude of a
value.
 3. Advantages:
o Handles large datasets efficiently.
o Allows for dense visual representation.
4. Limitations:
o May require zooming or interaction for detailed analysis.

Geometric Projection Visualization Techniques

These techniques reduce high-dimensional data into lower-dimensional spaces for

visualization while preserving key relationships.

1. Principle:
o Project high-dimensional data onto a 2D or 3D plane.
2. Techniques:
o Principal Component Analysis (PCA): Reduces dimensions by capturing the
directions of maximum variance.
o Multidimensional Scaling (MDS): Preserves pairwise distances between data
points in the projection.
o t-SNE (t-Distributed Stochastic Neighbor Embedding): Focuses on retaining
local relationships, ideal for clustering.
3. Applications:
o Visualizing clusters or patterns in high-dimensional datasets, such as gene
expression data or image embeddings.

Icon-Based Visualization Techniques

These use icons to represent multidimensional data, with each attribute mapped to a visual
property of the icon.

1. Examples:
o Chernoff Faces: Multivariate data is mapped to facial features like eyes, nose,
or mouth shapes.
o Star Glyphs: Variables are represented as rays extending from a central point,
with the length of each ray indicating the value.
o Stick Figures: Uses stick figures where limb positions or lengths represent
different attributes.
2. Advantages:
o Makes multivariate data intuitive and memorable.
o Useful for qualitative analysis and comparisons.
3. Limitations:
o Hard to interpret when data is dense or highly dimensional.

Hierarchical Visualization Techniques

These techniques are designed to represent data with inherent hierarchical structures, such as
organizational charts or file systems.

1. Types of Representations:
o Tree Diagrams: Traditional nodes and edges format to show parent-child
relationships.
o Treemaps: Uses nested rectangles to represent hierarchical levels, where size
and color encode data attributes.
o Sunburst Charts: Circular treemaps, where layers of the hierarchy are
represented as concentric rings.
2. Applications:
o Visualizing file systems, organizational structures, or biological taxonomies.
3. Advantages:
o Clear representation of hierarchical relationships.
o Facilitates navigation and comparison of sub-levels.

Visualizing Complex Data and Relationships

1. Complex Data:
o Includes data with intricate relationships, temporal variations, or spatial
components.
o Examples: Social networks, financial markets, or sensor data.
2. Techniques:
o Network Graphs: Use nodes and edges to represent entities and their
relationships. Examples include social networks or citation networks.
o Temporal Visualizations: Line graphs, Gantt charts, or time-series plots for
visualizing changes over time.
o Geospatial Visualizations: Maps with overlays of data points or heatmaps to
analyze location-based data.
3. Tools for Handling Complexity:
o Interactive dashboards (e.g., Tableau, Power BI).
o Libraries like D3.js, Plotly, and Gephi for custom visualizations.

Theories Related to Visual Information Processing

1. Gestalt Principles:
o Explains how humans perceive patterns and groupings in visual elements.
o Key principles:
▪ Proximity: Elements close together are perceived as a group.
▪ Similarity: Similar shapes, colors, or sizes are grouped.
▪ Closure: Incomplete shapes are perceived as complete.
▪ Continuity: The eye follows continuous lines or paths.
▪ Figure-Ground: Differentiating an object (figure) from its background.
2. Dual-Coding Theory:
o Suggests that humans process information through two systems: verbal and non-
verbal (visual).
o Combining visual and textual elements improves comprehension and memory
retention.
 3. Pre-Attentive Processing:
o Certain visual properties (e.g., color, size, orientation) are perceived quickly and
effortlessly.
o Useful for highlighting critical data points in a visualization.
4. Cognitive Load Theory:
o The human brain has limited capacity for processing information.
o Effective visualizations minimize unnecessary elements to reduce cognitive
load.
5. Color Perception Theory:
o Humans perceive colors differently based on context, brightness, and contrast.
o Using color effectively enhances clarity and prevents misinterpretation.

Colour Theory and Its Application

1. Basics of Color Theory:

o Primary Colors: Red, blue, yellow (traditional); red, green, blue (RGB for
digital).
o Secondary Colors: Formed by mixing primary colors (e.g., green, orange,
purple).
o Tertiary Colors: Mixing primary and secondary colors.
2. Color Models:
o RGB (Red, Green, Blue): Used for digital screens.
o CMYK (Cyan, Magenta, Yellow, Key/Black): Used for printing.
o HSV (Hue, Saturation, Value): Useful for selecting and understanding colors.
3. Color Harmonies:
o Complementary: Colors opposite each other on the color wheel (e.g., blue and
orange).
o Analogous: Colors adjacent to each other (e.g., green, yellow, blue).
o Triadic: Colors evenly spaced around the wheel (e.g., red, yellow, blue).
4. Applications in Visualization:
o Use contrasting colors to highlight differences.
o Employ sequential palettes for ordered data (e.g., light to dark for low to high
values).
o Use diverging palettes for data with a midpoint (e.g., temperature differences).
o Avoid overuse of color or relying on color alone, as it may be inaccessible to
colorblind users.

Data Types and Visual Variables

1. Data Types:
o Nominal: Categories without a natural order (e.g., gender, regions).
o Ordinal: Categories with a meaningful order (e.g., rankings, levels of
satisfaction).
o Interval: Numeric data without a true zero (e.g., temperature in Celsius).
o Ratio: Numeric data with a true zero (e.g., height, weight, income).
2. Visual Variables:
 o Position: Most effective for quantitative comparisons (e.g., bar positions on an
axis).
o Size: Represents magnitude (e.g., bubble chart).
o Shape: Differentiates categories (e.g., different markers in a scatterplot).
o Color: Represents categories or gradients (e.g., heatmaps).
o Orientation: Used for directional data (e.g., wind patterns).
o Texture: Adds detail in qualitative data.

Chart Types

1. Bar Charts:
o Compare categorical data.
o Variants: Stacked bar chart, grouped bar chart.
2. Line Charts:
o Track changes over time.
o Variants: Multi-series line charts.
3. Pie Charts:
o Show proportions of a whole. Best for simple datasets.
4. Scatter Plots:
o Explore relationships between two variables.
5. Bubble Charts:
o Add a third variable through bubble size.
6. Histogram:
o Display frequency distributions for continuous data.

Statistical Graphs

1. Box Plots:
o Summarize data distribution, including medians, quartiles, and outliers.
2. Histograms:
o Visualize data distributions by dividing it into bins.
3. Violin Plots:
o Combines box plots with a density plot for showing distribution shape.
4. Cumulative Distribution Function (CDF):
o Represents cumulative probabilities.

Maps

1. Types of Maps:
o Choropleth Maps: Use color gradients to represent data (e.g., population
density).
o Heat Maps: Show data density or intensity over a geographic region.
o Cartograms: Distort map areas to represent data magnitude.
o Flow Maps: Visualize movement or connections (e.g., migration patterns).
2. Applications:
o Geospatial data, demographic analysis, or logistics.
Trees and Networks

1. Trees:
o Represent hierarchical data.
o Visualizations:
▪ Tree Diagrams: Nodes and edges show relationships.
▪ Treemaps: Nested rectangles represent hierarchical proportions.
2. Networks:
o Represent relationships between entities.
o Visualizations:
▪ Node-Link Diagrams: Nodes represent entities, and edges represent
relationships.
▪ Adjacency Matrix: A grid format showing connections.
▪ Force-Directed Graphs: Automatically arrange nodes based on their
relationships.

Chapter-3

Acquisition of Data and Classification of Information Sources

1. Data Acquisition:
o The process of collecting data from various sources for further analysis.
o Methods:
▪ Manual Data Entry: Human-recorded data (e.g., survey responses).
▪ Automated Collection: Using sensors, web scraping, APIs, or data
logs.
▪ Third-Party Sources: Acquiring datasets from vendors, government,
or research organizations.
2. Classification of Information Sources:
o Primary Sources: Data collected directly from the source (e.g., experiments,
surveys).
o Secondary Sources: Data gathered from existing works (e.g., research papers,
reports).
o Tertiary Sources: Aggregations or summaries of primary and secondary
sources (e.g., encyclopedias).
o By Format:
▪ Structured (e.g., databases, spreadsheets).
▪ Semi-structured (e.g., JSON, XML).
▪ Unstructured (e.g., text, images).

Database Issues

1. In-Memory Database Storage:

o Definition: Databases that store data in RAM rather than on disk for faster
access.
o Advantages:
▪ Reduced latency and faster query execution.
▪ Ideal for real-time analytics and applications.
o Challenges:
 ▪Limited by RAM capacity.
▪Requires robust backup mechanisms to prevent data loss in case of
power failure.
2. Data Retrieval:
o Efficiently accessing data from databases using indices, caching, and optimized
queries.
o Common retrieval methods:
▪ Indexed lookups.
▪ Full-text searches for unstructured data.
3. Query Languages:
o Tools for interacting with databases to retrieve or manipulate data.
▪ SQL (Structured Query Language): For relational databases like
MySQL, PostgreSQL.
▪ NoSQL Queries: For non-relational databases like MongoDB,
Cassandra.
▪ Graph Query Languages: Cypher for Neo4j or Gremlin for graph
databases.

Ensuring Reliability of Data Patterns

1. Reliability in Data Patterns:

o Refers to the consistency and accuracy of patterns detected in data analysis.
o Techniques to ensure reliability:
▪ Data Validation: Ensuring data input adheres to predefined rules.
▪ Cross-Validation: Splitting data into training and testing sets to validate
model performance.
▪ Noise Reduction: Removing irrelevant or erroneous data points.
▪ Statistical Testing: Using techniques like hypothesis testing to confirm
patterns are significant.
2. Challenges:
o Overfitting: Models capturing noise instead of true patterns.
o Bias in Data: Skewed datasets leading to unreliable patterns.

Predicting Continuous and Discontinuous Variables

1. Continuous Variables:
o Variables that can take any value within a range (e.g., temperature, sales).
o Prediction Techniques:
▪ Linear Regression: Models the relationship between variables using a
straight line.
▪ Polynomial Regression: Fits a polynomial curve for non-linear
relationships.
▪ Neural Networks: Advanced models for complex, non-linear patterns.
2. Discontinuous (Categorical) Variables:
o Variables that take discrete values (e.g., yes/no, classes).
o Prediction Techniques:
▪ Logistic Regression: For binary outcomes.
▪ Decision Trees: Splits data into categories based on conditions.
▪ Naive Bayes: Based on probability distribution.
 ▪ Support Vector Machines (SVMs): Finds decision boundaries for
classification tasks.

Techniques for Plotting Data

1. Exploratory Data Analysis (EDA):

o Visual methods to understand data distributions, trends, and relationships.
▪ Histograms: For frequency distributions.
▪ Box Plots: To detect outliers and summarize distributions.
2. Advanced Techniques:
o Heatmaps: For visualizing correlations or density.
o Violin Plots: Combines box plots and distribution curves.
o Interactive Plots: Tools like Plotly and Tableau for dynamic exploration.
3. Temporal Data:
o Line Charts: To track trends over time.
o Time-Series Decomposition: Separating trends, seasonality, and noise.
4. Geospatial Data:
o Choropleth Maps: Visualizing values across geographical regions.
o Scatter Maps: For plotting data points on a map.

Evaluating Suitability for Different Data Types

1. Data Types and Visualization:

o Nominal Data: Use bar charts or pie charts to represent categories.
o Ordinal Data: Use bar charts with ordered categories.
o Interval/Ratio Data: Use histograms, scatter plots, or line charts.
2. Criteria for Suitability:
o Data Volume:
▪ Large datasets may require aggregation (e.g., heatmaps, treemaps).
o Dimensionality:
▪ High-dimensional data may require dimensionality reduction (e.g.,
PCA, t-SNE).
o Relationship Type:
▪ Correlations: Use scatter plots or correlation matrices.
▪ Hierarchies: Use tree maps or hierarchical charts.
3. Challenges:
o Misrepresentation of data due to poor visualization choices.
o Loss of information during dimensionality reduction.