Lec2

 Data Science Pipeline:

 The pipeline involves several steps: formulating questions, collecting data,
cleaning and preprocessing data, generating hypotheses, drawing inferences,
visualizing results, and evaluating solutions.
 Types of Data Science Problems:
 Regression: Predicting continuous outcomes (e.g., house prices).
 Classification: Categorizing data into predefined classes (e.g., spam
detection, disease diagnosis).
 Clustering: Grouping similar data (e.g., customer segmentation).
 Optimization: Finding the best solution by tuning variables (e.g.,
scheduling staff).
 Variables:
 Quantitative Variables: Numerical values like age, height, income.
 Qualitative Variables: Categories like marital status or product brand.
 Classification and Regression:
 Classification predicts labels based on features (e.g., predicting if a patient
has diabetes).
 Regression predicts continuous outcomes (e.g., stock prices or
temperatures).
 Optimization:
 Focuses on adjusting controllable factors (e.g., number of staff) to achieve
the best possible outcomes (e.g., reducing overtime in staff scheduling).
 Clustering:
 Used to group data without predefined labels. Example: Finding customer
segments based on purchase history.
 Comparison:
 Used to compare different groups or models (e.g., A/B testing to compare
marketing strategies).
 Data Description:
 Descriptive statistics (mean, standard deviation) and visualization are
essential for understanding the data before advanced analysis.
 Identifying outliers and understanding the distribution of variables helps in
data preparation.
 Data Science Project Life Cycle:
 Stages include problem formulation, data cleaning, feature engineering,
model development, testing (shadow mode), and A/B testing.
 Shadow Mode: Running the model in observation without making decisions
based on it.
  A/B Testing: Comparing control and treatment groups to evaluate model
performance.
 Common Mistakes in Data Science Projects:
 Solving the wrong problem.
 Poor data exploration.
 Poor model evaluation.
 Failing to scale the model for real-time applications.
 This covers the main points from the first lecture. Let me know if you'd like
more details on any specific part!


Questions in lec2

Memory updated
Here are the most important points from the first lecture, focusing on those that
could appear as exam questions and fully cover the lecture content:
1. What are the key steps in the Data Science Pipeline?
 Formulating the right questions.
 Collecting, cleaning, and preprocessing data.
 Generating hypotheses and drawing inferences.
 Visualizing findings and evaluating solutions.
2. Describe the types of Data Science Problems.
 Regression: Predicting continuous values (e.g., house prices, temperature).
 Classification: Categorizing data (e.g., spam detection, disease diagnosis).
 Clustering: Grouping similar data without predefined labels (e.g., customer
segmentation).
 Optimization: Finding the optimal solution by tuning controllable variables
(e.g., staff scheduling).
3. What is the difference between Quantitative and Qualitative Variables?
 Quantitative Variables: Numerical values like age, height, or income.
 Qualitative Variables: Categorical values like marital status, product
brands.
4. Explain Classification and provide an example.
 Classification categorizes data into predefined classes (e.g., predicting
whether a customer will buy a product).
 Example: Spam detection or disease diagnosis.
5. What is Regression, and how is it used in data science?
 Regression predicts continuous outcomes based on input features (e.g.,
predicting stock prices or temperature).
6. What is Optimization, and provide an example of its use.
 Optimization focuses on finding the best solution by tuning variables,
balancing controllable and uncontrollable factors.
 Example: Hospital staff scheduling to minimize overtime while ensuring
adequate coverage.
7. What is Clustering, and how is it applied in data science?
 Clustering groups data points based on similarity without predefined labels
(e.g., segmenting customers based on purchasing behavior).
8. Describe the process and purpose of A/B Testing.
 A/B Testing compares a control group with a treatment group to evaluate
the impact of a change or model on performance.
 It helps assess the effectiveness of a machine learning model in real-world
scenarios.
9. What are some common mistakes in Data Science Projects?
 Solving the wrong problem.
 Insufficient data exploration.
 Poor model evaluation.
 Failing to scale in real-time applications.
10. Explain Shadow Mode in a data science project.
 Shadow Mode is the observation period where a model runs without making
decisions, used to detect errors or issues before full deployment.
By focusing on these key points, you cover the most important exam-worthy topics
from the lecture. Let me know if you'd like to explore any of these topics in more
depth!
Lec3

Data Exploration:
 Purpose: Examine data using summary statistics and visualizations to
identify problems like missing values, invalid entries, and outliers before
model building.
 Key Elements: Use tools like means, medians, and variances, along with
graphs, to understand data patterns.
Data Cleaning:
 Importance: Ensuring clean data is crucial for accurate model predictions.
 Issues Identified: Common problems include missing data, invalid values,
and outliers, which need to be addressed before proceeding.
Handling Missing Values:
 Solutions: Missing values can be handled by creating new categories for
categorical data or replacing missing numerical values with averages or
other strategies.
Invalid Values and Outliers:
 Checking for Accuracy: Even if there are no missing values, it’s important
to verify data accuracy, ensuring there are no invalid entries or extreme
outliers (e.g., a customer age of 250).
 Outlier Handling: Ensure that outliers are managed correctly to maintain
the dataset's integrity.
Data Range and Variation:
 Ensuring Sufficient Variation: It's important to check that key variables,
like age or income, have enough variation to reveal potential relationships in
predictive models.
Expert Insights:
 Domain Expertise: Experts help identify inconsistencies or incorrect results
in the data that may not align with real-world expectations, adding
credibility to the data analysis.
Data Visualization:
 Role of Visuals: Data visualization tools (charts, graphs) are used to identify
trends, patterns, and anomalies in the data, making interpretation easier and
aiding in decision-making.
Statistical and Database Views:
 Statistical Perspective: Data is probabilistic, and the dataset is a sample of
larger processes. Adjusting for bias is essential for accuracy.
 Database Perspective: Recognizes common data issues (missing, corrupted,
duplicated data), emphasizing the need for data enhancement.
Visualization's Importance:
  Visuals are key for representing data clearly and spotting potential issues
like trends, outliers, and correlations.

Question of lec3
1. What is the importance of Data Exploration?
 Data Exploration uses summary statistics (means, variances) and
visualizations (graphs) to understand the dataset.
 Helps identify problems such as missing values, invalid entries, and outliers
before model building.
2. Why is Data Cleaning essential before Model Building?
 Data Cleaning ensures that the dataset is usable and accurate.
 Common issues include missing values, invalid data, and inconsistent
ranges, which must be addressed to avoid flawed models.
3. How are Missing Values handled in data science?
 Missing values can be problematic and must be dealt with through
strategies like:
o Creating a new category for categorical data (e.g., "Missing").
o Replacing missing numerical values with the mean or other
imputation methods.
4. What are Invalid Values and how are they identified?
 Invalid values are entries that don't make sense, such as negative values in
fields like age or income.
 These need to be identified and corrected to maintain the integrity of the
data.
5. What are Outliers, and why do they matter?
 Outliers are extreme values that fall far outside the expected range.
 Example: An age value of 250 when the realistic range is 18-80.
 Outliers must be handled carefully as they can distort model results.
6. Why is it important to check Data Range and Variation?
 Ensuring sufficient variation in key variables (like age or income) is crucial
for revealing relationships in the data.
 Without variation, the model may not be able to identify significant
patterns.
7. What is the Statistical View of data?
 Data is probabilistic, and every dataset is a sample from a larger process.
  Bias correction is often needed to ensure that the dataset represents the
population accurately.
8. How do Domain Experts contribute to Data Science?
 Domain experts can spot inconsistencies in the data that don’t match real-
world expectations.
 They help ensure the data and model outputs are realistic and trustworthy
by providing valuable insights.
9. What role does Data Visualization play in Data Exploration?
 Data Visualization (using charts, graphs, and maps) is critical for identifying
trends, outliers, and patterns in the data.
 It simplifies the process of interpreting complex datasets and enhances
decision-making.
10. What common issues arise from poor Data Exploration?
 Failing to explore the data thoroughly can lead to incorrect models, wasted
time, and poor results.
 Data exploration helps prevent reworking analyses later by identifying
issues early in the process.