1.

Bias
 Definition: Bias is the error that occurs when a model is too
simple and cannot capture the underlying pattern in the data.
 Bias arises when the model makes assumptions about the data
that are too strong, leading to underfitting.
 Characteristics:
o Predictions are far off from the actual values.
o The model has a high systematic error.
 Example: Using a straight line (linear model) to fit data that
follows a curve. The model is too simple and doesn’t capture
the curve.
2. Variance
 Definition: Variance is the error that occurs when a model is
too complex and learns not only the patterns but also the noise
in the training data.
 Variance arises when the model is too flexible and overfits the
training data, performing poorly on unseen data.
 Characteristics:
o Predictions vary a lot when the model is trained on
different datasets.
o The model has low generalization to new data.
 Example: Using a very wiggly curve (overfitting) to fit a small
dataset with noise. The model learns the noise, not the true
trend.
3. Model Complexity
 Definition: Model complexity refers to how sophisticated or
flexible a model is.
 Simple models (low complexity) have low flexibility, like linear
regression, and make strong assumptions.
 Complex models (high complexity) are very flexible, like high-
degree polynomials or deep neural networks, and can fit almost
any data.
 Impact:
o Simple models: High bias, low variance (underfitting).
o Complex models: Low bias, high variance (overfitting).

4. Bias-Variance Trade-off
 Definition: The bias-variance trade-off describes the balance
between bias and variance to achieve the best model
performance.
 The goal is to minimize the total error, which is the sum of:
o Bias²: Error from underfitting.
o Variance: Error from overfitting.
o Irreducible Error: Noise in the data that cannot be
eliminated.
 Concept:
o Increasing model complexity reduces bias (model fits the
data better).
o However, increasing complexity also increases variance
(model learns noise).
 o A good model strikes a balance where both bias and
variance are low.
Example of Bias-Variance Trade-off
 High Bias Example:
o A simple linear model trying to fit a nonlinear dataset.
o It underfits the data, resulting in high error (bias).
 High Variance Example:
o A very complex model (e.g., high-degree polynomial) that
fits every data point, including noise.
o It overfits the data, performing poorly on new data.
 Balanced Example:
o A moderately complex model that captures the main
patterns without overfitting.
1. Bayesian Approach
 Definition: The Bayesian approach is a method in statistics that
combines prior knowledge (what we believe before seeing
data) with evidence from the data to make predictions or
decisions.
 How it works:
1. Start with a prior belief (what you already know or
assume).
2. Collect data (new evidence).
3. Update the belief using Bayes’ Theorem to get a posterior
belief (more accurate knowledge).
 Example:
o Before checking the weather, you believe there’s a 50%
chance of rain (prior belief).
o After seeing dark clouds (new evidence), you update your
belief to a higher chance of rain (posterior belief).

2. Cross-Validation
 Definition: Cross-validation is a method used to check how well
a model will perform on new, unseen data.
 How it works:
1. Split your data into k parts (called "folds").
2. Train the model on k-1 parts and test it on the remaining
part.
3. Repeat this process k times so each part is tested once.
4. Average the results to estimate the model's performance.
  Why it’s useful: It helps avoid overfitting and gives a better idea
of how the model performs on different data.
 Example: If you split data into 5 parts, you train on 4 parts and
test on the 5th, repeating this 5 times.

3. Bootstrap Methods
 Definition: Bootstrap is a method that creates many small
datasets by randomly selecting data points (with replacement)
from the original dataset. It is used to estimate the accuracy of
a model or a statistic.
 How it works:
1. Randomly select data points from your dataset with
replacement (some points can appear more than once).
2. Create many "new datasets" (bootstrap samples).
3. Train models or calculate statistics on each sample.
4. Combine the results to estimate performance or
variability.
 Why it’s useful: It helps understand the uncertainty or
variability in predictions.
 Example: If you have 100 data points, you create 100 new
datasets by sampling from the original data (some data points
are repeated).

4. Conditional or Expected Error

 Definition: Conditional or expected error refers to the average
error a model makes on new data, given a particular input.
 Types:
 o Expected Error: The error you expect on average for any
input.
o Conditional Error: The error for a specific input or set of
inputs.
 Why it matters: It helps measure how well a model will perform
on new, unseen data.
 Example:
o If a student takes many exams under the same conditions,
the expected error measures how far off their scores are,
on average, from the correct answer.