Unit No.

4 Statistical Methods
Standard deviation is a statistical measure of the dispersion or variability of a set of values. It
quantifies how much the values in a dataset deviate from the mean (average). A low standard
deviation indicates that the values tend to be close to the mean, while a high standard deviation
indicates that the values are spread out over a wider range.

Mathematically, the standard deviation (σ) is calculated using the following formula:

Key points about standard deviation:

1. Interpretation:
o A small standard deviation indicates that the values are close to the mean, while
a large standard deviation indicates that the values are spread out.
o It provides a measure of the average distance between each data point and the
mean.
o It is expressed in the same units as the original data.
2. Relationship with Variance:
o Standard deviation is the square root of variance. Variance is the average of the
squared differences from the mean, while standard deviation is the square root
of this average.
o Standard deviation is preferred over variance in some cases because it is in the
same units as the original data, making it easier to interpret.
3. Applications:
o Standard deviation is widely used in various fields, including finance,
engineering, natural sciences, and social sciences.
o It is used to quantify risk, assess the variability of data, evaluate the precision
of measurements, and determine the spread of distributions.

1. Normalization:
o Normalization is the process of scaling numerical features to a standard range
or distribution to ensure that they have a similar scale.
o It helps improve the performance and stability of machine learning algorithms,
especially those sensitive to the scale of input features.
 o Common normalization techniques include Min-Max scaling and Z-score
normalization.
2. Feature Scaling:
o Feature scaling is a specific type of normalization that involves scaling each
feature (or variable) individually to a similar range.
o It ensures that features with larger magnitudes do not dominate those with
smaller magnitudes during the learning process.
o Feature scaling is particularly important for algorithms that use distance-based
metrics or gradient descent optimization.
3. Min-Max Scaling:
o Min-Max scaling is a normalization technique that rescales features to a fixed
range, typically between 0 and 1.
o It subtracts the minimum value of the feature and then divides by the difference
between the maximum and minimum values.
o Min-Max scaling preserves the original distribution of the data but can be
sensitive to outliers.
4. Bias and Variance:
o Bias and variance are two types of errors in machine learning models that affect
their predictive performance.
o Bias refers to the error introduced by approximating a real-world problem with
a simplified model. High bias can lead to underfitting.
o Variance refers to the error introduced by the model's sensitivity to small
fluctuations in the training data. High variance can lead to overfitting.
o Balancing bias and variance is crucial for building models that generalize well
to unseen data.
5. Regularization:
o Regularization is a technique used to prevent overfitting by adding a penalty
term to the model's loss function.
o It discourages complex models that fit the training data too closely by imposing
constraints on model parameters.
o Common regularization techniques include L1 regularization (Lasso), L2
regularization (Ridge), and ElasticNet regularization.
o Regularization helps improve the generalization ability of models by reducing
variance at the cost of introducing some bias.

Normalization plays a crucial role in regularization techniques like Ridge Regression and Lasso
Regression. Let's explore how normalization interacts with these methods:

1. Ridge Regression:
o Ridge Regression is a linear regression technique that adds a penalty term to the
ordinary least squares (OLS) loss function.
o The penalty term is proportional to the squared magnitudes of the coefficients,
multiplied by a regularization parameter (λ or alpha).
o The goal of Ridge Regression is to shrink the coefficients towards zero,
effectively reducing the model's complexity and variance.
o Normalization is important in Ridge Regression because it ensures that all
features are on a similar scale, preventing features with larger magnitudes from
dominating the regularization term.
 o When using Ridge Regression, it's common to normalize the features using
techniques like Min-Max scaling or Z-score normalization before fitting the
model.
2. Lasso Regression:
o Lasso Regression (Least Absolute Shrinkage and Selection Operator) is another
linear regression technique that adds a penalty term to the OLS loss function.
o Unlike Ridge Regression, Lasso Regression uses the L1 norm of the coefficients
as the penalty term.
o Lasso Regression has a tendency to shrink some coefficients all the way to zero,
effectively performing feature selection.
o Normalization is also important in Lasso Regression to ensure that all features
are on a similar scale, as it influences the magnitude of the penalty applied to
each coefficient.
o Like Ridge Regression, it's common to normalize the features before applying
Lasso Regression to prevent any single feature from dominating the penalty
term.

Cross-validation (CV) techniques are used to evaluate the performance of machine learning
models and to tune model hyperparameters. Here are some common cross-validation
techniques:

1. K-fold Cross-Validation:
o In K-fold cross-validation, the original dataset is randomly partitioned into K
equal-sized subsets or "folds".
o The model is trained K times, each time using K-1 folds for training and the
remaining fold for validation.
o The performance metrics (e.g., accuracy, error) are then averaged over the K
folds to obtain an overall estimate of model performance.
o K-fold cross-validation helps to reduce the variability in the performance
estimate compared to a single train-test split.
2. Leave-One-Out Cross-Validation (LOOCV):
o LOOCV is a special case of K-fold cross-validation where K equals the number
of samples in the dataset.
o For each iteration, one data point is left out as the validation set, and the model
is trained on the remaining data.
o LOOCV is computationally expensive, especially for large datasets, but it
provides an unbiased estimate of model performance with low bias.
3. Stratified K-fold Cross-Validation:
o Stratified K-fold cross-validation ensures that each fold contains approximately
the same proportion of target classes as the original dataset.
o It is particularly useful for imbalanced datasets where one class is much more
prevalent than the others.
o Stratified K-fold helps to produce more reliable performance estimates,
especially when the target classes are unevenly distributed.
4. Grid Search Cross-Validation:
o Grid Search Cross-Validation is a technique used to tune hyperparameters by
exhaustively searching through a predefined grid of parameter values.
o For each combination of hyperparameters, the model is trained and evaluated
using K-fold cross-validation.
 oThe optimal hyperparameters are selected based on the performance metric
(e.g., accuracy) obtained during cross-validation.
5. Cross-Validation Error:
o Cross-validation error refers to the error estimate obtained during cross-
validation, typically averaged over multiple folds.
o It provides an unbiased estimate of the model's generalization error on unseen
data.
o Cross-validation error is commonly used to compare the performance of
different models or to select the best hyperparameters through techniques like
grid search.

Overall, cross-validation techniques are essential for assessing and optimizing the performance
of machine learning models, providing reliable estimates of model performance, and helping
to prevent overfitting.