‭Supervised Learning - Classification Algorithms‬

‭ upervised learning classification algorithms are used to predict categorical or discrete‬

S
‭target variables based on a set of input features. Here are some common types of‬
‭classification algorithms:‬

‭‬
● ‭ ogistic Regression‬
L
‭●‬ ‭Support Vector Machines (SVM)‬
‭●‬ ‭K - Nearest Neighbors (KNN)‬
‭●‬ ‭Decision Trees‬
‭●‬ ‭Random Forests‬
‭●‬ ‭Xgboost etc.‬

‭Logistic Regression‬
‭ ogistic‬‭regression‬‭is‬‭a‬‭popular‬‭statistical‬‭algorithm‬‭used‬‭for‬‭binary‬‭classification‬‭problems.‬‭It‬
L
‭is‬‭a‬‭supervised‬‭learning‬‭algorithm‬‭that‬‭predicts‬‭the‬‭probability‬‭of‬‭an‬‭input‬‭belonging‬‭to‬‭one‬‭of‬
‭two classes, typically represented as 0 or 1.‬

‭ he‬ ‭key‬ ‭idea‬ ‭behind‬ ‭logistic‬ ‭regression‬ ‭is‬ ‭to‬ ‭model‬ ‭the‬ ‭relationship‬ ‭between‬ ‭the‬ ‭input‬
T
‭features‬ ‭and‬ ‭the‬ ‭probability‬ ‭of‬ ‭the‬ ‭input‬ ‭belonging‬ ‭to‬ ‭a‬ ‭specific‬ ‭class.‬ ‭Unlike‬ ‭linear‬
‭regression,‬ ‭which‬ ‭predicts‬ ‭a‬ ‭continuous‬ ‭value,‬ ‭logistic‬ ‭regression‬ ‭uses‬ ‭a‬ ‭logistic‬ ‭function‬
‭(also called a sigmoid function) to map the output to a value between 0 and 1.‬

‭Here's how logistic regression works in simple terms:‬

‭●‬ D‭ ata‬‭Preparation:‬‭You‬‭start‬‭with‬‭a‬‭dataset‬‭that‬‭contains‬‭input‬‭features‬‭(independent‬
‭variables)‬ ‭and‬ ‭corresponding‬ ‭class‬ ‭labels‬ ‭(0‬ ‭or‬ ‭1).‬ ‭The‬ ‭features‬ ‭could‬ ‭be‬ ‭any‬
‭measurable characteristics or attributes that are relevant to the classification task.‬

‭●‬ M‭ odel‬ ‭Training:‬ ‭During‬ ‭the‬ ‭training‬ ‭phase,‬ ‭logistic‬ ‭regression‬ ‭estimates‬ ‭the‬
‭parameters‬ ‭(coefficients)‬‭that‬‭define‬‭the‬‭relationship‬‭between‬‭the‬‭input‬‭features‬‭and‬
‭the‬‭class‬‭probabilities.‬‭It‬‭uses‬‭a‬‭process‬‭called‬‭maximum‬‭likelihood‬‭estimation‬‭to‬‭find‬
‭the optimal parameter values that maximize the likelihood of the observed data.‬

‭●‬ L‭ ogistic‬‭Function:‬‭The‬‭logistic‬‭function‬‭is‬‭the‬‭core‬‭component‬‭of‬‭logistic‬‭regression.‬
‭It‬‭transforms‬‭the‬‭linear‬‭combination‬‭of‬‭input‬‭features‬‭and‬‭their‬‭associated‬‭coefficients‬
‭into a value between 0 and 1. The logistic function equation is:‬

‭P(Y=1|X) = 1 / (1 + exp(-(b0 + b1X1 + b2X2 + ... + bn*Xn)))‬

I‭n‬‭this‬‭equation,‬‭P(Y=1|X)‬‭represents‬‭the‬‭probability‬‭of‬‭the‬‭input‬‭belonging‬‭to‬‭class‬‭1‬
‭given‬‭the‬‭input‬‭features‬‭X.‬‭b0,‬‭b1,‬‭b2,‬‭...,‬‭bn‬‭are‬‭the‬‭coefficients‬‭associated‬‭with‬‭each‬
‭feature, and X1, X2, ..., Xn are the corresponding feature values.‬
 ‭●‬ D‭ ecision‬ ‭Boundary:‬ ‭To‬ ‭make‬ ‭a‬ ‭prediction,‬ ‭logistic‬ ‭regression‬ ‭uses‬ ‭a‬ ‭decision‬
‭boundary.‬ ‭Typically,‬ ‭if‬ ‭the‬ ‭predicted‬ ‭probability‬ ‭is‬ ‭greater‬‭than‬‭a‬‭specified‬‭threshold‬
‭(often‬‭0.5),‬‭the‬‭input‬‭is‬‭classified‬‭as‬‭belonging‬‭to‬‭class‬‭1;‬‭otherwise,‬‭it‬‭is‬‭classified‬‭as‬
‭belonging to class 0.‬

‭●‬ M‭ odel‬‭Evaluation:‬‭After‬‭training,‬‭you‬‭evaluate‬‭the‬‭logistic‬‭regression‬‭model‬‭using‬‭a‬
‭separate‬ ‭dataset‬ ‭to‬ ‭assess‬ ‭its‬ ‭performance.‬ ‭Common‬ ‭evaluation‬ ‭metrics‬‭for‬‭binary‬
‭classification include accuracy, precision, recall, and F1-score.‬

‭ ogistic‬‭regression‬‭is‬‭widely‬‭used‬‭due‬‭to‬‭its‬‭simplicity‬‭and‬‭interpretability.‬‭It‬‭can‬‭handle‬‭both‬
L
‭linear‬ ‭and‬ ‭nonlinear‬ ‭relationships‬ ‭between‬ ‭the‬ ‭input‬ ‭features‬ ‭and‬ ‭the‬ ‭class‬ ‭probabilities.‬
‭However,‬ ‭it‬ ‭assumes‬ ‭that‬ ‭the‬ ‭relationship‬ ‭is‬ ‭log-linear,‬ ‭meaning‬ ‭the‬ ‭logarithm‬ ‭of‬ ‭the‬ ‭odds‬
‭ratio is a linear function of the input features.‬
 ‭Types of Logistic Regression‬
‭ here‬ ‭are‬‭different‬‭types‬‭or‬‭variations‬‭of‬‭logistic‬‭regression‬‭that‬‭can‬‭be‬‭used‬‭depending‬‭on‬
T
‭the‬ ‭specific‬ ‭requirements‬ ‭of‬ ‭the‬ ‭classification‬ ‭problem.‬ ‭Here‬ ‭are‬ ‭some‬ ‭common‬ ‭types‬ ‭of‬
‭logistic regression:‬

‭1.‬ B‭ inary‬ ‭Logistic‬ ‭Regression:‬ ‭Binary‬ ‭logistic‬ ‭regression‬ ‭is‬ ‭the‬ ‭most‬ ‭basic‬ ‭form‬ ‭of‬
‭logistic‬ ‭regression.‬ ‭It‬ ‭is‬ ‭used‬ ‭when‬ ‭the‬ ‭target‬ ‭variable‬ ‭has‬ ‭only‬ ‭two‬ ‭classes‬ ‭or‬
‭categories.‬‭The‬‭logistic‬‭function‬‭in‬‭binary‬‭logistic‬‭regression‬‭maps‬‭the‬‭input‬‭features‬
‭to the probability of belonging to one of the two classes.‬

‭2.‬ M‭ ultinomial‬‭Logistic‬‭Regression:‬‭Multinomial‬‭logistic‬‭regression‬‭is‬‭used‬‭when‬‭the‬
‭target‬ ‭variable‬ ‭has‬ ‭more‬ ‭than‬ ‭two‬ ‭mutually‬ ‭exclusive‬ ‭classes.‬ ‭It‬ ‭extends‬ ‭binary‬
‭logistic‬‭regression‬‭to‬‭handle‬‭multi-class‬‭classification‬‭problems.‬‭The‬‭logistic‬‭function‬
‭is modified to handle multiple classes, typically using the softmax function.‬

‭3.‬ O‭ rdinal‬ ‭Logistic‬ ‭Regression:‬ ‭Ordinal‬ ‭logistic‬ ‭regression‬ ‭is‬ ‭used‬ ‭when‬‭the‬‭target‬
‭variable‬ ‭has‬ ‭ordered‬ ‭or‬ ‭ordinal‬ ‭categories.‬ ‭In‬ ‭this‬ ‭case,the‬ ‭classes‬ ‭have‬ ‭a‬ ‭natural‬
‭ordering‬ ‭or‬ ‭ranking.‬ ‭The‬ ‭logistic‬ ‭regression‬ ‭model‬ ‭is‬ ‭adapted‬ ‭to‬ ‭estimate‬ ‭the‬
‭cumulative probabilities of each class relative to the others.‬

‭4.‬ P‭ enalized‬ ‭Logistic‬ ‭Regression:‬ ‭Penalized‬ ‭logistic‬ ‭regression,‬ ‭also‬ ‭known‬ ‭as‬
‭regularized‬ ‭logistic‬ ‭regression,‬ ‭includes‬ ‭penalty‬ ‭terms‬ ‭in‬ ‭the‬ ‭model‬ ‭to‬ ‭prevent‬
‭overfitting‬ ‭and‬ ‭improve‬ ‭generalization.‬ ‭Common‬ ‭penalty‬ ‭terms‬ ‭include‬ ‭L1‬
‭regularization‬ ‭(Lasso‬ ‭regression)‬ ‭and‬ ‭L2‬ ‭regularization‬ ‭(Ridge‬ ‭regression).‬ ‭These‬
‭penalties encourage sparsity and shrinkage of the coefficients.‬

‭ hese‬‭are‬‭some‬‭of‬‭the‬‭common‬‭types‬‭of‬‭logistic‬‭regression.‬‭The‬‭choice‬‭of‬‭the‬‭type‬‭of‬‭logistic‬
T
‭regression‬ ‭depends‬ ‭on‬ ‭the‬ ‭nature‬ ‭of‬ ‭the‬ ‭data,‬ ‭the‬ ‭number‬ ‭of‬ ‭classes,‬ ‭the‬ ‭presence‬ ‭of‬
‭ordered‬ ‭categories,‬ ‭the‬ ‭need‬ ‭for‬ ‭regularization‬ ‭or‬ ‭penalization,‬ ‭and‬ ‭other‬ ‭specific‬
‭requirements of the classification problem at hand.‬

‭Assumptions of Logistic Regression‬

‭ ogistic‬‭regression‬‭makes‬‭several‬‭key‬‭assumptions‬‭to‬‭ensure‬‭the‬‭validity‬‭and‬‭reliability‬‭of‬‭its‬
L
‭results. Here are the main assumptions of logistic regression:‬

‭➢‬ B‭ inary‬ ‭or‬ ‭Ordinal‬ ‭Outcome:‬ ‭Logistic‬ ‭regression‬ ‭assumes‬ ‭that‬ ‭the‬ ‭dependent‬
‭variable‬‭(outcome)‬‭is‬‭binary‬‭(two‬‭categories)‬‭or‬‭ordinal‬‭(ordered‬‭categories).‬‭It‬‭is‬‭not‬
‭suitable for continuous outcome variables.‬
 ‭➢‬ L‭ inearity‬ ‭of‬ ‭Log-Odds:‬ ‭Logistic‬ ‭regression‬ ‭assumes‬ ‭that‬‭the‬‭relationship‬‭between‬
‭the‬‭independent‬‭variables‬‭and‬‭the‬‭log-odds‬‭of‬‭the‬‭outcome‬‭is‬‭linear.‬‭This‬‭means‬‭that‬
‭the‬‭effect‬‭of‬‭the‬‭independent‬‭variables‬‭is‬‭additive‬‭on‬‭the‬‭log-odds‬‭scale.‬‭If‬‭there‬‭are‬
‭non-linear‬ ‭relationships,‬ ‭appropriate‬ ‭transformations‬ ‭or‬ ‭higher-order‬ ‭terms‬ ‭may‬ ‭be‬
‭needed.‬

‭➢‬ I‭ndependence‬ ‭of‬ ‭Observations:‬ ‭Logistic‬ ‭regression‬ ‭assumes‬ ‭that‬‭observations‬‭in‬

‭the‬‭dataset‬‭are‬‭independent‬‭of‬‭each‬‭other.‬‭Each‬‭observation‬‭should‬‭be‬‭unrelated‬‭to‬
‭the‬ ‭others‬ ‭and‬ ‭not‬ ‭influenced‬ ‭by‬ ‭the‬ ‭presence‬ ‭or‬ ‭absence‬ ‭of‬ ‭other‬ ‭observations.‬
‭Independence is crucial for obtaining reliable coefficient estimates.‬

‭➢‬ A‭ bsence‬ ‭of‬ ‭Multicollinearity:‬ ‭Logistic‬ ‭regression‬ ‭assumes‬ ‭that‬‭there‬‭is‬‭little‬‭to‬‭no‬

‭multicollinearity‬‭among‬‭the‬‭independent‬‭variables.‬‭Multicollinearity‬‭occurs‬‭when‬‭there‬
‭are‬ ‭high‬ ‭correlations‬ ‭between‬ ‭independent‬ ‭variables,‬ ‭making‬ ‭it‬ ‭difficult‬ ‭to‬ ‭separate‬
‭their‬‭individual‬‭effects.‬‭Multicollinearity‬‭can‬‭lead‬‭to‬‭unstable‬‭coefficient‬‭estimates‬‭and‬
‭reduced interpretability.‬

‭➢‬ L‭ arge‬ ‭Sample‬ ‭Size:‬ ‭Logistic‬ ‭regression‬ ‭performs‬ ‭well‬ ‭with‬ ‭large‬ ‭sample‬ ‭sizes.‬ ‭A‬
‭rule‬ ‭of‬ ‭thumb‬ ‭is‬ ‭to‬ ‭have‬ ‭at‬ ‭least‬ ‭10-15‬ ‭outcome‬ ‭events‬ ‭(cases)‬ ‭per‬ ‭independent‬
‭variable‬ ‭to‬ ‭ensure‬ ‭stable‬ ‭estimates‬ ‭and‬ ‭reliable‬ ‭statistical‬ ‭inference.‬ ‭Insufficient‬
‭sample size may lead to overfitting or unreliable results.‬

I‭t's‬ ‭important‬‭to‬‭note‬‭that‬‭violating‬‭these‬‭assumptions‬‭may‬‭impact‬‭the‬‭validity‬‭and‬‭reliability‬
‭of‬ ‭the‬ ‭logistic‬ ‭regression‬ ‭results.‬ ‭Therefore,‬ ‭it‬ ‭is‬ ‭advisable‬ ‭to‬ ‭assess‬ ‭and‬ ‭address‬ ‭these‬
‭assumptions appropriately when applying logistic regression in practice.‬

‭Limitations of Logistic Regression‬

‭ ogistic‬ ‭regression,‬ ‭like‬ ‭any‬ ‭statistical‬ ‭model,‬ ‭has‬ ‭certain‬ ‭limitations‬ ‭that‬ ‭should‬ ‭be‬
L
‭considered‬ ‭when‬‭applying‬‭it‬‭to‬‭a‬‭classification‬‭problem.‬‭Here‬‭are‬‭some‬‭common‬‭limitations‬
‭of logistic regression:‬

‭➢‬ L‭ inear‬‭Relationship‬‭Assumption:‬‭Logistic‬‭regression‬‭assumes‬‭a‬‭linear‬‭relationship‬
‭between‬ ‭the‬ ‭independent‬ ‭variables‬ ‭and‬ ‭the‬ ‭log-odds‬ ‭of‬ ‭the‬ ‭outcome.‬ ‭If‬ ‭the‬
‭relationship‬ ‭is‬ ‭non-linear,‬ ‭logistic‬ ‭regression‬ ‭may‬ ‭not‬ ‭capture‬ ‭it‬ ‭accurately.‬ ‭In‬ ‭such‬
‭cases,‬ ‭more‬ ‭flexible‬ ‭models‬ ‭like‬ ‭decision‬ ‭trees‬ ‭or‬ ‭neural‬ ‭networks‬ ‭might‬ ‭be‬ ‭more‬
‭appropriate.‬

‭➢‬ S‭ ensitivity‬‭to‬‭Outliers:‬‭Logistic‬‭regression‬‭can‬‭be‬‭sensitive‬‭to‬‭outliers,‬‭especially‬‭if‬
‭they‬ ‭are‬ ‭influential‬ ‭in‬ ‭affecting‬ ‭the‬ ‭estimated‬ ‭coefficients.‬ ‭Outliers‬ ‭with‬ ‭extreme‬
‭values‬ ‭can‬ ‭disproportionately‬ ‭influence‬ ‭the‬ ‭model,‬ ‭leading‬ ‭to‬ ‭biased‬ ‭coefficient‬
‭estimates.‬ ‭Robust‬ ‭regression‬ ‭techniques‬ ‭or‬ ‭outlier‬ ‭detection‬ ‭methods‬ ‭can‬ ‭be‬
‭employed to mitigate this issue.‬
 ‭➢‬ I‭mbalanced‬ ‭Data:‬ ‭Logistic‬ ‭regression‬ ‭may‬ ‭not‬ ‭perform‬ ‭well‬ ‭with‬ ‭imbalanced‬
‭datasets,‬‭where‬‭one‬‭class‬‭is‬‭significantly‬‭more‬‭prevalent‬‭than‬‭the‬‭other.‬‭It‬‭tends‬‭to‬‭be‬
‭biased‬ ‭towards‬ ‭the‬ ‭majority‬‭class,‬‭leading‬‭to‬‭poor‬‭predictions‬‭for‬‭the‬‭minority‬‭class.‬
‭Techniques‬ ‭like‬ ‭oversampling,‬ ‭undersampling,‬ ‭or‬ ‭using‬‭weighted‬‭loss‬‭functions‬‭can‬
‭help address this issue.‬

‭➢‬ L‭ imited‬ ‭Decision‬ ‭Boundaries:‬ ‭Logistic‬ ‭regression‬ ‭produces‬ ‭straight-line‬ ‭decision‬

‭boundaries‬ ‭in‬ ‭the‬ ‭feature‬ ‭space.‬ ‭It‬ ‭may‬ ‭not‬ ‭capture‬ ‭complex‬ ‭shapes‬ ‭of‬ ‭decision‬
‭boundaries for more complicated datasets.‬

‭➢‬ P‭ robability‬‭Calibration‬‭Issues:‬‭Logistic‬‭regression‬‭provides‬‭predicted‬‭probabilities,‬
‭but‬ ‭these‬ ‭probabilities‬‭may‬‭not‬‭be‬‭well-calibrated,‬‭meaning‬‭they‬‭may‬‭not‬‭accurately‬
‭reflect the true probabilities of the outcome.‬

‭ espite‬ ‭these‬ ‭limitations,‬ ‭logistic‬ ‭regression‬ ‭remains‬ ‭a‬ ‭widely‬ ‭used‬ ‭and‬ ‭valuable‬ ‭tool‬ ‭for‬
D
‭binary‬ ‭and‬ ‭ordinal‬ ‭classification‬ ‭tasks.‬ ‭It‬ ‭provides‬ ‭interpretable‬ ‭results,‬ ‭is‬ ‭computationally‬
‭efficient,‬‭and‬‭works‬‭well‬‭in‬‭scenarios‬‭where‬‭the‬‭assumptions‬‭are‬‭reasonably‬‭met.‬‭However,‬
‭it's‬ ‭essential‬ ‭to‬ ‭carefully‬ ‭evaluate‬ ‭these‬ ‭limitations‬ ‭and‬ ‭consider‬ ‭alternative‬ ‭models‬ ‭when‬
‭necessary to ensure accurate and robust predictions.‬

‭Difference between Linear and Logistic Regression‬

‭Linear Regression‬ ‭Logistic Regression‬

‭Used to solve regression problems‬ ‭Used to solve classification problems‬

‭The response variables are continuous‬ ‭The response variable is categorical in‬
‭in nature‬ ‭nature‬

‭It helps estimate the dependent variable‬ ‭It helps to calculate the possibility of a‬
‭when there is a change in the‬ ‭particular event taking place‬
‭independent variable‬

‭It is a straight line‬ ‭It is an S-curve (S = Sigmoid)‬