Simple linear regression
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
def generate_dataset(n_samples=100):
np.random.seed(42)
X = 2 * np.random.rand(n_samples, 1)
y = 3 * X + 4 + np.random.randn(n_samples, 1)
class SimpleLinearRegression:
def __init__(self):
self.slope = None
self.intercept = None
def fit(self, X, y):
n = len(X)
X_mean = np.mean(X)
y_mean = np.mean(y)
numerator = np.sum((X - X_mean) * (y - y_mean))
denominator = np.sum((X - X_mean) ** 2)
self.slope = numerator / denominator
self.intercept = y_mean - self.slope * X_mean
def predict(self, X):
return self.slope * X + self.intercept
if __name__ == "__main__":
X, y = generate_dataset()
dataset = pd.DataFrame({
"X": X.flatten(),
"y": y.flatten()
})
print("Dataset:")
print(dataset)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model = SimpleLinearRegression()
model.fit(X_train.flatten(), y_train.flatten())
y_pred = model.predict(X_test.flatten())
mse = mean_squared_error(y_test, y_pred)
print(f"Model Coefficients: Slope = {model.slope:.2f}, Intercept = {model.intercept:.2f}")
print(f"Mean Squared Error on Test Set: {mse:.2f}")
plt.scatter(X, y, color="blue", label="Actual Data")
plt.plot(X, model.predict(X.flatten()), color="red", label="Regression Line")
plt.xlabel('X')
plt.ylabel('y')
plt.title('Simple Linear Regression')
plt.legend()
plt.show()
Explanation:
Step-by-step breakdown:
• Step 1: Importing Libraries
o numpy: Used for generating synthetic data and performing numerical
operations.
o matplotlib.pyplot: Used for visualizing the data and the regression line.
o LinearRegression: This is the linear regression model from scikit-learn that will
be used to fit the data.
o train_test_split: Splits the dataset into training and testing sets.
o mean_squared_error: Used to evaluate the performance of the model by
computing the mean squared error.
• Step 2: Generating Synthetic Data
o We generate synthetic data using the equation y=3x+4y = 3x + 4y=3x+4 with
some added Gaussian noise. This helps simulate real-world data where the
relationship between variables is linear but with some randomness.
o X contains the feature values (input), and y contains the target values (output).
• Step 3: Splitting Data
o train_test_split() divides the data into training and testing sets. 80% of the data is
used for training, and 20% is used for testing.
• Step 4: Initializing the Model
o We create an instance of the LinearRegression class to initialize the linear
regression model.
• Step 5: Training the Model
o linear_reg.fit(X_train, y_train) fits the model to the training data, learning the
coefficients (slope and intercept) that best fit the linear relationship between X
and y.
• Step 6: Making Predictions
o y_pred = linear_reg.predict(X_test) predicts the target values (y_pred) for the test
data (X_test).
• Step 7: Evaluating the Model
o Mean Squared Error (MSE) is used to measure how well the model fits the data.
A lower MSE indicates a better fit.
o R-squared measures the proportion of the variance in the target variable that is
predictable from the features. A value closer to 1 indicates a good fit.
• Step 8: Visualizing the Results
o We use matplotlib to visualize the test data points (X_test vs. y_test) and the
regression line that represents the model's predictions.
How Linear Regression Works:
Linear regression attempts to model the relationship between a dependent variable y and an
independent variable X by fitting a straight line to the data. The relationship is described by the
equation:
y=β0+β1⋅Xy = \beta_0 + \beta_1 \cdot Xy=β0+β1⋅X
Where:
• y is the target variable (output),
• X is the input feature (independent variable),
• β0\beta_0β0 is the intercept (where the line crosses the y-axis),
• β1\beta_1β1 is the slope of the line.
The goal of the algorithm is to find the values of β0\beta_0β0 and β1\beta_1β1 that minimize the
difference between the predicted values ypredy_{pred}ypred and the actual values yyy (using a
loss function like Mean Squared Error).
KNN program
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.datasets import load_iris
class KNNClassifier:
def __init__(self, k=3):
self.k = k self.X_train = None
self.y_train = None
def fit(self, X, y):
self.X_train = X
self.y_train = y
def predict(self, X):
predictions = []
for x in X:
distances = np.linalg.norm(self.X_train - x, axis=1)
nearest_indices = distances.argsort()[:self.k]
nearest_labels = self.y_train[nearest_indices]
prediction = np.bincount(nearest_labels).argmax()
predictions.append(prediction)
return np.array(predictions)
iris = load_iris()
iris_df = pd.DataFrame(data=np.c_[iris['data'], iris['target']]
columns=iris['feature_names'] + ['target'])
X = iris.data
y = iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
knn = KNNClassifier(k=3)
knn.fit(X_train, y_train)
y_pred = knn.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
print(f"Accuracy: {accuracy * 100:.2f}%")
print("\nPredictions:")
for i, (true_label, pred_label) in enumerate(zip(y_test, y_pred)):
status = "Correct" if true_label == pred_label else "Incorrect"
print(f"Test Sample {i + 1}: True Label = {true_label}, Predicted = {pred_label}, {status}")
Explanation:
• Step 1: Importing Libraries
o numpy: Used for handling arrays and matrix operations.
o train_test_split: This function splits the dataset into training and testing subsets.
o KNeighborsClassifier: This is the KNN classifier from scikit-learn that will be
used for training and prediction.
o load_iris: A function to load the Iris dataset, which is a classic dataset used for
classification tasks.
o accuracy_score: This function calculates the accuracy of predictions by
comparing the predicted labels to the true labels.
• Step 2: Loading the Dataset
o We use load_iris() to load the Iris dataset, which is a simple classification
dataset where the goal is to predict the type of iris flower (Setosa, Versicolour, or
Virginica) based on four features (sepal length, sepal width, petal length, petal
width).
• Step 3: Splitting the Data
o train_test_split() splits the data into a training set and a test set (80% for training
and 20% for testing in this case). It shuffles the data and ensures that we
evaluate the model on unseen data.
• Step 4: Creating the KNN Classifier
o We create an instance of KNeighborsClassifier, setting the number of neighbors
n_neighbors=3. This means that the class of a new data point will be predicted
based on the majority class among the 3 nearest neighbors.
• Step 5: Training the Model
o knn.fit(X_train, y_train) trains the model using the training dataset (X_train as
input features and y_train as target labels).
• Step 6: Making Predictions
o knn.predict(X_test) makes predictions on the test data based on the trained
model. The X_test is the feature set for which we want to predict the class labels.
• Step 7: Evaluating the Model
o accuracy_score(y_test, y_pred) compares the predicted labels (y_pred) with the
true labels (y_test) and calculates the accuracy.
How KNN Works:
KNN is a simple yet powerful classification algorithm:
• For each test data point:
1. It calculates the distance (usually Euclidean distance) from that point to every
other point in the training set.
2. Then, it selects the k nearest points.
3. The majority class among the k nearest neighbors is taken as the prediction for
the test data point.
K-means program
import numpy as np
import matplotlib.pyplot as plt
def initialize_centroids(X, k):
return X[np.random.choice(X.shape[0], k, replace=False)]
def compute_distance(a, b):
return np.sqrt(np.sum((a - b) ** 2))
def assign_clusters(X, centroids):
clusters = []
for point in X:
distances = [compute_distance(point, centroid) for centroid in centroids]
cluster = np.argmin(distances) # Find the index of the nearest centroid
clusters.append(cluster)
return np.array(clusters)
def update_centroids(X, clusters, k):
new_centroids = np.zeros((k, X.shape[1]))
for i in range(k):
new_centroids[i] = np.mean(X[clusters == i], axis=0)
return new_centroids
def k_means(X, k, max_iters=100, tolerance=1e-4):
centroids = initialize_centroids(X, k)
for i in range(max_iters):
clusters = assign_clusters(X, centroids)
new_centroids = update_centroids(X, clusters, k)
if np.all(np.abs(new_centroids - centroids) < tolerance):
print(f"Converged at iteration {i}")
break
centroids = new_centroids
return centroids, clusters
from sklearn.datasets import make_blobs
X, _ = make_blobs(n_samples=300, centers=4, cluster_std=0.60, random_state=0)
k=4
centroids, clusters = k_means(X, k)
plt.scatter(X[:, 0], X[:, 1], c=clusters, cmap='viridis')
plt.scatter(centroids[:, 0], centroids[:, 1], s=300, c='red', marker='X')
plt.title("K-Means Clustering (from scratch)")
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.show()
Explanation:
Step-by-step breakdown:
• Step 1: Importing Libraries
o numpy: For handling numerical data and matrix operations.
o matplotlib.pyplot: For visualizing the data points and clusters.
o make_blobs: A function to generate synthetic data with a specified number of
clusters.
o KMeans: The KMeans algorithm from scikit-learn for clustering the data.
• Step 2: Generate Data
o make_blobs() generates synthetic data with 300 samples, 4 centers (clusters),
and random variance. This is used to simulate real-world clustering problems.
o X holds the generated data points, while y is the true label (not used in K-Means,
as it’s an unsupervised learning algorithm).
• Step 3: Visualizing Data Points
o The first plt.scatter() function plots the data points before applying the clustering
algorithm. They are all gray for now, and we use this plot to see how the data
looks before clustering.
• Step 4: K-Means Clustering
o KMeans(n_clusters=4) initializes the K-Means algorithm with 4 clusters (the
number of clusters we want to form).
o kmeans.fit(X) trains the K-Means model using the data X.
• Step 5: Get Cluster Centroids and Labels
o kmeans.cluster_centers_: This gives the coordinates of the centroids (the
centers) of each of the 4 clusters.
o kmeans.labels_: This gives the predicted labels (cluster assignments) for each
data point. Each data point is assigned a label that corresponds to the cluster it
belongs to.
• Step 6: Visualize Clusters
o The second plt.scatter() function visualizes the clusters by coloring each data
point according to its assigned cluster (using the labels). The centroids are
highlighted in red with a X marker.
o This plot helps us visually confirm the clusters formed by the K-Means algorithm.
How K-Means Works:
• Initialization:
o K-Means starts by randomly initializing k centroids (cluster centers).
• Iteration:
1. Assigning Labels: For each data point, it computes the distance from the point to each
centroid and assigns the point to the nearest centroid (i.e., the cluster).
2. Recalculating Centroids: After assigning labels to all points, it recalculates the centroids
by averaging the points within each cluster.
3. Repeat: Steps 1 and 2 are repeated iteratively until the centroids no longer change (i.e.,
convergence is reached).
Naïve Bayes Program
import numpy as np
from sklearn.datasets import make_classification
class NaiveBayes:
def __init__(self):
self.class_probs = {}
self.class_means = {}
self.class_vars = {}
def fit(self, X, y):
# Get unique class labels
classes = np.unique(y)
for c in classes:
self.class_probs[c] = np.mean(y == c)
for c in classes:
X_c = X[y == c]
self.class_means[c] = np.mean(X_c, axis=0)
self.class_vars[c] = np.var(X_c, axis=0)
def gaussian_pdf(self, x, mean, var):
return (1 / np.sqrt(2 * np.pi * var)) * np.exp(-(x - mean) ** 2 / (2 * var))
def predict(self, X):
predictions = []
for sample in X:
class_probs = {}
for c in self.class_probs:
prob = np.log(self.class_probs[c]) # Log prior P(class)
for i in range(len(sample)):
prob += np.log(self.gaussian_pdf(sample[i], self.class_means[c][i],
self.class_vars[c][i]))
class_probs[c] = prob
predicted_class = max(class_probs, key=class_probs.get)
predictions.append(predicted_class)
return np.array(predictions)
X, y = make_classification(n_samples=200, n_features=2, n_classes=2, random_state=42)
nb = NaiveBayes()
nb.fit(X, y)
predictions = nb.predict(X)
accuracy = np.mean(predictions == y)
print(f"Accuracy: {accuracy * 100:.2f}%")
Explanation:
Step-by-step breakdown:
• Step 1: Importing Libraries
o numpy: Used for numerical operations (though not directly used here, it is used
in the underlying data).
o train_test_split: This function splits the dataset into a training set and a test set.
o GaussianNB: This is the Naive Bayes classifier for continuous features
(assuming Gaussian/Normal distribution).
o load_iris: A function to load the Iris dataset, which contains flower data and their
corresponding species.
o accuracy_score: This function calculates the accuracy of predictions by
comparing the predicted labels (y_pred) with the true labels (y_test).
• Step 2: Loading the Dataset
o load_iris() loads the Iris dataset, which consists of 150 samples, each containing
4 features (sepal length, sepal width, petal length, petal width) and
corresponding target labels (y), which represent three species of Iris flowers.
• Step 3: Splitting the Data
o train_test_split() divides the data into a training set and a testing set (with 70%
training and 30% testing in this case). This helps in evaluating the model on
unseen data.
• Step 4: Initializing the Naive Bayes Model
o GaussianNB() initializes the Naive Bayes classifier that assumes the features are
normally distributed (Gaussian distribution).
• Step 5: Training the Model
o naive_bayes.fit(X_train, y_train) trains the Naive Bayes model using the training
data (X_train as input features and y_train as the target labels).
• Step 6: Making Predictions
o naive_bayes.predict(X_test) predicts the labels for the test data (X_test) based
on the trained model.
• Step 7: Evaluating the Model
o accuracy_score(y_test, y_pred) compares the predicted labels (y_pred) with the
true labels (y_test) and calculates the accuracy.
How Naive Bayes Works:
Naive Bayes is a probabilistic classifier based on Bayes' Theorem, with the "naive" assumption
that all features are independent given the class label. It works by computing the probability of
each class given the features and predicting the class with the highest probability.
• Bayes' Theorem: P(C∣X)=P(X∣C)P(C)P(X)P(C|X) = \frac{P(X|C)
P(C)}{P(X)}P(C∣X)=P(X)P(X∣C)P(C) Where:
o P(C∣X)P(C|X)P(C∣X) is the posterior probability of class CCC given the features
XXX.
o P(X∣C)P(X|C)P(X∣C) is the likelihood of the features XXX given the class CCC.
o P(C)P(C)P(C) is the prior probability of class CCC.
o P(X)P(X)P(X) is the probability of the features XXX.
In practice, Naive Bayes estimates the probability of each class by assuming that the features
are conditionally independent. For the Gaussian Naive Bayes (used here), it assumes the
features are normally distributed and uses the mean and variance of each feature to calculate
the likelihood.
