k-nearest neighbor algorithm using Sklearn - Python

K-Nearest Neighbors (KNN) works by identifying the 'k' nearest data points called as neighbors to a given input and predicting its class or value based on the majority class or the average of its neighbors. In this article we will implement it using Python's Scikit-Learn library.

1. Generating and Visualizing the 2D Data

• We will import libraries like pandas, matplotlib, seaborn and scikit learn.
• The make_moons() function generates a 2D dataset that forms two interleaving half circles.
• This kind of data is non-linearly separable and perfect for showing how k-NN handles such cases.

from sklearn.datasets import make_moons
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd

# Create synthetic 2D data
X, y = make_moons(n_samples=300, noise=0.3, random_state=42)

# Create a DataFrame for plotting
df = pd.DataFrame(X, columns=["Feature 1", "Feature 2"])
df['Target'] = y

# Visualize the 2D data
plt.figure(figsize=(8, 6))
sns.scatterplot(data=df, x="Feature 1", y="Feature 2", hue="Target", palette="Set1")
plt.title("2D Classification Data (make_moons)")
plt.grid(True)
plt.show()

Output:

2. Train-Test Split and Normalization

• StandardScaler() standardizes the features by removing the mean and scaling to unit variance (z-score normalization).
• This is important for distance-based algorithms like k-NN as it ensures all features contribute equally to distance calculations.
• train_test_split() splits the data into 70% training and 30% testing.
• random_state=42 ensures reproducibility.
• stratify=y maintains the same class distribution in both training and test sets which is important for balanced evaluation.

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# Normalize the features
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Split into train and test
X_train, X_test, y_train, y_test = train_test_split(
    X_scaled, y, test_size=0.3, random_state=42, stratify=y
)

3. Fit the k-NN Model and Evaluate

• This creates a k-Nearest Neighbors (k-NN) classifier with k = 5 meaning it considers the 5 nearest neighbors for making predictions.
• fit(X_train, y_train) trains the model on the training data.
• predict(X_test) generates predictions for the test data.
• accuracy_score() compares the predicted labels (y_pred) with the true labels (y_test) and calculates the accuracy i.e the proportion of correct predictions.

from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score

# Train a k-NN classifier
knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train, y_train)

# Predict and evaluate
y_pred = knn.predict(X_test)
print(f"Test Accuracy (k=5): {accuracy_score(y_test, y_pred):.2f}")

Output:

Test Accuracy (k=5): 0.87

4. Cross-Validation to Choose Best k

Choosing the optimal k-value is critical before building the model for balancing the model's performance.

• A smaller k value makes the model sensitive to noise, leading to overfitting (complex models).
• A larger k value results in smoother boundaries, reducing model complexity but possibly underfitting.

This code performs model selection for the k value in the k-NN algorithm using 5-fold cross-validation:

• It tests values of k from 1 to 20.
• For each k, a new k-NN model is trained and validated using cross_val_score which automatically splits the dataset into 5 folds, trains on 4 and evaluates on 1, cycling through all folds.
• The mean accuracy of each fold is stored in cv_scores.
• A line plot shows how accuracy varies with k helping visualize the optimal choice.
• The best_k is the value of k that gives the highest mean cross-validated accuracy.

from sklearn.model_selection import cross_val_score
import numpy as np

# Range of k values to try
k_range = range(1, 21)
cv_scores = []

# Evaluate each k using 5-fold cross-validation
for k in k_range:
    knn = KNeighborsClassifier(n_neighbors=k)
    scores = cross_val_score(knn, X_scaled, y, cv=5, scoring='accuracy')
    cv_scores.append(scores.mean())

# Plot accuracy vs. k
plt.figure(figsize=(8, 5))
plt.plot(k_range, cv_scores, marker='o')
plt.title("k-NN Cross-Validation Accuracy vs k")
plt.xlabel("Number of Neighbors: k")
plt.ylabel("Cross-Validated Accuracy")
plt.grid(True)
plt.show()

# Best k
best_k = k_range[np.argmax(cv_scores)]
print(f"Best k from cross-validation: {best_k}")

Output:

Best k from cross-validation: 6

5. Training with Best k

• The model is trained on the training set with the optimized k (Here k = 6).
• The trained model then predicts labels for the unseen test set to evaluate its real-world performance.

# Train final model with best k
best_knn = KNeighborsClassifier(n_neighbors=best_k)
best_knn.fit(X_train, y_train)

# Predict on test data
y_pred = best_knn.predict(X_test)

6. Evaluate Using More Metrics

• Calculate the confusion matrix comparing true labels (y_test) with predictions (y_pred).
• Use ConfusionMatrixDisplay to visualize the confusion matrix with labeled classes

Print a classification report that includes:

• Precision: How many predicted positives are actually positive.
• Recall: How many actual positives were correctly predicted.
• F1-score: Harmonic mean of precision and recall.
• Support: Number of true instances per class.

from sklearn.metrics import confusion_matrix, classification_report, ConfusionMatrixDisplay

# Confusion Matrix
cm = confusion_matrix(y_test, y_pred)
disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=["Class 0", "Class 1"])
disp.plot(cmap="Blues")
plt.title(f"Confusion Matrix (k={best_k})")
plt.grid(False)
plt.show()

# Detailed classification report
print("Classification Report:")
print(classification_report(y_test, y_pred, target_names=["Class 0", "Class 1"]))

Output:

7. Visualize Decision Boundary with Best k

• Use the final trained model (best_knn) to predict labels for every point in the 2D mesh grid (xx, yy).
• Reshape the predictions (Z) to match the grid’s shape for plotting.
• Create a plot showing the decision boundary by coloring regions according to predicted classes using contourf.
• Overlay the original data points with different colors representing true classes using sns.scatterplot.

# Predict on mesh grid with best k
Z = best_knn.predict(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)

# Plot decision boundary
plt.figure(figsize=(8, 6))
plt.contourf(xx, yy, Z, cmap=plt.cm.coolwarm, alpha=0.3)
sns.scatterplot(x=X_scaled[:, 0], y=X_scaled[:, 1], hue=y, palette="Set1", edgecolor='k')
plt.title(f"Decision Boundary with Best k = {best_k}")
plt.xlabel("Feature 1 (scaled)")
plt.ylabel("Feature 2 (scaled)")
plt.grid(True)
plt.show()

Output:

We can see that our KNN model is working fine in classifying datapoints.