Interpretability and Explainability in AI Using Python: Decrypt AI Decision-Making Using Interpretability and Explainability with Python to Build Reliable Machine Learning Systems

CHAPTER 1

Interpreting Interpretable Machine Learning

Introduction

This chapter starts with an overview of Machine Learning and its various types. In the process, we will revisit supervised, unsupervised, and other model types, and explore deep learning along with the factors that add to its complexity. We will then explore Interpretability in Machine Learning, discussing its importance, expectations, and the key terms associated with it. Lastly, we will discuss the challenges and tradeoffs related to Interpretability.

Structure

In this chapter, we will discuss the following topics:

All Pervasive Machine Learning

Machine Learning Methods

Deep Learning

The Age of the Transformer

Machine Learning in the Real World

Interpretability and its Importance

Challenges and Tradeoffs

All Pervasive Machine Learning

Machine Learning influences our everyday lives in many subtle ways. The buzzing smartphones in our pockets, movies recommended on Netflix, and the discounts offered on our e-commerce purchases are all instances of all-pervasive machine learning. The smartphone that efficiently categorizes our pictures into albums and identifies people in them using vision models. Similarly, the movies recommended to you, which seem very much in line with your taste, are, in fact, a consequence of recommendation systems, which have learned from your previous viewing patterns. Reinforcement learning techniques identify the discounts offered to an end user, which most often results in a purchase to enhance the likelihood of future sales. There are many other instances of machine learning in our vicinity which often go unnoticed and unacknowledged and yet create a positive impact.

Historically, the term Machine Intelligence was coined by Alan Turing in the 1950s marking the start of early Artificial Intelligence. Over the decades, there were years of prolific AI activity while some years witnessed AI winters. In the late 1960s, there were two streams emerging in AI. One camp was working on rules-based systems (also called expert systems or symbolic), while the other group placed their bets on neural networks. The rules-based camp took the lead in the intervening years, primarily because the resources required for neural networks were scarce. Neural networks require relevant data, a strong algorithm, a narrow domain and a concrete goal (source: AI Superpowers - Kai Fu Lee). The availability of data was difficult, and relevant data in particular. A big element of the spurt in Machine Learning was evident when big data came into the fray and data storage became affordable.

There are multiple schools of thought on the definition of Artificial Intelligence and Machine Learning; artificial intelligence is considered all-encompassing and includes machine learning, deep learning among other areas. Recently, the terms Artificial Intelligence and Machine Learning are often used interchangeably. We will continue to use the two terms interchangeably in this book to stick to the prevailing trend.

Figure 1.1: Machine Learning builds models

In the past decade, machine learning methods have driven modern data analysis across Industry applications. When we interact with search engines and recommender systems, it is machine learning algorithms under the hood for content recommendation. Similarly, advertisers and financial institutions use machine learning to predict customer behavior, compliance, or risk. Machine learning is termed a general-purpose technology, as it can be used across Industries.

Machine Learning Methods

There are multiple types of machine learning algorithms which are generally divided into four types: Supervised Learning, Unsupervised Learning, Semi-Supervised Learning, and Reinforcement Learning. Deep learning fits into the larger artificial intelligence space and it leverages neural networks at its core. Machine learning algorithms include classification analysis, regression analysis, data clustering, association rule learning, and feature engineering for dimensionality reduction.

In the following machine learning-based classification model, the model is trained from historical data in phase 1, and the outcome is generated in phase 2 for the new test data.

Figure 1.2: Machine Learning process

Machine learning algorithms are divided based on the amount and type of supervision provided while training.

Figure 1.3: Machine Learning Types

Supervised Machine Learning

In Supervised Learning, the input includes the training data and the outcomes, also called labels. Supervised Learning can be further divided into Classification and Regression.

Classification

Identifying fraud is a classic example of Classification in supervised learning. Many transactions are provided along with their class (fraud or not fraud), and using this training dataset, the model learns to identify the patterns associated with fraudulent transactions.

Document Classification is another example, where the documents can be classified into multiple classes. For instance, an organization with many departments is regularly classifying the documents based on the content. In this instance, there are as many classes as departments.

Mathematically, it maps a function (f) from the input variables (X) to output variables (Y) as a target. The target serves as a label in the training phase and becomes an outcome prediction or category when using the model. Identifying or predicting the class for a data input is a classification problem. For instance, the transaction can be classified as fraud or not fraud.

The following is a list of common types of Classification problems:

Binary classification is a classification which has two class labels. In the preceding example, spam detection is a binary with the outcome being either spam or not spam.

Multiclass classification has more than two class labels. The data can be used to predict multiple different classes. Extending our example earlier, the transaction can be classified as fraudulent based on the type of pattern utilized, or a cybersecurity attack can be classified based on the type of attack mechanism utilized. Hence, a network attack can be classified as DDOS (Distributed Denial of Service), U2R (User to Root Attack), R2L (Root to Local Attack), and Probing Attack.

In machine learning, multi-label classification is an important consideration where an example is associated with several classes or labels. In the preceding document classification, a document can have multiple department labels when the content is relevant across departments. This is unlike traditional classification tasks where class labels are mutually exclusive.

Regression

Regression analysis of machine learning predicts a continuous output based on the value of one or more predictor variables. The difference between classification and regression is that classification is to predict distinct class labels, while in regression, the predicted output is a continuous quantity. Regression models are used in many applications across various fields. For instance, in manufacturing maintenance prediction, marketing, financial forecasting, trend analysis, disease prediction, and many more. There are multiple regression algorithms: linear, polynomial, lasso, ridge regression, and so on.

Note that some regression algorithms can be used for classification as well, and vice versa. For example, Logistic Regression is commonly used for classification, as it can output a value that corresponds to the probability of belonging to a given class (for example, 20% chance of being spam).

Here are some of the most commonly used supervised learning algorithms:

Linear Regression

Logistic Regression

Support Vector Machines (SVMs)

Decision Trees and Random Forests (Can be used for both Classification and Regression problems)

k-Nearest Neighbors (Can be used for both Classification and Regression problems)

Unsupervised Learning

As implied in the name, unsupervised learning is devoid of training, which translates to the training phase not having data with the labels, that is, unlabeled data. The model discerns patterns using the features in the data.

Figure 1.4: Supervised vs Unsupervised Learning

For instance, we have a dataset of purchase history on an e-commerce website and want to group customers with similar purchase habits. An algorithm can be used to detect groups with similar habits. At no point do you tell the algorithm which groups to consider and earmark the customers into groups; it finds those connections without your help. It might identify that 40% of the customers are males who love comic books and generally shop on weekends, while 20% are young sci-fi lovers who purchase multiple books periodically, and so on.

Clustering

Cluster analysis, also known as clustering, is an unsupervised machine learning technique that identifies and groups related data points within large datasets. Unlike supervised methods, clustering does not rely on specific outcomes or labels. Instead, it organizes objects into clusters based on their similarity. These clusters reveal interesting trends or patterns, such as consumer behavior groups as mentioned in the example of e-commerce website. Applications of clustering span diverse domains, including cybersecurity, e-commerce, mobile data processing, health analytics, and user modeling.

When employing a hierarchical clustering algorithm, it can further subdivide groups into smaller clusters. This subdivision can be advantageous for targeted communication, such as tailoring posts to specific groups. Consider an example where animals are distinctly separated from vehicles, horses are closely related to deer but distant from birds, and so on.

Clustering visualization algorithms serve as excellent illustrations of unsupervised learning techniques. These algorithms process intricate, unlabeled data and produce 2D or 3D representations that can be easily visualized. Their primary objective is to preserve the inherent structure within the data. By avoiding overlap of clusters in the input space during visualization, these algorithms provide insights into the organization of data and may even reveal unexpected patterns.

Another related task is dimensionality reduction, which aims to simplify data without significant information loss. One approach involves merging correlated features into a single feature. For instance, a car’s mileage often correlates with its age. In this case, a dimensionality reduction algorithm would combine these features to represent the overall wear and tear of the car—a process known as feature extraction.

Dimensionality Reduction and Feature Learning

In the realm of machine learning and data science, handling high-dimensional data poses challenges for researchers and developers. Dimensionality reduction, an unsupervised learning technique, plays a crucial role by enhancing human interpretability, reducing computational complexity, and mitigating overfitting and redundancy in models.

Two primary approaches for dimensionality reduction are feature selection and feature extraction. The key distinction lies in their outcomes:

Feature Selection: This method retains a subset of the original features, discarding irrelevant or redundant ones. Before feeding data to another machine learning algorithm (such as a supervised learning model), it is advisable to reduce dimensionality. Doing so results in faster processing, reduced disk and memory usage, and potentially improved performance.

Feature Extraction: Here, entirely new features are created, capturing essential information from the original data. Correlated features can be merged into a single feature. For instance, a car’s mileage often correlates with its age. In this case, a dimensionality reduction algorithm would combine these features to represent the overall wear and tear of the car—a process known as feature extraction.

Association Rule Learning

Association Rule Learning is a rule-based machine learning approach that uncovers interesting relationships expressed as IF-THEN statements within extensive datasets. For instance, consider the association: If a customer purchases a computer or laptop, they are likely to buy anti-virus software simultaneously.

Association rules are used in various industries for multiple solutions, including behavior analytics, customer purchase patterns, web user behaviors, cybersecurity, medical diagnosis, and others. Unlike sequence mining, association rule learning typically disregards the order of events within or across transactions. To assess the utility of association rules, we rely on parameters such as support and confidence.

Anomaly Detection

Lastly, anomaly detection is a critical unsupervised task. While it is generally classified as unsupervised, it can be a supervised model when the required labeled data of normal and anomalous instances is available. It involves identifying unusual instances, such as detecting fraudulent credit card transactions, catching manufacturing defects, or automatically removing outliers from a dataset before further analysis.

Semi-Supervised Learning

In the realm of machine learning, semi-supervised learning bridges the gap between supervised and unsupervised approaches. Unlike purely supervised learning, where every data point has an associated label, and unsupervised learning, which lacks labeled data, semi-supervised learning operates with a mix of both.

Consider photo-hosting services such as Google Photos as prime examples. When you upload family photos, the system automatically identifies recurring individuals (Person A in photos 1, 5, and 11, and Person B in photos 2, 5, and 7). This initial recognition is the unsupervised part, achieved through clustering. However, the system still requires your input to assign names to these individuals—just one label per person. This labeling process enhances photo search functionality.

Recently there has been an increase in self-supervised learning, which can be considered as a variation of this method. The key difference is that self-supervised learning does not require labeled data.

Reinforcement Learning

Reinforcement Learning (RL) operates in a distinct manner. In this context, the learning system—referred to as an agent—interacts with its environment, making observations, selecting actions, and receiving rewards (or penalties) in return. The agent’s ultimate goal is to discover the optimal strategy, known as a policy, that maximizes cumulative rewards over time. Essentially, a policy guides the agent’s actions based on its current situation. The policy also needs to account for the actions repeatedly exploiting one known high-reward action. The explore vs exploit tradeoff means that the agent needs to sometimes explore new actions to understand their rewards/penalties. This is especially important in a dynamic real-world application where the penalties/rewards for certain actions could have changed and periodic revisiting is required to keep learning.

For instance, robots often employ RL algorithms to learn how to walk. Additionally, DeepMind’s AlphaGo program serves as a remarkable RL example. In 2016, it gained fame by defeating world champion Lee Sedol in the ancient game of Go. AlphaGo honed its winning policy by analyzing millions of games and even playing against itself.

Deep Learning

Deep Learning gained momentum with growth in data and growth in scale. Machine Learning works well for a lot of scenarios, but as the data increases, counterintuitively the effective impact on the model performance does not increase after a threshold point. On the other hand, Deep Learning models benefit from the increase in data, especially the large neural networks. Deep learning involves layers of neurons in various architectures to process data and simulate human thinking. Some common use cases for deep learning are visual object recognition and understanding human speech. Deep learning uses neural networks composed of multiple layers for processing, such as input, hidden, and output layers, to learn from the data provided.

Deep learning involves a highly diverse set of concepts with deep neural network models of various architectures and modalities, including speech, vision, and text. There are architectures for training deep learning systems, including CNN, RNN, LSTM, Graph Neural Networks, and Transformers. CNNs are typically used for visual data, while more complex tasks in visual data, such as image segmentation, could benefit from Transformers. Similarly, LSTMs work well for time series forecasting since the pattern-matching principles for an image have parallels in time series data. There are, of course, transformer models which are being used in a wide range of activities.

Figure 1.5: Deep Learning

Applications in deep learning range across conversation AI, recommendation systems, computer vision, and so on. Conversational AI provides mechanisms to interact using natural language and now also utilize foundational models such as Large Language Models to augment the conversational experience. Conversational AI can also provide a voice-enabled interface to make the user interaction richer and seamless. Recommendation systems offer meaningful and relevant search results and services based on the observed patterns, user interests, and demographics, enriching end-user experiences. Computer vision applications gain knowledge from images and videos and are applicable in many scenarios where image data is highly prominent, such as healthcare.

Figure 1.6: Model Training process

(Source: A Survey of Deep Learning for Scientific Discovery (2020) Raghu et al)

Deep learning follows a similar machine learning process, but the complexity of the model implies more resource-intensive. Training tasks which originally took days and weeks can now utilize GPU accelerators and deep learning frameworks to speed up training. The deployment of models using GPU-accelerated inference aids in delivering high performance and optimal end-user experience across the cloud, embedded into devices or in edge systems such as cars. The past decade has witnessed many breakthroughs in AI, including DeepMind’s AlphaGo, intelligent assistants, and self-driving cars. Now the Transformer has created more ripples with its various models for natural language processing, vision, and other areas.

The Age of the Transformer

The RNNs and LSTMs were an improvement from the previous Deep Learning architectures, but they lacked the ability to handle large sequences and specifically the dependencies. For instance, a sentence to be translated from English to German is as follows:

Maya likes reading and music, and she is a voracious reader.

The pronoun she refers to Maya in the sentence. Similarly, the sentence implies Maya is a voracious reader. This is a toy example, but as sentences and paragraphs increase in size, prior context and dependency need to be preserved and ascertained to improve the performance of the model.

In natural language processing, the input is first tokenized, implying the text is tokenized and then converted into vectors understood by the neural network. The input can be a series of tokens with implied dependencies buried across long token sequences; these dependencies are important to capture so as to get a correct output. Recurrent Neural Networks (RNN) processes the input, token by token, and tracking long dependencies is not supported in its inherent design. Attention is a technique which introduces a memory concept by focusing on relevant elements of long sequences. This provides a mechanism to model long-term dependencies in the input to build on selected connections. Self-attention adds an additional element where it focuses on the same sequence, and it is a crucial design element in enhancing the performance for the transformer.

The RNN deficit with memory can be improved with attention techniques, but it still remained slow to train and had inherent optimization challenges with vanishing and exploding gradients. The transformer model addresses these shortcomings with a feedforward neural network which utilizes the self-attention mechanism. Additionally, it also has positional embeddings to track the location aspect of sequential data. The transformer model is based on the exceptionally popular paper, Attention is all you need, which has become the basis for many new technologies, including Generative AI.

Generative AI

Generative AI (GenAI), a subset of Artificial Intelligence learns from diverse content—text, images, and audio—to create novel artifacts. Unlike discriminative ML algorithms, which focus on decision boundaries, GenAI produces outputs with a wide spectrum of variety and complexity.

Figure 1.7: Generative AI in the scheme of things

A significant recent advancement in GenAI is OpenAI’s GPT-3 model. GPT-3 is classified as a Large Language Model, as it has 175 billion parameters. The other remarkable aspect of LLMs is that they are trained on vast volumes of data, leading to a model which is highly adept at many different tasks. Unlike traditional machine learning, where the models are trained for a single task and hence can do a single task, these models can do multiple tasks, such as completion, summarizing, and so on. ChatGPT had witnessed the fastest adoption curve, reaching 100 million users in only a few months. This versatile language model generates remarkably human-like text. These models excel at tasks such as auto-completing code, translating between programming languages and even converting natural language to code.

Additionally, domain-specific generative language models have emerged, addressing areas such as software engineering. The industry has embraced GenAI for software engineering practices, exemplified by tools such as GitHub CoPilot2. Starting from November 2022, after ChatGPT created