The Art of Machine Learning with .

NET
Table Of Contents

Introduction

This Path is Right For You!

Guide Overview

How to Use This Guide

About This Guide

Features

Benefits

Handling Data

Example Datasets

Load from CSV

Normalization

Standardization
Regression

Linear Regression

Ordinary Least Squares Regression

Stepwise Linear Regression

Principal Component Regression

Partial Least Squares Regression

Penalized Linear Regression

Ridge Regression

Least Absolute Shrinkage and Selection Operator

Elastic Net

Non-Linear Regression

Multivariate Adaptive Regression Splines

Support Vector Machine

k-Nearest Neighbor

Neural Network
Decision Trees for Regression

Classification and Regression Trees

Conditional Decision Trees

Model Trees

Rule System

Bagging

CART

Random Forest

Gradient Boosted Machine

Cubist

Classification

Linear Classification

Logistic Regression

Linear Discriminant Analysis

Partial Least Squares

Discriminant Analysis

Non-Linear Classification

Mixture Discriminant Analysis

Quadratic Discriminant Analysis

Regularized Discriminant Analysis

Neural Network
 Flexible Discriminant Analysis

Support Vector Machine

k-Nearest Neighbors

Naive Bayes

Non-Linear Classification with Decision Trees

Classification and Regression Trees

C4.5

PART

Random Forest

Gradient Boosted Machine

Boosted C5.0

Natural Algorithms

Overview

Particle Swarm Optimization

Ant System

Ant Colony System

Bees Algorithm

Bacterial Foraging Optimization Algorithm

This book is a quick reference that provides a large compendium of machine learning
algorithm descriptions. It is a handbook of practical computational recipes to be
referred when one is confronted with implementation of machine learning
problems.
A given algorithm is more than just a procedure or code listing, each approach is an
island of research. The meta-information that define the context of a technique is
just as important to understanding and application as abstract recipes and concrete
implementations.
A standardized algorithm description is adopted to provide a consistent presentation
of algorithms with a mixture of softer narrative descriptions, programmatic
descriptions both abstract and concrete, and most importantly useful sources for
finding out more information about the technique.
The standardized algorithm description template covers the following subjects:
Name: The algorithm name defines the canonical name used to refer to the
technique, in addition to common aliases, abbreviations, and acronyms. The name is
used as the heading of an algorithm description.
Taxonomy: The algorithm taxonomy defines where a technique fits into the field, of
machine learning. The taxonomy also provides a context for determining the
relationships between algorithms.
Inspiration: (where appropriate) The inspiration describes the specific system or
process that provoked the inception of the algorithm.
Metaphor: (where appropriate) The metaphor is a description of the technique in
the context of the inspiring system or a different suitable system.
Strategy: The strategy is an abstract description of the machine learning model.
Procedure: The algorithmic procedure summarizes the specifics of realizing a
strategy as a systemized and parameterized computation. It outlines how the
algorithm is organized in terms of the computation, data structures, and
representations.
Heuristics: The heuristics section describes the common sense, best practice, and
demonstrated rules for applying and configuring a parameterized algorithm.
Code Listing: The code listing description provides a minimal but functional version
of the technique implemented with a programming language.
References: The references section includes a listing of both primary sources of
information about the technique as well as useful introductory sources for novices to
gain a deeper understanding of the theory and application of the technique.
Sample:

6.3 Ant System

6.3.1 Taxonomy

The Ant System algorithm is an example of an Ant Colony Optimization method from the
field of Swarm Intelligence, Meta-heuristics and Computational Intelligence. Ant System was
originally the term used to refer to a range of Ant based algorithms, where the specific
algorithm implementation was referred to as Ant Cycle. The so-called Ant Cycle algorithm is
now canonically referred to as Ant System. The Ant System algorithm is the baseline Ant
Colony Optimization method for popular extensions such as Elite Ant System, Rank-based
Ant System, Max-Min Ant System, and Ant Colony System.

6.3.2 Inspiration

The Ant system algorithm is inspired by the foraging behaviour of ants, specifically the
pheromone communication between ants regarding a good path between the colony and a
food source in an environment. This mechanism is called stigmergy.

6.3.3 Metaphor

Ants initially wander randomly around their environment. Once food is located an ant will
begin laying down pheromone in the environment. Numerous trips between the food and
the colony are performed and if the same route is followed that leads to food then
additional pheromone is laid down. Pheromone decays in the environment, so that older
paths are less likely to be followed. Other ants may discover the same path to the food and
in turn may follow it and also lay down pheromone. A positive feedback process routes more
and more ants to productive paths that are in turn further refined through use.

6.3.4 Strategy

The objective of the strategy is to exploit historic and heuristic information to construct
candidate solutions and fold the information learned from constructing solutions into the
history. Solutions are constructed one discrete piece at a time in a probabilistic step-wise
manner. The probability of selecting a component is determined by the heuristic
contribution of the component to the overall cost of the solution and the quality of solutions
from which the component has historically known to have been included. History is updated
proportional to the quality of candidate solutions and is uniformly decreased ensuring the
most recent and useful information is retained.

6.3.5 Procedure

Algorithm 6.3.1 provides a pseudo-code listing of the main Ant System algorithm for
minimizing a cost function. The pheromone update process is described by a single equation
 that combines the contributions of all candidate solutions with a decay coefficient to
determine the new pheromone value, as follows:

τi,j ← (1 − ρ) × τi,j +

m X k=1

∆ki,j (6.2)

where τi,j represents the pheromone for the component (graph edge) (i,j), ρ is the decay
factor, m is the number of ants, and Pm k=1 ∆ki,j is the sum of 1 S cost (maximizing solution
cost) for those solutions that include component i,j. The Pseudocode listing shows this
equation as an equivalent as a two step process of decay followed by update for simplicity.
The probabilistic step-wise construction of solution makes use of both history (pheromone)
and problem-specific heuristic information to incrementally construction a solution piece-
by-piece. Each component can only be selected if it has not already been chosen (for most
combinatorial problems), and for those components that can be selected from (given the
current component i), their probability for selection is defined as:

Pi,j ←

τα i,j × ηβ i,j

Pc k=1 τα i,k × ηβ i,k

(6.3)

where ηi,j is the maximizing contribution to the overall score of selecting the component
(such as 1.0 distancei,j for the Traveling Salesman Problem), α is the heuristic coefficient, τi,j
is the pheromone value for the component, β is the history coefficient, and c is the set of
usable components.

6.3.6 Heuristics

The Ant Systems algorithm was designed for use with combinatorial problems such as the
TSP, knapsack problem, quadratic assignment problems, graph coloring problems and many
others.

The history coefficient (α) controls the amount of contribution history plays in a components
probability of selection and is commonly set to 1.0.
Algorithm 6.3.1: Pseudocode for Ant System.

The heuristic coefficient (β) controls the amount of contribution problem-specific heuristic
information plays in a components probability of selection and is commonly between 2 and
5, such as 2.5.

The decay factor (ρ) controls the rate at which historic information is lost and is commonly
set to 0.5.

The total number of ants (m) is commonly set to the number of components in the problem,
such as the number of cities in the TSP.

6.3.7 Code Listing

<implementation code in .net goes here>

Listing 6.2 provides an example of the Ant System algorithm implemented in the C#
Programming Language. The algorithm is applied to the Berlin52 instance of the Traveling
Salesman Problem (TSP), taken from the TSPLIB. The problem seeks a permutation of the
 order to visit cities (called a tour) that minimized the total distance travelled. The optimal
tour distance for Berlin52 instance is 7542 units. Some extensions to the algorithm
implementation for speed improvements may consider pre-calculating a distance matrix for
all the cities in the problem, and pre-computing a probability matrix for choices during the
probabilistic step-wise construction of tours.

6.3.8 References
<References goes here>