15/9/2019 Image recognition with C# and Emgu libraries - CodeProject

14,297,708 members 263 ☰

articles Q&A forums stuff lounge ? Search for articles, questions,

Follow

Image recognition with C# and Emgu libraries

Emiliano Musso Rate me! 4.93 (39 votes)

16 May 2017 CPOL

In the following we'll see how to realize an image recognition program, using C# and EmGu, a .NET wrapper for the Intel OpenCV
image-processing library. At the end of the article, the reader will be able to develop a simple application which will search into a list
of images for the one containing a

Is your email address OK? You are signed up for our newsletters but your email address is either unconfirmed, or
has not been reconfirmed in a long time. Please click here to have a confirmation email sent so we can confirm
your email address and start sending you newsletters again. Alternatively, you can update your subscriptions.

Abstract
In the following we'll see how to realize an image recognition program, using C# and EmGu, a .NET wrapper for the Intel OpenCV
image-processing library.

At the end of the article, the reader will be able to develop a simple application which will search into a list of images for the one
containing a smaller portion of the original one, graphically showing the points of intersection between the two.

Download and install EmGu

EmGu libraries can be downloaded at https://sourceforge.net/projects/emgucv/files/. The latest version of the package is located at
https://sourceforge.net/projects/emgucv/files/latest/download"source=files, and comes in the form of an exe installer. For a pretty
complete reference of the library, the reader is invited to visithttp://www.emgu.com/wiki/index.php/Main_Page , which is the official
Wiki of EmGu. Here will be considered only the indispensable aspects and functions to reach the declared goal of matching two
images to find the smaller one in a list of possible candidates.

https://www.codeproject.com/Articles/1187512/Image-recognition-with-Csharp-and-Emgu-libraries 1/14
15/9/2019 Image recognition with C# and Emgu libraries - CodeProject

Once downloaded, the package can be installed in a folder of choice. Upon completion, the install folder will appear like the
following, with the \bin folder containing the core components of the package, i.e. the DLLs that will be referenced in a project.

Create a EmGu referenced project

Let's create a new project (CTRL+N) in Visual Studio. For the present example, choose "Visual C#" and "Winforms application" from
the Templates menu. Now, from Visual Studio top menu go to "Project", "Add References". Select "Browse", and - going to the
EmGu path created by the previously launched installer - select Emgu.CV.World.dll and Emgu.CV.UI.dll (fig.2). Confirming with OK
button, Visual Studio will proceed to add those references to the project, as shown in fig.3.

https://www.codeproject.com/Articles/1187512/Image-recognition-with-Csharp-and-Emgu-libraries 2/14
15/9/2019 Image recognition with C# and Emgu libraries - CodeProject

Important: to make the compiled solutions work, the x86 and x64 folders from \emgucv-windesktop 3.1.0.2504\binmust be copied
into the compiled executable path. Otherwise, the programs which use Emgu won't be able to referencecvextern.dll, msvcp140.dll,
opencv_ffmpeg310_64.dll, vcruntime140.dll, raising an exception and preventing them to run properly.

SURF detection
As said in the Wikipedia page for Speeded up robust features (https://en.wikipedia.org/wiki/Speeded_up_robust_features ), "[...] is
a patented local feature detector and descriptor. It can be used for tasks such as object recognition, image registration, classification
or 3D reconstruction. It is partly inspired by the scale-invariant feature transform (SIFT) descriptor. The standard version of SURF is
several times faster than SIFT and claimed by its authors to be more robust against different image transformations than SIFT. To
detect interest points, SURF uses an integer approximation of the determinant of Hessian blob detector , which can be computed
with 3 integer operations using a precomputed integral image. Its feature descriptor is based on the sum of the Haar wavelet
response around the point of interest. These can also be computed with the aid of the integral image. SURF descriptors have been
used to locate and recognize objects, people or faces, to reconstruct 3D scenes, to track objects and to extract points of interest."

FeatureMatching sample
In the \emgucv-windesktop 3.1.0.2504\Emgu.CV.Example\FeatureMatching folder, there is a sample which was written to show
image recognition capabilities described as above, so it's a great point to start further implementations. Let's start from the
FeatureMatching.cs file: few lines of code are present into the static method Main(). To the present means, those of interest are the
following:

Hide   Copy Code

https://www.codeproject.com/Articles/1187512/Image-recognition-with-Csharp-and-Emgu-libraries 3/14
15/9/2019 Image recognition with C# and Emgu libraries - CodeProject

long matchTime;
using(Mat modelImage = CvInvoke.Imread("box.png", ImreadModes.Grayscale))
using (Mat observedImage = CvInvoke.Imread("box_in_scene.png", ImreadModes.Grayscale))
{
Mat result = DrawMatches.Draw(modelImage, observedImage, out matchTime);
ImageViewer.Show(result, String.Format("Matched in {0} milliseconds", matchTime));
}

The Mat class can be roughly paired to an Image, with a set of method and properties serving for the most diverse purposes. As
previously said, here we'll be limited to the essential ones. A complete reference for the Mat class can be found at
http://www.emgu.com/wiki/files/3.1.0/document/html/2ec33afb-1d2b-cac1-ea60-0b4775e4574c.htm

Imread method will read an image from a given path and load type (grayscale, color, and so on), returning a Mat object for further
uses. We have two Mat variables from two different sources, i.e. "box.png" and "box_in_scene.png" (both included into the sample
folder).

As we can see at the page which discuss this example (http://www.emgu.com/wiki/index.php/SURF_feature_detector_in_CSharp ),

the final result of the elaboration (namely and obviously, showing the result of the DrawMatches.Draw method) will be like this:

So it's clear what is done here: two images will be loaded, with one of them being a subpart of the other one, and the two get
matched to show their common points, determining similarities/inclusiveness. That match is executed by theDraw() method of
the static class DrawMatches, that can be found into the sample folder or at the SURF detector URL listed above. The source code
is as follows:

Hide   Shrink   Copy Code

//----------------------------------------------------------------------------
// Copyright (C) 2004-2016 by EMGU Corporation. All rights reserved.
//----------------------------------------------------------------------------
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Drawing;
using Emgu.CV;
using Emgu.CV.CvEnum;
using Emgu.CV.Features2D;
using Emgu.CV.Flann;
using Emgu.CV.Structure;
using Emgu.CV.Util;

namespace FeatureMatchingExample
{
public static class DrawMatches
{
public static void FindMatch(Mat modelImage, Mat observedImage, out long matchTime, out
VectorOfKeyPoint modelKeyPoints, out VectorOfKeyPoint observedKeyPoints, VectorOfVectorOfDMatch
matches, out Mat mask, out Mat homography)
{
int k = 2;
https://www.codeproject.com/Articles/1187512/Image-recognition-with-Csharp-and-Emgu-libraries 4/14
15/9/2019 Image recognition with C# and Emgu libraries - CodeProject
double uniquenessThreshold = 0.80;

Stopwatch watch;
homography = null;

modelKeyPoints = new VectorOfKeyPoint();

observedKeyPoints = new VectorOfKeyPoint();

using (UMat uModelImage = modelImage.GetUMat(AccessType.Read))

using (UMat uObservedImage = observedImage.GetUMat(AccessType.Read))
{
KAZE featureDetector = new KAZE();

//extract features from the object image

Mat modelDescriptors = new Mat();
featureDetector.DetectAndCompute(uModelImage, null, modelKeyPoints, modelDescriptors, false);

watch = Stopwatch.StartNew();

// extract features from the observed image

Mat observedDescriptors = new Mat();
featureDetector.DetectAndCompute(uObservedImage, null, observedKeyPoints, observedDescriptors,
false);

// Bruteforce, slower but more accurate

// You can use KDTree for faster matching with slight loss in accuracy
using (Emgu.CV.Flann.LinearIndexParams ip = new Emgu.CV.Flann.LinearIndexParams())
using (Emgu.CV.Flann.SearchParams sp = new SearchParams())
using (DescriptorMatcher matcher = new FlannBasedMatcher(ip, sp))
{
matcher.Add(modelDescriptors);

matcher.KnnMatch(observedDescriptors, matches, k, null);

mask = new Mat(matches.Size, 1, DepthType.Cv8U, 1);
mask.SetTo(new MCvScalar(255));
Features2DToolbox.VoteForUniqueness(matches, uniquenessThreshold, mask);

int nonZeroCount = CvInvoke.CountNonZero(mask);

if (nonZeroCount >= 4)
{
nonZeroCount = Features2DToolbox.VoteForSizeAndOrientation(modelKeyPoints, observedKeyPoints,
matches, mask, 1.5, 20);
if (nonZeroCount >= 4)
homography = Features2DToolbox.GetHomographyMatrixFromMatchedFeatures(modelKeyPoints,
observedKeyPoints, matches, mask, 2);
}
}
watch.Stop();

}
matchTime = watch.ElapsedMilliseconds;
}

/// <summary>
/// Draw the model image and observed image, the matched features and homography projection.
/// </summary>
/// <param name="modelImage">The model image</param>
/// <param name="observedImage">The observed image</param>
/// <param name="matchTime">The output total time for computing the homography matrix.</param>
/// <returns>The model image and observed image, the matched features and homography projection.
</returns>
public static Mat Draw(Mat modelImage, Mat observedImage, out long matchTime)
{
Mat homography;
VectorOfKeyPoint modelKeyPoints;
VectorOfKeyPoint observedKeyPoints;
using (VectorOfVectorOfDMatch matches = new VectorOfVectorOfDMatch())
{
Mat mask;

https://www.codeproject.com/Articles/1187512/Image-recognition-with-Csharp-and-Emgu-libraries 5/14
15/9/2019 Image recognition with C# and Emgu libraries - CodeProject
FindMatch(modelImage, observedImage, out matchTime, out modelKeyPoints, out observedKeyPoints,
matches,
out mask, out homography);

//Draw the matched keypoints

Mat result = new Mat();
Features2DToolbox.DrawMatches(modelImage, modelKeyPoints, observedImage, observedKeyPoints,
matches, result, new MCvScalar(255, 255, 255), new MCvScalar(255, 255, 255), mask);

#region draw the projected region on the image

if (homography != null)
{
//draw a rectangle along the projected model
Rectangle rect = new Rectangle(Point.Empty, modelImage.Size);
PointF[] pts = new PointF[]
{
new PointF(rect.Left, rect.Bottom),
new PointF(rect.Right, rect.Bottom),
new PointF(rect.Right, rect.Top),
new PointF(rect.Left, rect.Top)
};
pts = CvInvoke.PerspectiveTransform(pts, homography);

#if NETFX_CORE
Point[] points = Extensions.ConvertAll<PointF, Point>(pts, Point.Round);
#else
Point[] points = Array.ConvertAll<PointF, Point>(pts, Point.Round);
#endif
using (VectorOfPoint vp = new VectorOfPoint(points))
{
CvInvoke.Polylines(result, vp, true, new MCvScalar(255, 0, 0, 255), 5);
}
}
#endregion

return result;

}
}
}
}

The core method is FindMatch(), used to discover similarities amongst the images, and populating arrays which will be used by
the Draw() method to show them graphically. We'll not analyze in detail the above code here, but we will modify it in the
following, to make it able not only to spot differences, but to tell the user how much of them are present, allowing the application to
tell - starting from a list of images - what image suits best the detailed one.

Developing the solution

In order to analyze an arbitrary number of images, to detect the one which contains our model image, it is necessary to have a score
of similarities for each analyzed image. The idea is simple: given a certain folder(s) containing images, and a small image to be
localized, the program must process each one of them, determining a value which represents how many key points from the model
image have been spotted into the cycled image. At the end of the loop, the image(s) that will have returned the higher values are
more likely to be the results we expect (or, in other word, higher the score, higher are the chances our model is contained into those
images).

Determining a similarity score

FindMatch() method could be customized to expose a count of matches found between images.
Its signature can be modified in:

Hide   Copy Code

https://www.codeproject.com/Articles/1187512/Image-recognition-with-Csharp-and-Emgu-libraries 6/14
15/9/2019 Image recognition with C# and Emgu libraries - CodeProject

public static void FindMatch(Mat modelImage, Mat observedImage, out long matchTime, out
VectorOfKeyPoint modelKeyPoints, out VectorOfKeyPoint observedKeyPoints, VectorOfVectorOfDMatch
matches, out Mat mask, out Mat homography, out long score)

Adding the score variable to serve as an output parameter. Its value can be calculated after the VoteForUniqueness()call,
looping between VectorOfVectorOfDMatch matches to increase score value every time we encounter a match:

Hide   Copy Code

score = 0;
for (int i = 0; i < matches.Size; i++)
{
if (mask.GetData(i)[0] == 0) continue;
foreach (var e in matches[i].ToArray())
++score;
}

FindMatch() method thus become as follows:

Hide   Shrink   Copy Code

public static void FindMatch(Mat modelImage, Mat observedImage, out long matchTime, out
VectorOfKeyPoint modelKeyPoints, out VectorOfKeyPoint observedKeyPoints, VectorOfVectorOfDMatch
matches, out Mat mask, out Mat homography, out long score)
{
int k = 2;
double uniquenessThreshold = 0.80;

Stopwatch watch;
homography = null;

modelKeyPoints = new VectorOfKeyPoint();

observedKeyPoints = new VectorOfKeyPoint();

using (UMat uModelImage = modelImage.GetUMat(AccessType.Read))

using (UMat uObservedImage = observedImage.GetUMat(AccessType.Read))
{
KAZE featureDetector = new KAZE();

Mat modelDescriptors = new Mat();

featureDetector.DetectAndCompute(uModelImage, null, modelKeyPoints, modelDescriptors, false);

watch = Stopwatch.StartNew();

Mat observedDescriptors = new Mat();

featureDetector.DetectAndCompute(uObservedImage, null, observedKeyPoints, observedDescriptors,
false);

// KdTree for faster results / less accuracy

using (var ip = new Emgu.CV.Flann.KdTreeIndexParams())
using (var sp = new SearchParams())
using (DescriptorMatcher matcher = new FlannBasedMatcher(ip, sp))
{
matcher.Add(modelDescriptors);

matcher.KnnMatch(observedDescriptors, matches, k, null);

mask = new Mat(matches.Size, 1, DepthType.Cv8U, 1);
mask.SetTo(new MCvScalar(255));
Features2DToolbox.VoteForUniqueness(matches, uniquenessThreshold, mask);

// Calculate score based on matches size

// ---------------------------------------------->
score = 0;
for (int i = 0; i < matches.Size; i++)
{
if (mask.GetData(i)[0] == 0) continue;
foreach (var e in matches[i].ToArray())
++score;
}
https://www.codeproject.com/Articles/1187512/Image-recognition-with-Csharp-and-Emgu-libraries 7/14
15/9/2019 Image recognition with C# and Emgu libraries - CodeProject
// <----------------------------------------------

int nonZeroCount = CvInvoke.CountNonZero(mask);

if (nonZeroCount >= 4)
{
nonZeroCount = Features2DToolbox.VoteForSizeAndOrientation(modelKeyPoints, observedKeyPoints,
matches, mask, 1.5, 20);
if (nonZeroCount >= 4)
homography = Features2DToolbox.GetHomographyMatrixFromMatchedFeatures(modelKeyPoints,
observedKeyPoints, matches, mask, 2);
}
}
watch.Stop();

}
matchTime = watch.ElapsedMilliseconds;
}

Note: for the sake of speed upon accuracy, in the above code LinearIndexParams has been substituted by KdTreeIndexParams.

Obviously, every method which calls FindMatch() must include the new score variable. So, Draw() method must be
customized accordingly. In short, every time we execute Draw() method we must pass a long variable in which the score will be
calculated, when it get passed to FindMatch() method. Please refer to the source code at the end of the article for further
references.

Image and score viewer

To check rapidly the score value for any image, a customized image viewer form can be created.

Declare a new form, naming it emImageViewer, adding to it a PictureBox (imgBox in the example), with Dock = Fill. The source code
will be the following:

Hide   Copy Code

using Emgu.CV;
using System.Drawing;
using System.Windows.Forms;

namespace ImgRecognitionEmGu
{
public partial class emImageViewer : Form
{
public emImageViewer(IImage image, long score = 0)
{
InitializeComponent();

this.Text = "Score: " + score.ToString();

if (image != null)
{
imgBox.Image = image.Bitmap;

Size size = image.Size;

size.Width += 12;
size.Height += 42;
if (!Size.Equals(size)) Size = size;
}
}
}
}

The constructor requires two parameters: the first, image, of type IImage, constitutes the image (casted Mat object) to be visualized
inside imgBox. The second parameter, score, is the value calculated by the methods seen above. So, in we compare two images with
the following code

Hide   Copy Code

https://www.codeproject.com/Articles/1187512/Image-recognition-with-Csharp-and-Emgu-libraries 8/14
15/9/2019 Image recognition with C# and Emgu libraries - CodeProject

long score;
long matchTime;

using (Mat modelImage = CvInvoke.Imread("box.png", ImreadModes.Grayscale))

using (Mat observedImage = CvInvoke.Imread("box_in_scene.png", ImreadModes.Grayscale))
{
Mat result = DrawMatches.Draw(modelImage, observedImage, out matchTime, out score);

var iv = new emImageViewer(result, score);

iv.Show();
}

The result will be

Which shows a score of 144. Obviously, changing the model image will result in different score values. If, for example, we wish to
compare the scene image with itself, we will obtain

Process a list of images

At this point, calculating the score for a list of images becomes trivial.

Let's assume we have a given directory, images, with its subfolder structure, each of which may contain images. What is needed now
is to traverse the entire directory tree and execute a comparison for every file found. The calculated score can be saved in an apt
memory structure, to be listed at the end of the process.

We start by defining that structure/class

Hide   Copy Code

class WeightedImages
{
public string ImagePath { get; set; } = "";

https://www.codeproject.com/Articles/1187512/Image-recognition-with-Csharp-and-Emgu-libraries 9/14
15/9/2019 Image recognition with C# and Emgu libraries - CodeProject
public long Score { get; set; } = 0;
}

WeightedImages will be used to memorize the path of a complete image, with its calculated score. The class will be used in
conjunction with a List<>:

Hide   Copy Code

List<WeightedImages> imgList = new List<WeightedImages>();

A simple method, ProcessFolder(), will parse every single file of each directory in the structure, recursively. For each of them,
it will call ProcessImage(), which is the heart of the process.

Hide   Shrink   Copy Code

private void ProcessFolder(string mainFolder, string detailImage)

{
foreach (var file in System.IO.Directory.GetFiles(mainFolder))
ProcessImage(file, detailImage);

foreach (var dir in System.IO.Directory.GetDirectories(mainFolder))

ProcessFolder(dir, detailImage);
}

private void ProcessImage(string completeImage, string detailImage)

{
if (completeImage == detailImage) return;

try
{
long score;
long matchTime;

using (Mat modelImage = CvInvoke.Imread(detailImage, ImreadModes.Color))

using (Mat observedImage = CvInvoke.Imread(completeImage, ImreadModes.Color))
{
Mat homography;
VectorOfKeyPoint modelKeyPoints;
VectorOfKeyPoint observedKeyPoints;

using (var matches = new VectorOfVectorOfDMatch())

{
Mat mask;
DrawMatches.FindMatch(modelImage, observedImage, out matchTime, out modelKeyPoints, out
observedKeyPoints, matches,
out mask, out homography, out score);
}

imgList.Add(new WeightedImages() { ImagePath = completeImage, Score = score });

}
} catch { }
}

ProcessImage() doesn't need to draw the resultant object, it can simply stop when the score is calculated, and that value (with
the file name) is added to imgList.

A button will start the process.

At the end of it, the imgList is ordered by descendant score value, and used as DataSource for a DataGridView, to show the results of
the process itself.

https://www.codeproject.com/Articles/1187512/Image-recognition-with-Csharp-and-Emgu-libraries 10/14
15/9/2019 Image recognition with C# and Emgu libraries - CodeProject

In this example, the searched image was this:

As it can be seen, the sample directories has been traversed, and for each found image, a score was calculated, comparing the
searched image to the complete one. Please note, since those image are not grayscaled, but coming in colours, that load type was
set accordingly ( ImreadModes.Color).

A second button allow to open the image represented by the current selected grid row, using the emImageViewer created above:

Hide   Copy Code

private void btnShow_Click(object sender, System.EventArgs e)

{
string imgPath = resultGrid.CurrentRow.Cells[0].Value.ToString();

long score;
long matchTime;

using (Mat modelImage = CvInvoke.Imread(_detailedImage, ImreadModes.Color))

using (Mat observedImage = CvInvoke.Imread(imgPath, ImreadModes.Color))
{
var result = DrawMatches.Draw(modelImage, observedImage, out matchTime, out score);
var iv = new emImageViewer(result, score);
iv.Show();
}
}

Opening all the images in the list results in the following:

https://www.codeproject.com/Articles/1187512/Image-recognition-with-Csharp-and-Emgu-libraries 11/14
15/9/2019 Image recognition with C# and Emgu libraries - CodeProject

And we can see how the correct image has been spotted by the higher score of the list.

Further implementations
The presented code must be considered valid in learning contexts only. A further implementation, to manage - for example - many
thousands images in a single run, can be made by making the program asynchronous / multi-threaded, in order to better split
resources and diminish elaboration time.

Source code
The source code used in the article can be downloaded at https://code.msdn.microsoft.com/Image-recognition-with-C-b13b2864

References
Emgu Wiki
Speeded up robust features

License
This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

https://www.codeproject.com/Articles/1187512/Image-recognition-with-Csharp-and-Emgu-libraries 12/14
15/9/2019 Image recognition with C# and Emgu libraries - CodeProject

Emiliano Musso
Software Developer Follow
Italy this Member

Working in IT since 2003 as Software Developer for Essetre Srl, a company in Northern Italy.
I was awarded in 2014, 2015 and 2016 with Microsoft MVP, for Visual Studio and Development Technologies expertise. My
technology interests and main skills are in .NET Framework, Visual Basic, Visual C# and SQL Server, but i'm proficient in PHP and
MySQL also.

Comments and Discussions

 
  Email Alerts Search Comments
Add a Comment or Question 

First Prev Next

SURF is patented one

Member 13486193 27-Feb-18 15:42

Give the Invalid Director y

Member 13493011 14-Feb-18 16:04

Re: Give the Invalid Directory

Member 13493011 14-Feb-18 17:33

Issue
Dhanish Balloo 2-Feb-18 0:13

Incredible Bug !
David Zenou 7-Aug-17 0:10

Nice work! 5+
Sergey Morenko 21-Jun-17 1:40

My vote of 5
D V L 15-Jun-17 18:04

My vote of 5
Alejandro Gaio 23-May-17 23:16

Re: My vote of 5
Emiliano Musso 30-May-17 3:11

Awesome!
Cr yptonite 23-May-17 3:47

Re: Awesome!
Emiliano Musso 23-May-17 19:12

https://www.codeproject.com/Articles/1187512/Image-recognition-with-Csharp-and-Emgu-libraries 13/14
15/9/2019 Image recognition with C# and Emgu libraries - CodeProject

My vote of 5
ThanhTrungDo 17-May-17 14:59

Re: My vote of 5
Emiliano Musso 30-May-17 3:11

Ver y good
ricardotrein 17-May-17 2:10

Re: Very good

Emiliano Musso 19-May-17 4:01

Great
Sophia95 16-May-17 3:45

Re: Great
Emiliano Musso 16-May-17 21:40

Refresh 1

General    News    Suggestion    Question    Bug    Answer    Joke    Praise    Rant    Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

Permalink Layout: fixed | fluid Article Copyright 2017 by Emiliano Musso

Advertise Everything else Copyright © CodeProject,
Privacy 1999-2019
Cookies
Terms of Use Web01 2.8.190913.2

https://www.codeproject.com/Articles/1187512/Image-recognition-with-Csharp-and-Emgu-libraries 14/14