Face Recognition in The Browser With Tensorflow - Js & JavaScript
Face Recognition in The Browser With Tensorflow - Js & JavaScript
js & JavaScript
As always we will look into a simple code example, that will get you started immediately
with the package in just a few lines of code. Let’s dive into it!
At first, I did not expect there being such a high demand for a face recognition package
in the javascript community. For a lot of people face-recognition.js seems to be a decent
free to use and open source alternative to paid services for face recognition, as
provided by Microsoft or Amazon for example. But I also have been asked a lot,
whether it is possible to run the full face recognition pipeline entirely in the browser.
Finally it is, thanks to tensorflow.js! I managed to implement partially similar tools using
tfjs-core, which will get you almost the same results as face-recognition.js**,** but in
https://school.geekwall.in/p/Hy29kFEGm 1/13
8/2/2021 Face Recognition in the Browser with Tensorflow.js & JavaScript
the browser! And the best part about it is, there is no need to set up any external
dependencies, it works straight out of the box. As a bonus it is GPU accelerated,
running operations on WebGL.
This was reason enough to convince me, that the javascript community needs such a
package for the browser! I’ll leave it up to your imagination, what variety of applications
you can build with this. ;)
Sounds like a plan! However, two problems remain. Firstly, what if we have an image
showing multiple persons and we want to recognize all of them? And secondly, we need
to be able to obtain such kind of a similarity metric for two face images in order to
compare them…
Face Detection
The answer to the first problem is face detection. Simply put, we will first locate all the
faces in the input image**.** For face detection, face-api.js implements a SSD (Single
Shot Multibox Detector), which is basically a CNN based on MobileNetV1, with some
additional box prediction layers stacked on top of the network.
The network returns the bounding boxes of each face, with their corresponding
scores, e.g. the probability of each bounding box showing a face. The scores are used
to filter the bounding boxes, as it might be that an image does not contain any face at
all. Note, that face detection should also be performed even if there is only one person
in order to retrieve the bounding box.
https://school.geekwall.in/p/Hy29kFEGm 2/13
8/2/2021 Face Recognition in the Browser with Tensorflow.js & JavaScript
For that purpose face-api.js implements a simple CNN, which returns the 68 point face
landmarks of a given face image:
From the landmark positions, the bounding box can be centered on the face. In the
following you can see the result of face detection (left) compared to the aligned face
image (right):
Face Recognition
Now we can feed the extracted and aligned face images into the face recognition
network, which is based on a ResNet-34 like architecture and basically corresponds to
https://school.geekwall.in/p/Hy29kFEGm 3/13
8/2/2021 Face Recognition in the Browser with Tensorflow.js & JavaScript
the architecture implemented in dlib. The network has been trained to learn to map the
characteristics of a human face to a face descriptor (a feature vector with 128 values),
which is also oftentimes referred to as face embeddings.
Now to come back to our original problem of comparing two faces: We will use the face
descriptor of each extracted face image and compare them with the face descriptors of
the reference data. More precisely, we can compute the euclidean distance between
two face descriptors and judge whether two faces are similar based on a threshold
value (for 150 x 150 sized face images 0.6 is a good threshold value). Using euclidean
distance works surprisingly well, but of course you can use any kind of classifier of
your choice. The following gif visualizes the comparison of two face images by
euclidean distance:
And now that we ingested the theory of face recognition, we can start coding an
example.
Coding time!
In this short example we will see step by step how to run face recognition on the
following input image showing multiple persons:
https://school.geekwall.in/p/Hy29kFEGm 4/13
8/2/2021 Face Recognition in the Browser with Tensorflow.js & JavaScript
Including the Script
First of all, get the latest build from dist/face-api.js or the minifed version from
dist/face-api.min.js and include the script:
<script src="face-api.js"></script>
npm i face-api.js
The model weights have been quantized to reduce the model file size by 75%
compared to the original model to allow your client to only load the minimum data
required. Furthermore, the model weights are split into chunks of max 4 MB, to allow
the browser to cache these files, such that they only have to be loaded once.
The model files can simply be provided as static assets in your web app or you can host
them somewhere else and they can be loaded by specifying the route or url to the files.
Let’s say you are providing them in a models directory along with your assets under
public/models:
loadAllModels.js
await faceapi.loadModels(MODEL_URL)
https://school.geekwall.in/p/Hy29kFEGm 5/13
8/2/2021 Face Recognition in the Browser with Tensorflow.js & JavaScript
await faceapi.loadFaceDetectionModel(MODEL_URL)
await faceapi.loadFaceLandmarkModel(MODEL_URL)
await faceapi.loadFaceRecognitionModel(MODEL_URL)
allFaces.js
A full face description holds the detecton result (bounding box + score), the face
landmarks as well as the computed descriptor. As you can see faceapi.allFaces does
everything discussed in the previous section under the hood for us. However, you can
also obtain the face locations and landmarks manually. There are several examples
available on the github repo, if this is your goal.
Note, that the bounding boxes and landmark positions are relative to the original image
/ media size. In case the displayed image size does not correspond to the original
image size you can simply resize them:
forSize.js
We can visualize the detection results by drawing the bounding boxes into a canvas:
drawDetection.js
fullFaceDescription.forEach((fd, i) => {
})
https://school.geekwall.in/p/Hy29kFEGm 6/13
8/2/2021 Face Recognition in the Browser with Tensorflow.js & JavaScript
drawLandmarks.js
fullFaceDescription.forEach((fd, i) => {
})
Face Recognition
Now that we know how to retrieve the locations and descriptors of all faces given an
input image, we will get some images showing one person each and compute their face
descriptors. These descriptors will be our reference data.
Assuming we have some example images for our subjects available, we first fetch the
images from an url and create HTML image elements from their data buffers using
faceapi.bufferToImage:
fetchImages.js
https://school.geekwall.in/p/Hy29kFEGm 7/13
8/2/2021 Face Recognition in the Browser with Tensorflow.js & JavaScript
['sheldon.png' 'raj.png', 'leonard.png', 'howard.png'].map(
))
Next, for each image we locate the subjects face and compute the face descriptor, just
as we did previously with our input image:
getDescriptors.js
))
Now, everything that remains to be done is to loop through the face descriptions of our
input image and find the descriptor with the lowest distance in our reference data:
faceRecognition.js
refDesc => ({
label: labels[i],
})
).sort(sortAsc)[0]
return {
detection: fd.detection,
label: bestMatch.label,
distance: bestMatch.distance
})
As mentioned before, we use euclidean distance as a similarity metric here, which turns
out to work pretty well. We end up with a best match for each face detected in our input
https://school.geekwall.in/p/Hy29kFEGm 8/13
8/2/2021 Face Recognition in the Browser with Tensorflow.js & JavaScript
image.
Finally we can draw the bounding boxes together with their labels into a canvas to
display the results:
drawResults.js
results.forEach(result => {
faceapi.drawText(
canvas.getContext('2d'),
x,
y + boxHeight,
text
})
There we go! By now, I hope you got a first idea how to use the api. Also I’d
recommend to take a look at the other examples in the repo. And now, have fun playing
around with the package! ;)
Suggest:
☞ Machine Learning Zero to Hero - Learn Machine Learning from scratch
☞ Introduction to Machine Learning with TensorFlow.js
☞ TensorFlow.js Bringing Machine Learning to the Web and Beyond
https://school.geekwall.in/p/Hy29kFEGm 9/13
8/2/2021 Face Recognition in the Browser with Tensorflow.js & JavaScript
https://school.geekwall.in/p/Hy29kFEGm 10/13
8/2/2021 Face Recognition in the Browser with Tensorflow.js & JavaScript
https://school.geekwall.in/p/Hy29kFEGm 11/13
8/2/2021 Face Recognition in the Browser with Tensorflow.js & JavaScript
Django & ReactJS Full Stack Course [ Python Backend React Frontend
]
https://school.geekwall.in/p/Hy29kFEGm 12/13
8/2/2021 Face Recognition in the Browser with Tensorflow.js & JavaScript
React Native Course for Beginners [Build Mobile Apps in React Native]
© Made with
by Geekwall
https://school.geekwall.in/p/Hy29kFEGm 13/13