UCS671:Embedded Vision

Jhilik Bhattacharya
February 12, 2024

Abstract
In this course you will learn to work on the Jetson Nano kit and the Jetbot. You will learn
about Nvidia NGC docker images,how to use them and deploy different CV applications on the
kit.

1 Prerequisites
• Mandatory: Dual boot with Ubuntu

• Mandatory: Your system should have atleast 1 USB-A to attach data cable, else get a converter
(USB-C to USB-A)
• Preferred: Carry an ethernet cable
• Preferred: Carry a USB-C to USB-A cable (to be attached to your USB-A laptop port, alternative
for power plug)

2 Lecture 1,2: Connecting with Jetson and Overview of dock-

ers
In this session you will issue the Jetson Nano board. Some kits are already mounted on the Jetbot
whereas some are available individually. We need the Jetson board, a micro-usb data cable and the
power cable to connect. Make sure your jetson kits do not touch the table. Once you power on, open
a linux terminal on your laptop and connect to the jetson using ssh.
ssh -X [email protected]
csed is the user id created on the jetson. IP assigned due to the data cable is 192.168.55.1. The laptop
is assigned 192.168.55.100. The −X option is used to transfer graphical transfer rights to the host(your
laptop). Once you are logged in, the terminal becomes your jetson terminal. You will be able to see
the difference in your terminal prompt. If you connect with ssh without the username , it will work
only if your laptop username is same as the jetson username. You can check whether the graphical
transfer is working by running gedit on your jetson terminal.
Now that you are in the jetson machine you can use linux based commands(ls,cd etc.) to move around
on your machine. Your jetson board is currently fully functional with installed OS as well as other
docker images. We will in a minute learn more about docker images. Before that, you can take a
quick look at the steps for installing OS on a new jetson board in case you have to start from scratch.
You can refer the Nvidia developer page for instructions to burn the OS image on the micro SD card
using any software like etcher and starting your jetson device. The first time you need to connect USB
mouse,keyboard and a monitor (using the HDMI port) to the Jetson. Once the GUI starts, follow the
steps to create user account and get get the board ready. To maintain consistency, use username as
”csed” and password as ”cslab768”.
Coming back to the concept of dockers, they are an executable package to run your application. They
include all software dependencies within it and can be run on any system. A docker builds images and
runs the instance of the image (called container). The nvidia NGC catalog hosts a number of docker
images which can be pulled using the following command
sudo docker pull xxxx

1
xxxx denotes the name of the image, for example nvcr.io/nvidia/l4t − ml : r36.2.0 − py3. In the
x : y format, x represents the repository and y gives the tag name. The tag name denotes the jetpack
version number as well as the software version. The jetpack version and associated tag numbers are
shown below:

• JetPack 5.1.1 (L4T R35.3.1)

• JetPack 5.1 (L4T R35.2.1)
• JetPack 5.0.2 (L4T R35.1.0)
• JetPack 5.0.1 Developer Preview (L4T R34.1.1)

• JetPack 5.0.0 Developer Preview (L4T R34.1.0)

• JetPack 4.6.1 (L4T R32.7.1)
• JetPack 4.6 (L4T R32.6.1)
• JetPack 4.5 (L4T R32.5.0)

• JetPack 4.4.1 (L4T R32.4.4)

• JetPack 4.4 (L4T R32.4.3)
• JetPack 4.4 Developer Preview (L4T R32.4.2)

Your jetpack version is 4.6. Recall this while downloading any docker image. At this point if a question
pops in your mind- ”Whats the advantage of using docker images over conda environments? ”. Doesnt
a conda environment do a similar thing? You can create an environment, activate it and have a
list of packages(softwares) installed there. This does not clash with software versions in any other
environment. If a docker image also offers us an environment with all softwares installed, whats the
advantage of dockers over conda environment. Well the simplest answer is that docker image allows
you a OS level virtualization. You can run a linux container from windows OS if you are using dockers.
You can view the list of docker images installed on your jetson by executing
sudo docker images
For checking the list of active containers you can use
sudo docker ps
For saving any container as an image you can commit the container using
sudo docker commit container id repository:tagname
You run a docker image using
sudo docker run xxxx
where xxxx is the repository:tag for a docker image. Take a note of different options like it, rm,
network, runtime, volume, device you can use
Now that you know how to run a container, you should be able to answer the following questions

• You have connected a device to your jetson nano but cannot access it from inside a container.
Why?
• How to prevent loosing all your data saved while running the container?

• When is interactive mode (-it) not required?

• When is it important to not use the rm option?
Finally you can delete images and containers using
sudo docker rmi image id
sudo docker rm container id
Alternatively you may also use repository:tag for images or names for containers.

2
3 Lab 1: Getting Acquainted with dockers
In this session you will try the following

• Run the docker image nvcr.io/nvidia/l4t-ml:r36.2.0-py3 and check which version of pytorch,
tensorflow, opencv, numpy are installed here
• Run the docker with rm option, install a software (say matplotlib or pillow) and exit. Run it
again and check if you are able to access the softwares

• Repeat the experiment without using rm option.

• Save the container as a new image
• Delete the new image you just created

• Check whether the container you saved is still available and save an image again
• This time delete the container you are using and check if you can run the image you have saved.

4 Lecture 3,4: Writing a docker file and building it to create

a docker image
In the last class you have experimented with some docker images from NGC catalog. You have run
them and installed softwares. Later you have commited the containers as a new image. In this class
you will learn to write a docker image from scratch. Begin by logging in to your jetson nano. Also
make sure that the internet is connected to your jetson board. On your jetson desktop create a folder
using mkdir command and cd into it. Create a file called Dockerfile here
mkdir ex2;cd ex2; vi Dockerfile

Once you open this file , use the insert mode (by pressing i) to edit the file. Once you finish editing,
escape the insert mode (using escape key) and save the file using (using :wq). You can build the docker
file as an image using the docker build command
sudo docker build -t jetpose:v1 .
The name jetpose:v1 is the image repository:tag you give to the image you have created. The ”.” in
the end signifies that your docker file is in this directory. You can check that your image is built by
using the sudo docker images command.
The docker file uses the following main keywords
• FROM keyword is used to include an already existing docker image.
• RUN keyword is used to execute any shell command like pip install matplotlib

• WORKDIR sets the path inside the container

• COPY command is similar to cp in shell and works in the same manner
A sample docker file is shown below. The file pulls the pytorch image, installs matplotlib, clones 2
github repos and installs the setup files.

FROM nvcr.io/nvidia/l4t-pytorch:r32.6.1-pth1.9-py3
RUN pip install matplotlib
WORKDIR /home
RUN git clone https://github.com/NVIDIA-AI-IOT/torch2trt
WORKDIR /home/torch2trt
RUN python3 setup.py install –plugins
RUN pip3 install tqdm cython pycocotools
WORKDIR /home
RUN git clone https://github.com/NVIDIA-AI-IOT/trt pose
WORKDIR /home/trt pose

3
RUN python3 setup.py install
WORKDIR /home

5 Lab2: Buiding docker images

Perform the following operations in this lab
• Create a docker file using the example in the previous section and build an image jetpose:v1
• Run the image using the docker run command you learnt previously. Traverse the folders and
check the installations of the github repo you used

• Replace the workdir command with run cd and build another docker image jetpose:v2. What
are the differences?
• Pull the github repos in you jetson and use the COPY command inside the docker file to copy
the folders from the jetson to the container. Do this instead of cloning inside the docker file.
Build this image as jetpose:v3.
At the end of the session you should be able to answer the following questions
• In how many ways can you create a docker image?
• What are the pros and con of using jetpose:v1 vs jetpose:v3

• Suppose you have to edit a setup file you pulled from a github repo. In how many ways can you
do it?

6 Lecture 5,6: Working with different Jetson Inference tasks

The primary CV tasks are Object Detection, Image Classification, Pose Estimation, Action Clas-
sification, Scene Segmentation, Depth Estimation, Image Enhancement. These are then pipelined
in a framework for any application. For example if the task is to build a driver assistance system
for navigation in a poor visibility condition such as fog, the system pipeline will include an image
enhancement module which can improve the image quality, then perform object detection from the
enhanced image and report the driver about objects i.e. vehicles, pedestrian in the scene. If the
application is to be made more specific, then a depth estimation module can be used to generate the
scene depth for each pixel. This will then be used to calculate the distance of the car from individual
objects(vehicle/pedestrian) etc.
There are a lot models which are already trained on benchmark datasets. While selecting a pretrained
model for an application you need to focus on the model size to understand its adaptibility for an edge
device. For object detection models you can take a look at the Tensorflow1 Object Detection Model
Zoo or the the Tensorflow2 Object Detection Model Zoo . You need to select a model which has the
most optimized time in ms and Map score. By optimized we mean, a model which can work fast with
acceptable accuracy. Detection models are mainly trained using COCO dataset. You will also find a
few trained with PascalVOC, Kitti. Other than the model size vs performance what other choices do
you need to make during a model selection? Consider the task you have to perform. The Kitti dataset
is a mostly a traffic scene dataset. If your end application is for on-road devices, this is a good choice.
However if you are looking for a visually-impaired assistive device, a COCO trained model is more
effective as the object classes include both indoor and outdoor objects.
You can also use the Tensorflow2 Classification Model Zoo for getting hold of classification models
pretrained using the 1000 classes ImageNet. Now that you are slowly becoming a more advanced
computer vision researcher, the names of popular datasets should be at your fingertips. Here’s a list of
them for you to checkout. You need to have a brief idea about what tasks these datasets are annotated
for.
• Kitti

4
 • COCO
• PascalVOC
• Cityscapes
• Bereley Driving Dataset

• Oxford Robotcar dataset

• ImageNet
• CIFAR

• CALTECH
• Openpose
• NYUDepth
While you check each dataset make sure you understand what are the labels in each cases. For example
for classification task where the entire image is labeled as a class, its just a name. For detection tasks,
annotation files in xml or txt files will be available. For segmentation or depth datasets, the label is
another image like a mask image or a depth image. For enhancement tasks, the output is also another
image known as the groundtruth. For example for a hazy dataset, the labels are clear hazefree images.
Once you have a clear understanding of different datasets and formats, its time to try the jetson in-
ference module. You can follow the steps given below:

#On your jetson run

sudo docker images

#If you find the below entry search jetson-inference folder on your jetson device
#dustynv/jetson-inference r32.6.1 5363eb4a3327 18 months ago 2.71GB

# If you dont find the dusty image then

git clone –recursive –depth=1 https://github.com/dusty-nv/jetson-inference

cd jetson-inference
docker/run.sh

You will recall we discussed that the run.sh file inside the docker folder has the sudo docker run
command with all options. Instead of typing it on the terminal you can always write the command in
a sh file and just run it.
#Once inside the container for running an image classification/segmentation example

cd jetson-inference/build/aarch64/bin
./imagenet.py images/orange 0.jpg images/test/output 0.jpg #classification
./segnet.py –network=fcn-resnet18-cityscapes images/city 0.jpg images/test /output.jpg #segmenta-
tion

The test .py files are available for you in the bin directory. You can check the files to see the default
parameters. You can add options like –network if you want to work with a different network. The list
of networks can be checked from jetson-inference/data/networks folder. You can provide the input

5
image path and the path to save the output. You can view them even after exiting the container inside
the jetson-inference/data/images folder on your jetson as you had mounted them inside the container.
Inorder to view the images either open jupyter-notebook on your jetson (not container). Alternatively
you can use the following from your jetson terminal (not container)

scp images/orange 0.jpg [email protected]:/home/jhilik/Desktop/

Use the username of your laptop (not jetson) instead of my name. The image is copied from the
jetson device to your laptop desktop. Assuming that ssh works on your laptop, you are in the correct
folder in your jetson terminal (here we assume you are inside jetson-inference folder and can see images
subfolder), it will work.
For taking continuous feed from the camera (/dev/video0) and saving the output in a video file(out.mp4),
you can run any of the detection/classification/segmentation codes with

./detectnet.py /dev/video0 out.mp4 # detection

Looking at the code, you will find that the camera captures continuous frames using an infinite while
loop. You can use a variable say c to limit the number of frames you want to use

c=0
while True:
.
.
.
c+ =1
if c>1000:
break

Also, you can use the detection object to extract particular information.

for detection in detections:

if detection.ClassID==1:
print(”......”)

This will print a message when the ClassID is 1. You can similarly extract bounding box locations
using detection.T op, detection.Bottom etc. The members of the object (i.e. ClassID, Top, Left etc)
will be visible when you print the object using print(detection). Use the saveImageRGBA function
of jetson.utils to save parts of Images extracted using the bounding box information.

7 Lab 3: Getting Familar with the Jetson Inference Module

provided by Dusty
• Execute the detection,classification,segmentation tasks using input images and output image
paths
• Execute the detection tasks using camera input and output image paths
• Execute the detection tasks using camera input and output video path
• Modify the detection code to give you the count of people detected

8 Lecture 7,8: Use Jetson Inference

Now that you have learnt how to use the camera with the inference code, you should be able to run
your code in realtime. Real applications do not need a display to view the result. The result can be

6
directly used to take different decisions. If you need to maintain a repository, you may save the results
as video files. Here are a few examples of what you should be able to do:
• Can your device save images of all the people it sees. Remember only the person, not the whole
image or video.
• Can you keep a note of all the people and animals seen in a live feed
• Assume that a feed is viewing a constant scene. Can you provide a total accuracy of all the
objects viewed. For example say in 30 frames, an object was detected as cat in 20, and as a
dog in the remaining frames. So you can say that the detected object may be a cat with 66%
confidence and dog with 33% chance.

9 Lab 4:
This is a good time to have a deeper understanding of label files and format. Take a look at the
following
• object annotation in KITTI format
• object annotation in COCO format
• object annotation in PASCALVOC format
• xml and txt files for annotation
• Find 3 pose datasets and 3 depth datasets other than the one mentioned above.

10 Lecture 9,10:
In this session you will run your first code for capturing a dataset, training it and running it for infer-
ence . Upon completion, you’ll be able to create your own deep learning classification and regression
models with the Jetson Nano. These are the steps
• Set up your Jetson Nano and camera. You have already done this.
• Collect image data for classification models
• Annotate image data for regression models
• Train a neural network on your data to create your own models
• Run inference on the Jetson Nano with the models you create
• Complete the certification
Some common issues are listed here
• While running the docker-run.sh file ensure that you are using the correct image version which
is already there in your jetson device. You can check the version with sudo docker images
command
• Make sure to check that you have used the appropriate code for your 2GB device and camera
usb/csi.
• If your camera isnt working check with attached display device
• If jupyter notebook isnt working: You can open the a browser from the jetson device (not inside
container) using the command chromium-browser. Use another new terminal in your laptop
to connect to the jetson again via ssh as chromium is running from the previous one. Once
connected run the container and open jupyter inside the container using jupyter notebook
–allow-root –no-browser. Once jupyter starts running, copy the ip (you will find that on the
second last line) and paste on the chromium browser you opened. Type the password and login.

7
11 Lab 5
In this session you will get familiar with theTransfer Learning Toolkit. Go through the CV Model
Zoo already available in details taking into consideration the training algorithm and use cases. Almost
all models mention Pruning aware training mechanisms. Its time that we step into model optimization
techniques using pruning aware, quantization aware tools paving way for TensorRT optimization tool.
A summary of popular pruning criterias and tools are provided here.