The complete guide

to image annotation
01 What does it mean to
annotate an image?
Image annotation is defined as the task of annotating an image with labels, typically involving
human-powered work and in some cases, computer-assisted help. Labels are predetermined
by a machine learning engineer and are chosen to give the computer vision model information
about what is shown in the image. The process of labeling images also helps machine learning
engineers hone in on important factors that determine the overall precision and accuracy of
their model. Example considerations include possible naming and categorization issues, how to
represent occluded objects, how to deal with parts of the image that are unrecognizable, etc.

02 How do you annotate an image?

From the example image below, a person applies a series of labels by applying bounding boxes
to the relevant objects, thereby annotating the image. In this case, pedestrians are marked
in blue and taxis are marked in yellow, while trucks are marked in yellow. This process is then
repeated and depending on the business use case and project, the quantity of labels on each
image can vary. Some projects will require only one label to represent the content of an entire
image (e.g., image classification). Other projects could require multiple objects to be tagged
within a single image, each with a different label (e.g., bounding boxes).

Bounding boxes applied to identify vehicle types and pedestrians

2
03 What are the different types
of image annotation?
In order to create a novel labeled dataset, data scientists and ML engineers have the choice
between a variety of annotation types. Let’s compare and summarize the three common
annotation types within computer vision:
1) classification 2) object detection and 3) image segmentation.

With whole-image classification, the goal is to simply identify which objects and other
properties exist in an image.

With image object detection, you go one step further to find the position (bounding boxes) of
individual objects.

With image segmentation, the goal is to recognize and understand what’s in the image at the
pixel level. Every pixel in an image belongs to at least one class, as opposed to object detection
where the bounding boxes of objects can overlap.

Classification Object detection

Image segmentation Image segmentation with instances

3
 Whole image classification provides a broad categorization on an image and is a step up from
unsupervised learning as it associates an entire image with just one label. A distinct benefit it
is by far the easiest and quickest to annotate out of the other common options. Whole-image
classification is also a good option for abstract information such as scene detection and time of
day.

Bounding boxes, on the other hand, are the standard for most object detection use cases and
requires a higher level of granularity than whole-image classification. Bounding boxes provide a
balance between quick annotation speed and targeting items of interest.

For specificity, image segmentation is chosen to support use cases in a model where you need
to definitively know whether or not an image contains the object of interest and also what
isn’t an object of interest. This is in contrast to other annotation types such as classification or
bounding boxes that may be faster in nature but less accurate.

04 How does a training data platform

support complex image annotation?
Image annotation projects begin by identifying and instructing annotators to perform the
annotation tasks. Annotators must be thoroughly trained on the specifications and guidelines
of each annotation project, as every company will have different requirements.

Once the annotators are trained on how to annotate the data, they will begin annotating
hundreds or thousands of images on a training data platform dedicated to image annotation. A
training data platform is software that is designed to have all the necessary tools for the desired
type of annotation and is commonly equipped with multiple tools which allow you to outline
complex shapes for image annotation.

In addition, training data platforms typically include additional features that specifically help
optimize your image annotation projects which include:

High-performance annotation tools:

An important point to consider and test is whether or not the tools provided by the training data
platform you are testing can support a high number of objects and labels per image without
sacrificing loading times. At Labelbox, our vector pen tool allows you to draw freehand as well
as straight lines. Blazingly fast and ergonomic drawing tools help reduce the time-consuming
nature of having pixel-perfect labels consistently.

4
Labelbox pen tool illustrated

Customization based on ontology requirements:

The ability to configure the label editor to your exact data structure (ontology) requirements,
with the ability to further classify instances that you have segmented. Ontology management
includes classifications, custom attributes, hierarchical relationships and more.

Configure the label editor to your exact data structure (ontology) requirements.

A streamlined user interface which emphasizes performance for a wide array of devices:

An intuitive design helps lower the cognitive load on labelers which enables fast labeling. Even
on lower spec PCs and laptops, performance becomes critical for professional labelers who are
working in an annotation editor all day.

5
A simple, intuitive UI reduces friction

Seamlessly connect your data via Python SDK or API:

Stream data into your training data platform and push labeled data into training environments
like TensorFlow and PyTorch. Labelbox was built to be developer friendly and API-first so you
can use it as infrastructure to scale up and connect your ML models to accelerate labeling
productivity and orchestrate active learning.

pip install labelbox

os.environ[“LABELBOX_API_KEY”] = ‘your_api_key’
lb = labelbox.Client()

dataset = lb.create_dataset(name=”Tesla dataset upload example”)

task = dataset.create_data_rows(local_file_paths)
task.wait_till_done()
print(‘Upload complete.’)

Simplified data import without writing and maintaining your own scripts

Benchmarks & Consensus:

Quality is measured by both the consistency and the accuracy of labeled data. The industry
standard methods for calculating training data quality are benchmarks (aka gold standard),
consensus, and review. As a data scientist in AI, an essential part of your job is figuring out
what combination of these quality assurance procedures is right for your ML project. Quality
assurance is an automated process that operates continuously throughout your training data

6
development and improvement processes. With Labelbox consensus and benchmark features,
you can automate consistency and accuracy tests. These tests allow you to customize the
percentage of your data to test and the number of labelers that will annotate the test data.

Benchmarks in action, highlighting the example labeled asset with a gold star

Collaboration and Performance Monitoring:

Having an organized system to invite and supervise all you labelers during an image annotation
project is important for both scalability and security. A training data platform should include
granular options to invite users and to review the work of each one.

With Labelbox, setting up a project and inviting new members is extremely easy, and there
are many options for monitoring their performance, including statistics on seconds needed to
label an image. You can implement several quality control mechanisms, including activating
automatic consensus between different labelers or setting gold standard benchmarks.

Seamless collaboration between data science teams, domain experts, and dedicated external labeling teams

7
05 Learn more
Want to learn more about image annotation capabilities at Labelbox?
Click here to speak with us.

8
Learn more about Labelbox at labelbox.com