Whale Keypoint

Work ow
Summary of Tasking
1. Annotate keypoints to the existing cuboid in the task -- use ‘n’ to quickly jump to the cuboid!
2. You MUST annotate the keypoint inside the body (except for Nose); keypoints are locations inside the
human body where two bones meet.  They are not on the LiDAR su ace. Use the side panel to move
keypoints into the body as needed. See below for explanations.
3. Do not delete/add/modify any cuboid in the task!
4. For Reviewers, impo ant rules must be followed here.
5. [Latest Update: 10/23/2020] Check the latest instruction updates here!

Whale Keypoint Spec Document

“Lidar Point  Annotation” | IN PROGRESS

Contents
● Annotation Logic Diagram
● Annotation Rules
● Keypoint Labels & A ributes Table
● Keypoint Description
● A ribute Description
● Correct Labeling Examples
● Critical vs. Non-Critical Errors
● Edge Case Scenario
● Impo ant Rules in Reviewing Keypoint Tasks
In this task, you will add points on pa s of the pedestrian in the 3D view. Add points by clicking the
“Key Point Annotation” bu on in the le side panel.
Annotation Logic Diagram

Situation How to Handle It

Pedestrian present in camera Annotate keypoint
view
Pedestrian not present at all in Do not annotate any keypoint
camera view
Obvious where the keypoint Annotate point
should be (using Lidar point Mark visible if it is visible in 2D
cloud as reference) camera view
Mark not visible if it is NOT
visible in 2D camera view
Not obvious where the keypoint Annotate point
should be, but we can guess with Mark not visible
high con dence based on
knowledge of human shape
NOT obvious where the keypoint Do not annotate the keypoint
should be

Annotation Rules

Rule Description Example

ADD The keypoints are locations Key Points

KEYPOINTS WHERE inside the human body where
JOINTS MEET two bones meet.  They are
 not su ace/skin points, except
Nose.
PEDESTRIAN PRESENT The pedestrian must be
IN CAMERA VIEW present in a camera view (i.e.
present in the lidar point cloud
and a corresponding camera
image).  However, not all
keypoints have to be visible in
the camera image.

Use the 3D LiDAR view to

place points in the correct
spot.

LEFT AND RIGHT Please use corresponding

LABELS RELATIVE TO Le /Right labels to be relative
THE PEDESTRIAN IN THE to the pedestrian in the
FRAME image.

OMIT OCCLUDED If the keypoint location is not In this example,

KEYPOINTS visible in the camera at all omit keypoints from Ankle-Le
AND and Ankle-Right since you do
It is impossible to not see lidar points for them:
guess where the keypoint Camera View:
should be, then DO NOT
ADD a keypoint.

Lidar View:
Keypoint Labels & A ributes Table

Class Label (Keypoint) A ribute

Human Skeleton Center of Head is_visible  
Nose
Shoulder-Le
Shoulder-Right
Elbow-Le
Elbow-Right
Wrist-Le
Wrist-Right
Hip-Le
Hip-Right
Knee-Le
Knee-Right
Ankle-Le
Ankle-Right

Correct Keypoint Labeling

2D View 3D View
^Notice
1. The key points are where the
bones meet (not on the
outer/su ace pa where the
skin is).
2. The points are inside the
pedestrian, not on the skin.
3. When you rst draw a
keypoint it will be on the lidar
su ace. You MUST move the
keypoint inside the body if
the keypoint is inside the
body, especially for HIPS. Use
the sidepanel to do so.
Keypoint Description

Label Description Example

Center of Head Center point between the ears In this example, you have to
and inside the head. make sure the Center of Head
keypoint is placed
center/middle in the side, top
and front/back lidar view.

PEDESTRIAN IN 2D CAMERA
Hip Hip keypoint should be where
the leg and hip bones meet -
inside the body, NOT at the
skin.

Nose Tip of the nose.

The Nose and Center of Head
point together create an
imaginary line in the direction
that the person is looking.
A ribute Description

A ribute Description Example

Is_visible: YES If the pedestrian is seen in 2D Wrist Le is visible since it is

camera view or keypoint obvious where the keypoint
location is obvious it should should be
be marked is_visible: yes

Is_visible: NO If the pedestrian is not

entirely seen in the camera
view but the keypoint
placement could be roughly
guessed based on prior
knowledge about the human
 body and other visible
keypoints — it should be still
labeled, but the speci c
keypoint that is not visible in
camera view should be
marked as is_visible: no.
NOTE:  Sometimes it is not obvious if a keypoint should be labeled as visible or not because of
image blur/dark image, occlusions colored similar to a person's closing. In such cases it is OK
to label a keypoint but is_visible a ribute should be NO.

Critical vs. Non-Critical Errors

Critical Errors
Pedestrian not seen at all via camera view.
Wrong: Labeled keypoint based on the lidar points alone.
Correct: do not annotate any keypoints 

Placement of keypoint where no lidar points are present

 
Wrong: Added ankle-le and ankle-right
Correct: omit ankle-le and ankle-right keypoints
Misplaced Keypoint
A. Su ace (Skin)
- keypoints should be placed “inside the body”

Example 1: 
● right shoulder keypoint
Example 2: 
● all points are on the su ace/skin

B. Incorrect Placement / Outside Data

- Keypoint is far from the expected annotation location

Example 1:
● Outside data annotation
Example 2:
● knees are too high
● both ankles are in incorrect locations

Example 3:
● all points except the right hip are in correct locations. The right hip is misplaced - far back, likely because  a bunch
of lidar points on the bag.

Example 4:
● Center of Head keypoint placed on top of the head
Incorrect Visibility A ribute
Wrong: Pedestrian not seen in camera but was added keypoints
Correct: do not annotate any keypoint 

Non-Critical Errors
A critical error message will automatically show upon submission of the task when detected not labeling 14/14
keypoints.

                

Example Scenario: For this example, L&R ankle keypoints are omi ed since no lidar points are present. In this case, the
overall number of keypoints is less than 14 WHICH IS FINE. To dismiss the error and to be able to submit the task, follow
these steps:
1. Go to the critical error message.
2. Dismiss the error message by clicking ✘, and con rm action.
3. Try to submit again!

Edge Case Scenario

Edge Case Scenario Solution
Labeling through glass Do not label keypoints for which lidar
- Keypoints that are visible in camera may not be visible in reference keypoints are missing.
lidar view.

Bad Data (Cuboid) Do not label/submit - flag the task to

your admin/QM
A. Floated/Sinked Cuboid

Impo ant Rules in Reviewing Keypoint Tasks

Overview
● Each keypoint has 4 boxes that can be marked check (✔) or marked wrong (✘),
corresponding to the 4 evaluation categories.
● To mark a category ✔ means the tasker correctly labeled the keypoint.
● To mark a category ✘ means the tasker incorrectly labeled the keypoint.
● If there are multiple errors found for each keypoint, mark ✘ for all a ected categories.
● All reviewers SHOULD MARK ALL CATEGORIES before approving or rejecting the review
task.
○ This includes the Presence category for unused keypoints
○ If there are any errors, they need to be xed before submi ing with changes
● All tasks that are approved or approved with changes should have NO ERRORS UPON
SUBMISSION

Evaluation  When to mark check:   When to mark wrong: How to x the error
Category
Presence Keypoint was labeled when Case A error: Keypoint was Case A error: add the
it should have been labeled. not labeled/missing when it keypoint to the correct
should have been labeled. location, select the
Keypoint was not labeled correct visibility,  and
when it should not have Case B error: Keypoint was mark the Presence box
been labeled. labeled when it should not with an ✘. Mark all
have been labeled. other categories with a
✔
Case B error: delete
the keypoint rst, and
then mark its Presence
category with an ✘

Type The keypoint label Case A error: The body pa Case A error: delete
corresponds to the correct is labeled with the wrong the incorrect keypoint
body pa . keypoint type, and the type, add the correct
correct keypoint type was keypoint type to the
not labeled body pa , and mark
the Type box with an ✘
Case B error: The body pa
is labeled with the wrong Case B error: Swap the
keypoint type, and the location of the
correct keypoint type is keypoints to x the
being used for a di erent error. Mark the Type
body pa category with an ✘ for
all a ected keypoints.

Note: do not ✘ the

position category when
xing this error, unless
the position of the
incorrect keypoint was
also wrong (e.g.,
keypoint outside of
point cloud)
Visibility The correct visibility The wrong visibility a ribute Update the visibility
A ribute a ribute was selected. was selected. a ribute with the
(is_visible) correct value and mark
 the Visibility A ribute
category with an ✘
Position The keypoint is placed in the The keypoint is placed in the Move the keypoint to
right location. wrong location. the correct position
and mark the position
category with an ✘ 

History / Change Log

Version 1.0
● 07/02/2020 - updated the entire handbook
Version 1.2
● 10/09/2020 - added more examples on Critical vs. Non-Critical Errors
● 10/10/2020 - updated Non-Critical Errors
Version 1.3
● 10/16/2020 - added Impo ant Rules in Reviewing Keypoint Tasks section
● 10/21/2020 - updated owcha and clari ed that camera view is to be used as sta ing reference
● 10/22/2020 - added additional instructions in Impo ant Rules in Reviewing Keypoint Tasks section on
how reviewers shall x the errors
● 10/23/2020 - added Edge Case Scenario section