​ ​AI for PRODUCT MANAGERS ND P1 CHEAT SHEET

Workflow For a Parking Sign Identification Annotation Job

This document is a step by step guideline for using Appen’s website to design a job for
identifying parking signs from a given dataset of images. You can use it as a guide in
completing project 1 : Create a Medical Image Annotation Job of your AI for Product Managers
ND. You can find the data set ​here​.

Table of Contents​ :

Step 1 Figure-eight login

Step 2 Job Decision

Step 3 Selecting Image Annotation Template

Step 4 How to Upload Data

Step 5 View of Uploaded Data

Step 6 Design a Job

Step 7 Map Data to Template

Step 8 Edit Title, CML code and Instructions

Step 9 Edit Overview, Steps, Rules

Step 10 Preview the Job instructions

Step 11 Create Test Questions - 1

Step 12 Create Test Questions - 2

Step 13 Answer Test Questions

Step 14 View Answer Distribution

Step 15 Finalise Design (Includes Step 10)

Login to Appen using your credentials. You will see the landing page as depicted below. On the
basis of the kind of job that you want to create, select one of the following options :

Step 2 :

I want to create a job for ​Image Annotation​, where the annotators will check the presence of
parking signs in a set of images given to them.

Step 3:

Under the ​Image Annotation category, you need to select the template that you think will be
best suited for creating the annotation job. Using the ​Preview section while screening for
templates will give you a fair idea of how useful a template is for your given job. Here, I want to
identify whether any given image contains a parking image or not, so I use the image
categorization template under the image annotation category.
Step 4 :

On selecting ​Use this template under the ​Image Categorization category, you are prompted
to upload your dataset as seen in the step below, here we can click on the ​browse button and
upload the required dataset.
Step 5:

On the successful upload of the dataset, you will see the entire dataset in the format shown
below.

Step 6:

Now, with data at your disposal and the notification in the lower right corner (in green) stating
that our file upload was successfully processed, we want to ​DESIGN our job to help the
annotators, so we go to the second tab above which is ​DESIGN ​ as can be seen below :

Step 7:

In this important step, we want to map our data to the existing template. Many students skip this
step, but this is a crucial step to align the template with our data.
In this mapping, we match (just drag and drop) appropriate columns from ​Your Data (on L.H.S)
to the ​Template​ column (on R.H.S) and click on ​Save Mapping.​

Step 8:

The next step is to edit the ​Title,​ ​CML code and ​Instructions according to the job we have in
hand. Since I want my annotators to identify whether an image contains a parking sign or not, I
edit all the three accordingly, as can be seen in the image below.
You can see that we have edited the ​Title to suit our job. The next critical step is to edit the
CML code for the same. We are simply asking the annotators a question, whether they think
that there is a parking sign in the given image or not, so we have two options ​Yes and ​No
(which were already present in the template). To account for uncertainty because of the lack of
clarity in pictures, we add one more checkbox to the existing Yes and No, a ​Maybe button for
when the annotators are unable to figure out if a parking sign actually exists and we simply edit
out the rest of the code. This is how the above CML code edit renders as a UI element :
This step might be baffling at first, but on thorough observation, you will figure out how you can
customise this ​CML code to suit your job. If it’s still not clear, you can always get additional help
from the help document on CML reference which is displayed to the right side of this editor (as
can be seen below) :

Clicking on any one of the items in this list will explain the what, how, when and whys of the
CML code elements being used in the editor. ​Please note that you simply need to edit the
basic CML code to suit our job, you do not need to delve too deep into the CML coding.

Step 9:

Next, we need to edit the ​Overview​, ​Steps and ​Rules/Tips suited to our job. Remember that in
this example, we are asking annotators to find whether an image contains a parking sign or not,
so we need to edit the instructions accordingly as seen below :
Step 10:

You don’t need to bother about the images displayed under this section that are a part of the
existing template, they can be edited out once we have our Test Questions designed. We can
ignore them for now and come back to this step later. Simply click on the ​Save​ button.

Please note that we can ​Preview the sections we have modified here, by clicking on the ‘eye’
button seen in the top right corner.

Also note that we can only see the modified changes in ​Preview if we have edited and saved
our changes using the ​Save​ button that is to the left of the ​Preview​ button.
Step 11:

The next step is to ​Create Test Questions,​ so click on the ​Create Test Questions button that
can be found right under the ​Preview​ button.

Step 12:

After we click on the ​Create Test Questions​ button in ​Step 11​, we see this page :

Click on the ​CREATE​ button, to create our own test questions.

Step 13:

Now we will start seeing images from the dataset that we just uploaded and we will need to
answer ​8 questions (they are displayed to us similarly to how annotators who we have created
the job for would see them), which will act as a guide to test the annotator’s competency when
we are onboarding them. Below is an example of the above mentioned view of the test :
The idea here is to have the entire range of reference test images divided equally among the
number of options we are providing. Since we are required to have 8 test images, we will find
and answer such images so that the tally for each possible option that we have designed (Yes
or No or Maybe) here is ​equal​. This is to make sure that our job isn’t biased towards accepting
and validating one specific option from the annotators. You can click the ​Skip Question ​button
if you feel that the responses are getting biased towards one specific option. You can keep
answering and checking the ​View Answer Distribution button (on top right) to keep a tally of
whether every option is being given a fair weightage in our designed test. Remember that for
each test question answered, we need to give a reason for why we chose that option in case
test takers falter so they have a clear understanding of why they faltered and which images they
annotated incorrectly. After checking the correct option and listing the reason, you click on ​Save
and Create Another.

NOTE : As you go about answering these questions, remember to take a few (2 or 3 max)
screenshots of the images displayed (one for each option would be ideal) and the options
provided below them so you can use them as a preview in ​Step 10​.
Step 14:

Once you have successfully answered the ​8 test questions, you will see this message to the
right.

You can click on ​View Answer Distribution to check whether each option has been given a
fair chance. The following is the distribution of my test questions:
We see here that the questions are fairly equally distributed among the three options, if you
want to continue to refine the weightages, you can answer a few more questions and check the
answer distribution again.

Step 15:

In the last step, we go back to the ​DESIGN ​tab (the second tab) on top, to eliminate the pictures
from the templates and insert the pictures that we just took while answering the test question in
Step 13, ​as can be seen below :

Here, we have the option to eliminate both the existing images and add and align the newer
images from our dataset to be displayed as a preview. Once we have those images, we can
save the preview page as ​.html and CHEERS! We have the file that we need to submit along
with the proposal! Congratulations!