Guidelines: Pecos v2

Contents
1. Project Overview
2. Round Two Rating Steps
a. Details Steps
b. Sensitive Content
3. Key Rating Concepts
a. Relevance
b. Granularity
c. Quality
d. Entities
4. Research Tips
a. Side Searches
5. Feedback

Project Overview

In this project, you will review a social media post and then label all of the topics to
which it's related. The labeling flow consists of two rounds:
• Round One: You will view a random social media post and then select from a
group of broad topics (e.g. Sports, Food), as well as a few other non-topical
attributes.
• Round Two: The posts are then routed to the appropriate topic-specific
Round Two queue(s), based on the topics selected in Round One. In Round
Two, you will focus on posts that were previously assigned to just one of the
broad topics, and then choose among a large group of much more specific
subtopics which relate to it.
These Guidelines are for Round Two of the project. In Round Two you will be
assigned to a number of the Round Two topic queues, based on topic knowledge (as
demonstrated in qualification) and task availability.
These Guidelines provide a summary of the rating process, and things to consider
when working in Round Two.

Round Two Rating Steps

Detailed Steps

1. Review Post Content

 Look at the post on the left (1) and spend 30-60 seconds to make sure that you
understand it. It's important to consider any displayed text in conjunction with any
media - for example, you'd want to think about how a photo might be interpreted in
combination with any text that's present. In addition to the text itself, if the post
contains...
a. ...a photo, consider the photo(s) (including text overlaid on it), as well
as any text appearing above the photo.
b. ...text only, consider the text (including emojis) in the post.
c. ...a link, click on the link and quickly scan the content on the page,
then also consider any accompanying text in the post itself.
d. ...a video, watch the video for up to 60 seconds (if the video is long
watch first 10-15 seconds, click through 5 spots and watch 5-10
seconds at each) then also consider any accompanying text.

2. Conduct Research if Needed

A Side Search can be used to help determine what the content of a post is about
when it is unfamiliar. Side searching involves searching for key terms or names
featured in the post via a search engine in a separate browser tab. It won’t be
necessary to do this for all posts, since many can be understood simply by reviewing
the content. You will also encounter posts which feature content that may be
familiar to you already. However, in some instances side searches can be highly
beneficial in helping you provide more accurate labels for unfamiliar content.

See the ‘Research Tips’ section of these Guidelines for more detailed examples of
how to use Side Searches to conduct research.

3. Use the Typeahead to Select the Subtopics that the Post is about
Using the Typeahead, enter any subtopics or entities that are relevant to the post.
Please try to be as thorough as possible and enter all that apply, regardless of
whether you’re in the queue for that subtopic. When you do so, the Typeahead will
automatically auto-populate with any parent subtopics as well:

Selecting a subtopic from the suggested list will show the different levels of
subtopics:
Important: Include all subtopics that are relevant to the post. If there are
subtopics that relate to the post, but are not part of the queue topic, it is still helpful
to include these as well. These can only be added from the Subtopics Typeahead. For
example, in the Relationships queue, you should select a relevant subtopic for the
post (here LGBTQ Relationships), and if the post is also related to a Social Issue, you
can add the relevant Social Issues subtopic in the additional subtopic box (here, the
subtopic Gender Identity & Sexual Orientation (LGBTQ) Rights & Discrimination was
added).

4. Use the Subtopic Picker

You are able to enter all of the subtopics that are relevant using the Typeahead, but
it is also recommended to check the Subtopic Picker afterwards, to see if there is
anything you may have missed.

In the tool, underneath where it says, ‘Please pick the subtopics that apply and page
next to the end’, you will find the Subtopic Picker. You’ll see two tooltips when you
hover over each subtopic. If you hover directly over the information icon, you’ll see
the definition of the subtopic. Review this right before selecting to make sure it is
the right subtopic, paying special attention to the “Excludes” categories listed. If you
hover around the icon, you’ll see a list of child subtopics for that subtopic. This can
provide some additional clarity on that particular subtopic. Remember that if a post
is related to the subtopic, you should select it even if none of the subtopics under it
apply.

Topics are organized in different colored sections for easier navigation. General
subtopics show up first, followed up by groupings of subtopics. For groupings of
entities (i.e. specific people, organizations, etc.) which often have many subtopics,
the tool will only show the first 5 based on popularity. Click the line that says “Click
to expand ## more searchable topics”.

1. You should carefully review the available subtopics here and click any that
apply based on your review of the content.

2. Once you do so, click Page Next and you will be provided a list of deeper
subtopics (where available), from the subtopics you selected.

3. You should continue to click ‘Page Next’ and select any relevant subtopics
from the list of deeper subtopics available until you exhaust the available
lists. At this point you will see the message below. Any subtopics you have
selected will be shown in the ‘Selected’ row.

Important: If the post is not directly related to any of the listed subtopics but is still
appropriate for the queue, you should select ‘None of the above’, and click on ‘Page
Next’:
5. Check and Submit
After selecting all the subtopics that are related to a post from the Typeahead and
Subtopic Picker (or ‘None of the above’ if no subtopic applies), take a moment to
review all of the subtopics you have selected. If there are any that you no longer feel
should be selected, you can click on the “X” button to remove them.
• NOTE: Removing a subtopic will also remove any of its children. When you
find a situation where it seems like the subtopic should apply without the
parent topic, please use your best judgement. If you think there is an error in
the taxonomy, please make note of it in the comments section.
Once you are confident that you have selected all relevant labels, you can press the
Submit button at the bottom of the screen to move to the next Post.

6. Skip Content
There are cases which require that you skip the rendered job. Skip jobs that include
Sensitive Content and immediately escalate those defined in the Immediate
Escalation section below.
• The tool doesn't load → Click 'Skip : Tool Error'
• The post is neither in English nor is it in the language of the queue →
Click'Skip : Other Language'
• You encounter a different problem that prevents you from labeling → Type a
brief explanation in the comment box, then click 'Skip : Other’
• The post is not about the high level topic of the queue → Click ‘Skip : Wrong
Queue’
o Note that sometimes a post will belong to the right queue, but there
won’t be any additional subtopics that apply. In this case, do not use
the ‘Skip : Wrong Queue’ option, but rather the ‘None of the Above’
checkbox

Sensitive Content
“Sensitive” is defined as containing or potentially containing any of the following
content:
 • Child Exploitation and/or Child Nudity
• Self-Injury and Suicidal Content
• Credible Threats, Violence or Calls to Violence
• Adult Sexual Content and/or Nudity
• Hate Speech
• Acts of Terrorism
• Human Trafficking
• Bullying and Harassment
For these types of content, select 'Skip: Other', and indicate “sensitive” as the reason
in the comment box.

Escalation Protocol
1. Immediate Escalation: If the content contains imagery or text/voice
indicating or soliciting Child Exploitation, Child Nudity or Human
Trafficking, escalate the Job ID immediately to your manager. *Vendor
managers should use the CO Contact Form
(https://www.intern.facebook.com/help/contact/481785358641261) to
route job ids into CO Safety queues(CEI should never be screenshot or
replicated in any way as it only further exacerbates the issue. Please always
look to use task/job number for issue identification.)*
a. Child Exploitive Imagery (CEI
(https://our.internmc.facebook.com/intern/wiki/CEI_Policy_Overvie
w/)) refers to imagery (images, videos) depicting the sexual
exploitation of a child.
b. Human Trafficking includes the recruiting, transporting, or harboring
of people by means of threat, coercion, or fraud for the purpose of
exploitation. That exploitation can come in many different forms,
including sexual exploitation, forced slavery, slavery, servitude, or the
removal of organs.”

Key Rating Concepts

Relevance

If it would make sense to see the post in a Facebook feed about the subtopic, then
the post is related to the subptopic and you should select it. However, it is not
enough for a post to merely mention a topic. Some posts explicitly mention a topic,
but do not address the topic such that it would make sense to show in a search for
that topic. Conversely, a post can be strongly related to a topic without calling it by
the same name. Consider the following examples -

• Post is a news article of a boy winning a school competition. There is photo of

the boy and his family and dog in front of their house with the caption:
 “Competition winner, pictured with his parents, sister and dog” → do NOT tag
‘Dogs’.
o While a dog is shown and mentioned, the post is not about dogs and
this would not make sense to show in a feed about dogs.
• Post is a meme about “Man's best friend” → Tag ‘Dogs’.
o While the word “dog” does not appear, the phrase “Man's best friend”
refers to dogs.
Remember to Tag ALL relevant subtopics that apply. Pay particular attention to
posts that are about one thing that is related to multiple topics.
• Post about someone using a new fitness routine to improve a health
condition→ Tag 'Health & Medical Treatments' + ‘Gym Workouts and Fitness
Activities’
• Post provides reviews of the best restaurant three friends found while
traveling through Italy → Tag ‘Restaurants and Dining’ + ‘Food Tourism’ +
'Travel Advice’ + 'Friendships’ + ‘Food Awards & Reviews’ + ‘Restaurant
Reviews & Ratings’

Granularity

In Round 2, some posts can be related to a very large number of subtopics. It is

sometimes tricky to know how ‘granular’ to be when selecting subtopics (i.e. how
deep or thorough should you be when a post has dozens of potentially relevant
subtopics).

Generally, you should strive to be as thorough as you can be and select all subtopics
which meet the relevance definition above. However, there may be cases where a
post relates primarily to a broader subtopic and it is not necessary to tag all the
deeper subtopics which are mentioned. Consider the difference between the
following examples -

• Post is photo with a bunch of different types of dogs → Tag 'Dogs' but do NOT
tag each dog type
o The post is not about the individual dog types, but rather dogs in
general so only the broad category 'Dogs' should be tagged.
• Post is photo of bunch of dogs with a list of the 10 best types of dogs → Tag
'Dogs' and tag all 10 specific dog types
o The post is about specific types of dogs, so each dog type should be
tagged.

If the post links to an article, video, or slideshow that is long and includes a lot of
subtopics, scan to see if there are topics it focuses on most. If so, select those. If not,
select the parent category and anything specifically mentioned or depicted in post
itself.
• Post is a 22 minute makeup tutorial. You click at 5 places throughout and all
5 are talking about eye makeup → select 'Eye Makeup'
 • Post is a 22 minute makeup tutorial. You click at 5 places throughout and all
5 are talking about a different kind of makeup → select “Makeup' but do NOT
select each makeup type
• Post is a 22 minute makeup tutorial covering many types of makeup, but the
caption of the post says “Eye makeup and other beauty tips” and the cover
image is of someone applying eye shadow → Tag 'Eye Makeup'

A few tips to keep in mind about the following subtopics:

• Music Roles [Music & Audio]: We would use these labels if the post were
about the roles in an industry perspective.
o Post is about Beyonce. We would not label ‘Singer’ unless it’s about
Beyonce and her experience in the profession
• Photography [Visual Arts, Architecture & Crafts]: We would use this label if
the post were about the art and profession of photography
o This would exclude photos that are shared for a purpose other than
appreciating the photo itself

Quality

The quality of a post does NOT impact whether a post is relevant to a subtopic or
not. You should always tag all of the subtopics the post is about without regard for
quality, professionalism, or the size of the audience for the post. For example–
• Post is a professional video of a Beyonce concert → Tag 'Musical
Performances’ (Plus any additional subtopics that apply)
• Post is a cellphone video of a band performing in a bar → Tag 'Musical
Performances’ (Plus any additional subtopics that apply)
• Post is a cellphone video of a 3-year-old singing in the car → Tag 'Musical
Performances’ (Plus any additional subtopics that apply)
• Post is a meme about a concert → Tag 'Musical Performances’ (Plus any
additional subtopics that apply)

Entities

Entities are unique people, places or things. You should pay special attention to
whether the post is related to any entities. If the post is about an entity, and that
entity is strongly related to a topic, then you should assign the post to that topic. For
example -
• Post is about McDonald's → Tag ‘Restaurants & Dining’, ‘Fast Food’ &
McDonald’s
• Post is about The Beatles → Tag ‘British Music’, ‘60’s Rock Music’, ‘Rock
Music’, ‘European Music’ & ‘The Beatles’
• Post is about Beyonce → Tag ‘R&B’, ‘Beyonce’
o Do NOT use the label ‘Singer’ unless the content is specifically about
the profession of singing
However, note that just because an entity is strongly associated with a topic, doesn’t
mean the post is about the topic.
Focus on the content itself rather than the author.
• A former president writes a post saying the Chicago Bulls are the best team
ever → This is NOT about ‘United States Government’ because even though
the author was a president, this post is expressing their opinion on a non-
political matter, the Chicago Bulls
• A former president write a post about other American political figures → This
IS about ‘United States Government’ because the post is expressing the
president’s opinions on political figures (not just because they used to be a
president).
Additionally, do not assume that every post which mentions an entity is about that
entity. For example, a video about the Olympics might show the logo of its sponsor,
McDonald's, but that isn't enough to consider the post related to McDonald's.

Research Tips
Side Searches

If there are names or phrases in a post that you do not understand, you should look
them up (i.e. the name of a famous person, company/organization or concept). You
can also look up the author of the post for more context (if relevant to the content of
the post) – they may be well known for writing about a particular topic.

Below the post there is sometimes a section called “Google Links”. The links
provided will open a new tab to Google search results for the specific term. These
can be a great shortcut, but keep in mind the links are auto generated based on
phrases in the post. Use your judgement to determine if they are relevant.

Example 1
Post is by CIZE. It may not be initially clear to someone unfamiliar with this workout
what it is. A quick side search shows that all the results relate to this dance workout
– see the repetition of these key phrases in the search results.
 • Dance and Gym Workouts & Fitness Activities can be selected from the
Subtopic Picker or Typeahead. As it is a form of rhythmic cardio fitness,
Aerobics would also be applicable, as would Fitness Workout Plans, since Cize
is a workout plan you can purchase.

Example 2
This post features ‘Wild Horses’ at the Wilbur. For those unfamiliar, it may be
unclear who Wild Horses are. It would be easy to incorrectly guess that they might
be a music group and tag as such. As we are unsure, we should conduct a side search
to check.
A side search for just ‘Wild Horses’ is too broad and shows many results that aren’t a
match for this group. We can make the search more specific to this post by adding
another key term from the post, ‘Wilbur’.
Doing so brings up many results indicating that they are an improv comedy team
who and that this performance is part of a comedy festival in Boston.

• Comedy and Improv Comedy can be selected from the Subtopic Picker or
Typeahead. As this is also a type of festival, a search for Festival in the
 Typeahead shows that Performing Arts Festivals would be an appropriate
topic. Vacations & Leisure Activities and Festivals & Cultural Events are
automatically selected as well as parent topics.

Example 3
• The post features a long video (over 1 hour) of a religious service. It may not
be immediately obvious what religious denomination the service is for.
However, we are given the name of the minister – and a quick side
search gives results on Wikipedia and his official website that confirms he is
a pastor for a Pentecostal Christian church, a type of evangelical
Protestantism.

• Wikipedia Entry – Note the references to Pentecostal Christianity and

Evangelical Christianity.
• Pentecostalism and Evangelical Protestantism can be selected from the
Typeahead, which also tags the parent topics, Christianity and Protestantism.
As this is a religious service on a spiritual channel, Prayers and Blessings and
Religion & Spirituality TV & Movies also apply.

Feedback

You feedback is valuable to this project and can directly lead to improvement of the
topic taxonomy. Use the comment section to flag where you think the taxonomy or
definitions can improve using these tags (i.e. include the tag, followed by your
reasoning/comment).
#FIT1 : For suggestions on new topics that should be added, topics that should be in
a different location, etc.

#DEF1 : For subtopics with confusing or have unclear definitions (please include
what in particular is confusing)

#TRICKY1 : For posts which are very tricky to label, and you are unsure how to tag
them after spending time researching and reviewing.