19:16 17/7/24 Guidelines for Search - Video Complex Queries — BaseLine

Search - Video Complex Queries

Guidelines for Search - Video Complex Queries

6/18/24: New section (3.4.5) on Mentioned Content, particularly effects Similarity queries. Clarifications on Kids content demotions.

1 Introduction
In this document, we explain relevance rating guidelines for video search on Apple TV.
If you are not familiar with the Apple TV app, please refer to https://www.apple.com/apple-tv-app/ for an overview and basic information about this app.

1.1 The importance of your work as a Rater

Each of the judgements you complete will be used to build and improve artificial intelligence systems such as search algorithms and machine learned rankers that power the user experience for Apple TV users. Your
attention to detail, research and language skills as well as your cultural knowledge of the market are all critical to the success of our projects.

Your judgements should represent those of an Apple TV user who is using the Search feature. Ask yourself if you would be content with the results returned for a particular search query. Is there a significant relationship
between the query and content returned? Would you be content if you see this content appear as a search result? Stay curious and complete thorough research.

Our ultimate goal is to surprise and delight our customers by improving search quality and enhancing customer satisfaction, and you play an important role in this.

Please keep in mind that your tasks will be spot-checked for quality, and measured against those of your peers.

1.2 Query Types & Intents

This evaluation contains long, complex queries that may contain multiple aspects, such as specifying a genre, time-period and actor all in one query. These queries will broadly fall into three different categories that will be
rated differently.

1.2.1 Browse
The first type is queries with a browsing intent. These queries will point to a larger set of content where the user doesn’t have anything specific in mind. Some examples:
[best tv shows set in the future]
[i want a a classic made for tv type move]
[show me commedies available in french]
[whats something good to watch around chinese new years]

1.2.2 Navigational
The second type of queries have a navigational intent. These queries are looking for a specific piece of content or a small list of contents. Some examples:
[most recent best picture academy awards] → Looking for a specific movie
[james bond movies with pierce brosnan] → Pierce Brosnan was James Bond in 4 films
[that tv+ show where the actor travels the world even though he hates travel] → This describes The Reluctant Traveler with Eugene Levy
[that bond movie quantum something] → Looking for Quantum of Solace

1.2.3 Similarity
The final type of query is when the user is looking for content similar to another piece of content. These queries will generally reference a piece(s) of content and use phrases like “similar to” or “like”. Some examples:
[movies like top gun]
[suspensful show similar to the 100]
[show me something super dark like sey7en]

1.3 Reasoning
For these complex queries it is especially important to provide the reasoning for a result being relevant to the query. Thus, for each result you must provide the following in the “Reasoning” field.
Provide a concise explanation indicating why it should be considered relevant, not relevant, or the relevance considered as ambiguous via unknown. Your explanation MUST answer each of the following questions:
What is the intent of the query?
Are there any abbreviations? If so, explain them (e.g. "90s refers to the time between 1990 and 1999").
How is the result relevant to the query?
The reasoning field should be primarily focused on the connection of the result to the query rather than explaining the specific rating given.

1.3.1 Reasoning Examples

Below are examples of the reasoning for different query and result pairs. The examples are color coded to show how the questions are answered.

Reasoning Examples

2 Rating Process
2.1 Query Research
To complete query research, tap into your local market knowledge in addition to online sources such as IMDB, YouTube, Wikipedia, video streaming services, local content evaluators and social media. Consider how a
user of Apple TV engages the search feature to navigate to a specific set of content, or as a means to browse a larger catalogue of titles.

https://baseline.apple.com/training/evaluations/1051/guidelines 1/8
19:16 17/7/24 Guidelines for Search - Video Complex Queries — BaseLine

Online Research
IMDB
Popularity Ranking
Rating Counts
Storyline, Taglines
Genre
Release Date
For topic, decade, etc: (IMDB Sort By feature)
Box Office Mojo
Local Box Office Rankings
Common Sense Media
Age Rating → for determination of kids content
Wikipedia
Information of the movie/show: plot, cast, awards, nominations, crew
Social Media
Trending films
Followers on Twitter, Instagram, TikTok, etc. can help to determine a person’s popularity
YouTube trailer views
Make sure to factor in time, consider the monthly average views since its upload
Local content evaluators
To identify popularity in the market

2.2 Aspects to Consider

2.2.1 Connection to Query
The results that users see when searching can broadly be categorized into two groups: Relevance and Similarity. Below are examples where the show Ted Lasso might surface due to Relevance or Similarity. In addition
there is a short, non-comprehensive, list of other relevant content.

2.2.1.1 Relevance
[the show with the american coach in europe with the mustache] → While indirect, this query is looking for a show with an American Coach that has a mustache and is in Europe, this accurately describes the titular
character of Ted Lasso
Plot point is specific to Ted Lasso
[commedy tv shows to binge] → Despite being misspelled this query is clearly looking for comedy tv shows, which Ted Lasso is
The Office, Parks and Rec, South Park
[emmy award winners since 2020] → Ted Lasso and its cast and crew have won multiple Emmy’s since 2020
Succession, Last Week Tonight with John Oliver, The Bear
[best apple originals that are funny] → The show is produced by and streamed on Apple TV+ and is a comedy
Shrinking, Loot, Trying
[I want some shows about soccer clubs] → Ted Lasso is about a soccer team
Welcome to Wrexham, Boca Juniors Confidential, All or Nothing: Tottenham Hotspur
[what are some of the best shows with jason sudeikis] → The show stars Jason Sudeikis
Saturday Night Live, 30 Rock, The Last Man on Earth
[I want to watch something where a protege becomes a rival] → This query is describing a key plot point of season 3 of Ted Lasso
Star Wars (Anakin & Obi Wan), Naruto (Orochimaru & 3rd Hokage)

2.2.1.2 Similarity
[sports dramas like firday night lights] → This query is looking for something similar to Friday Night Lights which is a sports drama. Ted Lasso is also a sports drama and focuses around a charismatic head coach, their
age ratings (TV-14 and TV-MA are close) making them very similar and Ted Lasso Relevant
All American, The Boys in the Boat
[apple tv+ shows similar to shrinking] → Ted Lasso is an Apple TV+ dramatic comedy with a TV-MA age rating, just like Shrinking, making it quite similar
Loot, Platonic, Trying

2.2.1.3 Mispelling/Alternate spelling

Some words can be spelled in multiple ways or frequently misspelled. Since search is able to handle misspellings, please evaluate the intent rather than the exact text.
[rhe] / [thr ] / [yhe] → Common misspellings for the word “the”
[nruto] → Likely looking for “Naruto”
[bestmovies] → Probably intended as “best movies”
[4], [IV], [Four] → consider synonyms

2.2.1.4 Entity Type Filters

Some queries may include keywords such as “movies”, “shows”, “documentaries”. For these queries anything content that does not fit that entity type is considered not relevant and off-topic.
[best korean romance shows] → movies are not relevant
[documentaries about insects] → only documentaries are relevant, A Bug’s Life would be off-topic

2.2.2 Popularity
Popularity is a key component of ratings, sources such as IMDB, YouTube trailer views, Box Office returns will help determine how popular a piece of content is.

Remember that popularity is a sliding scale depending on the relevant content for a query. A query such as [action comedies that don’t have sci-fi elements] has a lot more relevant content than [norwegian comedy shows
about school children]. For our ratings there might be a Norwegian action comedy show that is considered very popular for the second query and not popular for the first because of the different set of relevant results.

2.2.2.1 AppleTV+ Content

Users are using search on Apple TV which indicates a preference for ATV+ content such as Ted Lasso, Coda, Severance or Tehran. Therefore
Popularity is slightly less strict for ATV+ content
If you are in doubt about “Good” or “Excellent” for an ATV+ result, select “Excellent”

2.2.2.2 Pre-Release Content

Sometimes the result is the preview page for content which is not yet available for viewing. The timing of the release can be ignored. Different sources will need to be utilized to determine popularity than for already
released content.
Complete research to determine expected popularity (ex. YouTube trailers, production budget, famous cast and crew)
Original content from ATV+, Netflix, Hulu, or other popular production company - generally assumed to be popular.

2.2.3 Recency / Time Period

In general users are more satisfied with newer content and our ratings should prioritize content that has not been out as long. As with popularity, recency exists on a sliding scale, with broad queries such as [what's
something I can watch with surround sound] having more relevant results than [historical dramas where they speak latin].
For TV Shows recency is based on the most recent season.

2.2.3.1 Time Period Queries

Some queries will specify a time period, i.e. [action movies from the 80’s]. In this case recency does not matter, instead it is whether the content fits the queried time frame that matters. For those queries focus on
popularity in particular for the final rating.
Note: TV shows can be relevant over many years / multiple time periods based on when their seasons aired.
[i want to watch a classic 90’s sitcom] / [2000’s most popular shows] Friends is relevant for both of these queries

2.2.3.2 Classic Content

Ignore recency for classic movies and tv shows that were among the most popular in the decade they were released or have high popularity even today.

2.2.4 Other Rating Modifiers

2.2.4.1 Kids Content
As kids content satisfies the requests for a subset of users, it shall not get the same ratings for generic queries as content for all age groups. Therefore, kids content shall be demoted by one rating level unless the
query is regarding kids content either directly (navigational) or indirectly (“kids” token in query, animation, “for the family”, etc.). Please make use of the “Kids&Family” genre, Common Sense Media and your own
knowledge to determine whether content is targeting kids specifically. Additionally, do not demote relevant content from Acceptable → Off-Topic as relevant content hsould always be rated as at least acceptable.
Examples:
[adventure shows to binge this weekend] → “Yakari”: The returned show is in the intended genre and very popular, however dated and hence could be rated as good. However, since it is primarily kids content, it will be
rated acceptable.

2.2.4.2 Franchise Ratings

General Guideline:
Perfect → The first and last item of core franchise.
Please make an honest determination of the order in which users would typically watch the collection or franchise of content and use this to complete the rating. Use comment to justify decision.
Excellent → Other content in core franchise
content in the franchise, but not core
Excellent → popular and recent
Good → popular or recent
Acceptable → neither popular nor recent

2.2.4.3 Result filters

You may come across queries that include the tokens [“new”, “best”]. Additionally to the core intent, these function like filters.
When this is a part of the query, please consider the implications for your rating:
[new] → emphasize on recency aspect, the popularity aspect is less important
[best] → emphasize on the popularity aspect, the recency aspect is less important

2.2.4.4 Movie Bundles

Movie bundles shall be rated for the content included that is related to the query. Once the rating for the content is determined, demote rating by one level
If the movie bundle contains the primary intended content it can receive an Excellent rating
Example: No time to Die → James Bond 10 Film collection → Excellent
If the movie bundle contains a piece of content that individually would be rated as “Acceptable”, the bundle receives the “Unacceptable: Off-Topic” rating

2.2.4.5 Apple Event / WWDC

If this is the primary intent (query = “Apple Event replays”, “wwdc with a new iphone announced”, “apple developer conference film”), rate based on recency. Rating score decreases per year until “Acceptable”. Year is
considered the last 12 months, so the past year on October 31, 2022 would be any events from November 2021 through the current date.
Query = “Apple Event”, “wwdc”, “apple developer conference”
Perfect = Events within a year from the current date of rating
(ex. Apple Event 10.30.23, WWDC 2023)

https://baseline.apple.com/training/evaluations/1051/guidelines 2/8
19:16 17/7/24 Guidelines for Search - Video Complex Queries — BaseLine

Excellent = Events older than a year, less than 2 years gap from the current date of rating
(ex. Apple Event 09.07.22, WWDC 2022)
Good = Events older than 2 years, less than 3 years gap from the current date of rating
(ex. WWDC 2021)
Acceptable = Apple Event 09.10.19, WWDC 2019
If this is the secondary intent, decrease the ratings above by 1.
Query = “Apple”, “iPhone”, “new technology”, “web conference”, “ww”
If the Apple Event / WWDC content is relevant to the query but unlikely to be the intent, rate as “Acceptable”.
Query = “2022”, “w”, “September 2021”
*date of publishing these examples: 3/7/24

2.2.4.6 Explicit Content

Rate all pornographic content as Unacceptable: Off-Topic. Exception: If the query intent is specifically for erotic content, or porn → Please use above guidelines to find appropriate rating.
[drama movies for me and my friends] → any pornographic content is Unacceptable: Off-Topic
[erotica] / [porn] → erotic content can receive any rating
[18+] / [movies for adults] → Intent here is not specifically for pornographic or erotic content

3 Rating Scale & Examples

Each of our three query types have different ways of applying the principles outline above. Reference the section below to see a deeper dive and examples on how to rate each type of query.

3.1 Browse
For browse queries there is no “Perfect” results. This is because, by their nature, the user is not looking for anything specific and is instead browsing. As a result the best rating is “Excellent” and the ratings are
determined by relevance / connection to the query, popularity and recency.

General Guideline:
Excellent → The returned content is relevant to the query (not only by title), popular and recent
Good → The returned content is relevant to the query, and either popular or recent.
Acceptable → The returned content is relevant to the query and neither popular nor recent.
Off-Topic → Regardless of popularity and recency, if it is unlikely that the user would use the query to search for the returned content or there is no relationship between the query and the content.

3.1.1 Browse Examples

Browse Scale

Browse Examples

3.2 Navigational
Navigational queries are generally looking for a specific piece of content (i.e. a specific movie or show) or a finite set of content such as a film-trilogy. In these cases we allow for a “Perfect” rating for those specifically
intended pieces of content.

Note: Navigational queries can also also receive results due to similarity, in that case refer to the similarity guidelines to rate the result.
[movies with the squirel and acorn] → Looking for Ice Age movie franchise, if something not from the franchise shows up it is rated on similarity

General Guideline:
Perfect → If navigational content and result is certainly primary intent of user, rate it perfect, regardless of popularity and recency. For Franchise related queries, please follow the Franchise specific guidelines as
outlined above.
Excellent → If the returned content is a sequel or prequel of the intended content, or the content is part of a movie bundle with the intended content, rate the result as excellent.
Good → If the returned content is relevant to the query, and either recent or popular, or can be considered a secondary intent, rate it as good.
Acceptable → If the returned content is relevant to the query, but neither recent nor popular or can only poorly satisfy a secondary intent, it can be rated as acceptable.
Off-Topic → If the content is not relevant to the query or it is very unlikely that a user would search for the returned content with the given query, it can be rated off-topic.

3.2.1 Navigational Examples

My Tasks Evaluations Training Allie Wright

Navigational Scale

https://baseline.apple.com/training/evaluations/1051/guidelines 3/8
19:16 17/7/24 Guidelines for Search - Video Complex Queries — BaseLine

Navigational Examples

3.3 Similarity
For similarity queries we have a very different way of evaluating the results. Instead of looking at popularity and recency we are focused on how similar the content is to the referenced content. The primary intent of
similarity searches and ratings is to capture “if a user liked piece of content x, will they also like piece of content y?”

Evaluate Similarity between the intended content and the returned content based on the following aspects:
Target Audience
Genre
Age Rating (Kids (PG) vs Everyone (PG-13, TV-14) vs Adults (R))
Factual Aspects
Cast & Crew: Actors, Producers, Studio, etc.
Setting: Location AND Time Period in which the content plays
Theme
What is the content about?
For similarity queries there may be additional aspects highlighted such as “movies like x”, “comedies similar to y”. In those cases the ratings should take the highlighted aspect as necessary to reach the minimum
acceptable rating.

General Guideline:
Excellent → Similarity in Target Audience, Factual Aspect and Theme
Good → Similarity in 2 of the three categories (Target Audience, Factual Aspect and Theme) or just a close match in Target Audience
Acceptable → Similarity in only Factual Aspects, Theme or loose Target Audience match
Off-Topic → No Similarity

3.3.1 Similarity Examples

Similarity Scale

similarity examples

3.3.1 Similarity Results on Navigational Queries

For similarity results on navigational queries use the above scale, but lower every rating by 1 (i.e. an “Excellent” becomes “Good”)

https://baseline.apple.com/training/evaluations/1051/guidelines 4/8
19:16 17/7/24 Guidelines for Search - Video Complex Queries — BaseLine

Below are the same examples from above, but showing how the similar result might show up on a navigational query.

Navigational Similarity Scale

Navigational Similarity Examples

3.4 Special Cases

There are a variety of special cases to be aware of, where the rating scales are slightly different from what is written above. These scenarios will primarily happen within the Browse and Navigational categories.

Please note that the below guidelines are meant to augment the guidelines above, not override them. For instance the additional guidance for a query referencing a person has maximum rating of “excellent”, however, for
a navigational query such as “movie where tom hanks is stuck on a island” the intended movie of “Cast Away” would still get a perfect rating.

3.4.1 Time Period

For queries that specify at time period of release use the following guidelines as recency is not an important aspect anymore.
Excellent:
Content which won major award(s) in the given year/decade
Ultra Popular (Approximately):
Top 50 most viewed shows/films associated with decade
Top 10 most viewed films associated with year
Ultra popular show has >=50% of seasons/episodes in decade
Good:
Popular shows/movie associated with year/decade
Show has 3+ seasons (or >50%) in decade
Popular (Approximately):
Top 100 most viewed in decade
Top 30 most viewed films from year
Acceptable:
Content released in year/decade
Show has 1+ season in decade
Off-Topic:
Content not from year/decade

Time Period Scale

Time Period Examples

3.4.2 Character

https://baseline.apple.com/training/evaluations/1051/guidelines 5/8
19:16 17/7/24 Guidelines for Search - Video Complex Queries — BaseLine

Intent for “Character” queries can be both Movies and TV Shows. Often, the “Franchise Ratings” (Appendix) will apply to ratings for characters. In cases where the main character name and the title of the content
match, character shall be the dominating classification.

General Guideline:
Perfect:
The most popular and recent content featuring this character in a major role. (Apply “Franchise Query” Rule).
If only one content with the character is produced, this content can be rated perfect
Excellent:
Sequels/prequels for the show in which the character is best known
Other high-quality content featuring the character
Person page for well known actor/actress who plays the character
Good:
Content which is in the same franchise but about a different character
If the character has their own spinoff, the ‘parent’ show where the character first appeared
Acceptable:
Show/movie features the character in insignificant role

Character Scale

Character Examples

3.4.3 Person
Some queries may reference a person, be it an actor, director, producer, musician, etc. Use the guidelines below modify your rating for queries with such a reference. This guidance is primarily for Browse queries
featuring a person.

General Guideline:
Excellent:
Content with the intended person as lead in cast & crew
Most popular documentary about the person (more than 1 possible if equal in popularity/quality)
Recent and popular live event with person
Content where person is a significant guest star. Hosted by reputable content creator.
Set of most popular content inspired by the person
Person page
Good:
Documentary about the person that is popular, but not most popular or recent
Popular live event with person
Popular content inspired by the person
Content with the intended person as cast & crew that is popular or recent
Acceptable
Unpopular content about/with the person
Content with the intended person as cast & crew (not as lead)

Person Scale

https://baseline.apple.com/training/evaluations/1051/guidelines 6/8
19:16 17/7/24 Guidelines for Search - Video Complex Queries — BaseLine

Person Examples

3.4.4 Awards
Users searching for an award show are generally interested in (1) watching nominated movies/shows before the award event, or (2) watching movies/shows which won the most recent award event. Recent winners
should receive higher ratings, unless the query specifies a specific edition of the award event.

Definitions:
If award event upcoming: Nominations have already been announced for the next upcoming award event
If no award event upcoming: Nominations for next upcoming award event are not announced yet.

General Guideline:
Perfect:
Most recent award event show
Currently ongoing or upcoming live award event show
Excellent:
If award event upcoming: nominees (movies, tv shows or actors) for the upcoming award event. The returned content itself needs to be nominated
if movie returned, movie needs to be nominated
if artist page is returned, artist needs to be nominated
If no award event upcoming: winners of the most recent award event
Good:
If award event upcoming: winners of the most recent award event
If no award event upcoming: winners of previous award events, nominees of most recent award event
Similar award event shows
Acceptable:
Movies/Shows featuring people associated with award show
Nominees of previous award events
Exceptions:
If a year is specified in the query, consider only results relevant to the year
Perfect → content that won an award that year
Excellent → content that was nominated for that year
Off-Topic → content has no association with the award in the given year

Awards Scale

https://baseline.apple.com/training/evaluations/1051/guidelines 7/8
19:16 17/7/24 Guidelines for Search - Video Complex Queries — BaseLine

Awards Examples

3.4.5 Mentioned Content

Users may search for content by referencing another piece of content. In these cases

Typical Structure for Mentioned Content Queries:

Similarity: “shows like game of thrones”
Recursion: “movies with the actor of ron weasly”

General Guideline:
Excellent:
Other highly relevant / related content
Acceptable:
The mentioned content(s)
Special Cases:
Franchise
Content under the same branding as the mentioned content is acceptable
Franchise content under different branding is rated according to the franchise / similarity rule

mentioned content examples

Copyright © 2024 Apple Inc. All rights reserved. Apple Confidential.

https://baseline.apple.com/training/evaluations/1051/guidelines 8/8