Question Answer

Query: The Rolling Stones Result: https://en.wikipedia.org/wiki/The_Rolling_Stones

Highly Satisfying
If the result does not contain original and useful content, what should you do? Flag as Inappropriate
How should the result in the image be rated for query Ram 1500 2022? Not Satisfying
How should the result in the image be rated for query How tall is Will Smith? Consider whether the knowledge card presents all pertinent
Highly Satisfying
information.
For the query "Friends " How would you rate this result: https://www.imdb.com/title/tt0583484/?ref_=ttep_ep2 Somewhat Satisfying
How should the result in the image be rated for query New York City Subway Branch Broad Street? Not Satisfying
How should the result in the image be rated for query: King Charles? (User in US on 2022-09-14) [2] Satisfying
OPR cannot be rated when just one result is shown FALSE
How should the result in the image be rated for query: flight ua188? User in Newmark on September 19th [4] Not Satisfying
How should the result in the image be rated for query: iPhone 13? (User in US) Satisfying
[1] Highly Satisfying
How should the result in the image be rated for query: samsung?
Query is Portland, USA Result is https://en.wikipedia.org/wiki/Portland,_Oregon [1] Highly Satisfying
When the result is in English, the following checkbox should be marked None
How should this result in the image be rated for query: LinkedIn? [2] Satisfying
How should the result in the image be rated for query: Dates spring break 2023? NS
If you ticked the "Wrong Language" flag, you can still rate the result as Satisfactory. FALSE
In OPR (Overall Preferance Rating), the side with more results is always better FALSE
Query: Barcelona FC Result: https://www.fcbarcelona.es/es/futbol/femenino-a/jugadores/711843/alexia NS
Spice girl members [2] Satisfying
[4] Not Satisfying
Query: how to make a pizza Result: https://www.theinfatuation.com/new-york/guides/best-pizza-nyc
For the query "thanksgiving turkey recipe " (Please assume that an English speaking user issued this query from US) How would [2]
youSatisfying
rate this
If the query is "Adidas" and the result is: https://www.adidas.com, it should be rated as: Highly Satisfying
Not Satisfying
For the query "Boreal Ski Resort " How would you rate this result: https://www.rideboreal.com/culture/blog
Highly Satisfying
How should the result in the image be rated for query: Mahatma Gandhi?
When you are given the user’s location, AND the query explicitly specifes that they are seeking results from a locale that differs from
[4] Not
theirSatisfying
current location, then any results that do n

[3] Somewhat Satisfying

How should the result in the image be rated for query: Birmingham?
[2] Satisfying
Query is Bulgakov Result is The Master and Margarita book
When rating grouped results you should consider each image separately. FALSE
[4] Not Satisfying
Query: best pubs in LA Result: https://www.tripadvisor.com/Restaurants-g60713-zfg11776-San_Francisco_California.html
[4] Not Satisfying
How should the result in the image be rated for query: where are the next Olympics?
The query is a business and the result is a map that show a location with incorrect information. How would you rate?
[4] Not Satisfying
How would you rate a news result for the query "queen elizabeth"? [2] Satisfying
How should the result in the image be rated for query: Disneyland? [4] Not Satisfying
How should the result in the image be rated for query: Empire State? [3] Somewhat Satisfying
How should the result in the image be rated for query Subway in Broad Street New York? [3] Somewhat Satisfying
Query: Skin care routine Result: https://www.nytimes.com/guides/tmagazine/skincare-routine [2] Satisfying
How should the result in the image be rated for query: what is Christmas? [4] Not Satisfying

How would you rate the result in the image for the query: sdj-002 (User in US)? [1] Highly Satisfying

How should this result in the image be rated for query freelance work?
[2] Satisfying
If the result has matching words with the query it will always be Highly Satisfying FALSE
The query is "Cabify" and the result is Uber app, how would you rate it? [3] Somewhat Satisfying
If the query is the name of an app and the result is a variant version (e.g., "Pro" or "Lite") of or sequel to that app, or another complementary
[2] Satisfying
app from the same vendor, how would y
How would you rate the result in the image for the query: Pizza recipe? User is in US. [4] Not Satisfying
If the query is "brad pitt" and result is wikipedia page for Oceans 11, a movie he is in, how would you rate? [2] Satisfying
How should the result in the image be rated for query: McDonalds? User is in San Francisco, US. [4] Not Satisfying
When rating OPR, the side with more results will be the best one. FALSE
Web/News Results should be graded based on the URL itself and NOT the page found when you click on them FALSE
If the result was a medical forum that mentions reproductive organs, this should be flagged as Inappropriate. FALSE
The query is a cafe and the result is a review of the cafe in Yelp. How should this be rated? [2] Satisfying
Query is dungeons and dragons Result: an episode of The Big Bang Theory where the game is being played [4] Not Satisfying
How should the result in the image be rated for query: Bluebottle Cafe? (User in North California, US, 2022) [4] Not Satisfying
Query is Al Aba song Result is an article about Francoist Spain [4] Not Satisfying
How should the result in the image be rated for query: Caffé Etc? (User in Los Angeles) [2] Satisfying
How should the result in the image be rated for query: smelly cat? [1] Highly Satisfying
How should the result in the image be rated for query: Star Wars Memes? [2] Satisfying
In Overall Preference Rating, if one side is empty, always rate in favor of the side with results TRUE
How would you rate the result in the image for the query: Marlin nemo? [2] Satisfying
Query is "current US president", and result is Obama, how do you rate it? [4] Not Satisfying
There is no need to take the user’s location and the date of the query into account when rating. FALSE
How would you rate the result seen in the image for query: how many hours does a koala sleep? [1] Highly Satisfying
How should the result in the image be rated for query "water with lemon, benefits"? [4] Not Satisfying
How should the result in the image be rated for query : Manhattan distance from New Jersey? [1] Highly Satisfying
How should this result in the image be rated for query: search replace regex vim? [1] Highly Satisfying
How should the result in the image be rated for query The Sandman [2] SS
OPR rating, the query is lasagna recipe. Which side would you prefer with the following results? Left Side: 3 sites with lasag About the same
How should the result in the image be rated for query New York City Subway Broad Street? [3] Somewhat Satisfying
Movie streaming sites offering free pirated content are not considered Inappropriate and can be rated as Satisfactory FALSE
How would you rate the result in the image for the query: paris? [2] Satisfying
[2] Satisfying
For the query "war in Ukraine " how would you rate the result: https://edition.cnn.com/europe/live-news/russia-ukraine-war-news-v2-09-15-22/index.html?
How should this result in the image be rated for query: Cervelo Road Bikes? [1] Highly Satisfying
If the result returned for any query does not load, how would you rate it? [4] Not Satisfying
How should the result in the image be rated for query: number of states in US? [4] Not Satisfying
If the query is "Madonna" her latest album would be considered [2] Satisfying
How should this result in the image be rated for query: Monza Grand Prix 2022 Results? [1] Highly Satisfying
How would you rate the result in the image for the query: Justin Bieber's last album (User in US on September 2020)? [4] Not Satisfying
For the query "Spice Girls Members" how would you rate this result: https://en.wikipedia.org/wiki/Spice_Girls [2] Satisfying
The query is "samsung tv", result is web page for Samsung washing machine, how would you rate? [4] Not Satisfying
If the user searched for adult content and the result contained pornographic material, this should not be flagged as Inappropriate because
FALSE
this is what the
How should the result in the image be rated for query: flight status miami aa2424? [1] Highly Satisfying
[1] Highly Satisfying
How should the result in the image be rated for query How tall is Will Smith? Consider whether the knowledge card presents all pertinent information.
Query: burguer (User in Chicago) Result: https://www.busyburger.com/ [2] Satisfying
When rating OPR, the side with more results will be the best one. FALSE
How should the result in the image be rated for query: quick healthy meals? [1] Satisfying
Results for advice or recommendation queries(e.g., "how to lose weight", "chicken parmesan recipe", "best beatles
[1] TRUEsong",
How should the result in the image be rated for query: margarita recipe? [4] Not Satisfying
How should this result in the image be rated for query: freelance work? [1] Satisfying
How should the result in the image be rated for query "collagen"? [1] Highly Satisfying
How should the result in the image be rated for query: Types of yoga? [4] Not Satisfying
Which would be correct for query "apollo project" (The query refers to the space program from the 1960s that first put a human on the moon.)[2] Left: Slightly Better
How should the result in the image be rated for query: Who is Harry Potter? [1] Highly Satisfying
How should the result in the image be rated for query: Spain? [1] Highly Satisfying
How should the result in the image be rated for query The Sandman? [3] Somewhat Satisfying
The query is the series Game of Thrones, and the result is an info card. How would you rate? [4] Not Satisfying
A wikipedia result for a historical event can be rated as Highly Satisfying FALSE
If the query is Messi and the result is https://messi.com/ How would you rate this result? [1] Highly Satisfying
OPR- How would you rate the results in the image? Much Better
For the query "detty outdoors ", how would you rate the result: https://dettyoutdoors.com/? [4] Not Satisfying
How should the result in the image be rated for query: Oasis? [3] Somewhat Satisfying
Query is Levis jeans Result is levis.com [2] Satisfying
How should the result in the image be rated for query: what do spiders eat? [1] Highly Satisfying
How would you rate?Query is Apple and the Result is https://www.apple.com/ [1] Highly Satisfying
Query: The Beatles. How would you rate this result seen in the image? [1] Highly Satisfying
How would you rate the result in the image for the query "the weather today (User is in London) "? [1] Highly Satisfying
As for OPR rating, the query is Netflix. Which side would you prefer with the following results? Left Side: Netflix App, Netflix.com and
[1] Left:
Episode
Much
to Better
Bridgerton series , Right Side: Articl
How should the result in the image be rated for query: yoga flow? User is in India. [4] Not Satisfying
Results for advice or recommendation queries(e.g., "how to lose weight", "chicken parmesan recipe", "best beatles song", "thai restaurant")
[1] TRUE can never be Highly Satisfactory
Query is Old Orchard Beach, ME Result is https://en.wikipedia.org/wiki/Old_Orchard_Beach,_Maine [1] Highly Satisfying
OPR rating, the query is lasagna recipe. Which side would you prefer with the following results? Left Side: 3 sites with lasagna recipes
[1] About
andthe
oneSame
with spaghetti recipe, Right Side: 3 s
How should the result in the image be rated for query: Titanic Cast? [4] Not Satisfying
If most users would be happy with a result, how would you rate it? [1] Highly Satisfying