Scribd
Upload
58 views1 page

Science 2015 Turing

Beyond the Turing Test researchers are developing a new set of challenges to more comprehensively test machine intelligence beyond just conversational abilities. The proposed challenges will look at abilities like commonsense reasoning, visual comprehension, physical skills integrated with language, and teamwork. By debating and testing these new types of challenges, researchers hope to advance AI and replace the outdated Turing Test with a battery of benchmarks that better evaluate the multiple dimensions of intelligence.

Uploaded by

Pedro Elias Romero Nieto
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
58 views1 page

Science 2015 Turing

Beyond the Turing Test researchers are developing a new set of challenges to more comprehensively test machine intelligence beyond just conversational abilities. The proposed challenges will look at abilities like commonsense reasoning, visual comprehension, physical skills integrated with language, and teamwork. By debating and testing these new types of challenges, researchers hope to advance AI and replace the outdated Turing Test with a battery of benchmarks that better evaluate the multiple dimensions of intelligence.

Uploaded by

Pedro Elias Romero Nieto
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 1

Beyond the Turing Test

Concluding that there is no one test for machine intelligence,


AI researchers develop a battery of research challenges
By Jia You

would motivate researchers to develop machines with a deeper understanding of the


world, argues Marcus, who is co-organizing
the workshop.
One set of proposed challenges focuses
on common-sense reasoning, which remains a tall order for machines yet is crucial for comprehending language. Take the
sentence, The trophy would not fit in the

s the movie The Imitation Game celebrates British mathematician Alan


Turings contributions to the Allied
victory in World War II, the artificial
intelligence (AI) community is rethinking another of his legacies: the
Turing Test.
In 1950, Turing laid out an appealingly simple test for whether a
machine possesses human-level intelligence: Will a person conversing with
it in text mistake it for another human
being? But more than 60 years later,
its time for new criteria, says computer scientist Leora Morgenstern of
Leidos Inc. in Reston, Virginia. We
now know a lot about AI and whats
needed to make progress. Its a big
leap from Turings time.
At a 25 January workshop at the
29th Association for the Advancement
of Artificial Intelligence conference
in Austin, Morgenstern and other researchers will discuss proposals for
a new Turing Championship. In contrast with Turings single litmus test,
the proposed challenges acknowledge that intelligence has multiple
dimensionsfrom language comprehension to social awarenessthat are
best tackled piece by piece.
Over the years, Turings original idea has grown into a small
industry while drawing increasing critAlan Turing, 1950
icism. Competitions such as the longrunning Loebner Prize ask human
judges to text chat with either a person or
brown suitcase because it was too big. Dea computer program for less than 30 minducing that it refers to the trophy, not the
utes and then determine the conversers
suitcase, requires general knowledge that is
identity. In June, a computer program
second nature for a person but difficult to
named Eugene Goostman, which adopts
program into a machine. Next fall, in what
the persona of a 13-year-old Ukrainian boy,
could be the first of the new Turing chalwas declared to have passed a Turing Test
lenges, the industry-sponsored Winograd
organized by the University of Reading in
Schema Challenge will test machines comthe United Kingdom after fooling a third of
prehension of such grammatically ambiguthe judges in 5-minute conversations. Yet
ous sentences.
researchers such as cognitive scientist Gary
A second set of proposed challenges cenMarcus of New York University in New York
ters on machine vision. With new machineCity argue that such competitions put a
learning techniques that train computers to
premium on stock answers and other ruses.
discern objects, researchers at places such
Its a parlor trick, Marcus says. Theres no
as Google and Facebook are developing alsense in which that program is genuinely
gorithms that can guide a self-driving car
intelligent. The new Turing Championship
or automatically identify any face in any

We can only see a short distance


ahead, but we can see plenty there
that needs to be done.

116

photograph. But AI researchers want machines to understand and reason with what
they see, says computer scientist Fei-Fei Li
of Stanford University in Palo Alto, California. The challenge Li will propose would ask
machines to tell stories from picturesnot
only identifying an object such as a coffee
mug, for example, but also noting that it
sits half-empty on a table because someone
drank from it. Such machines might one day
interpret what she calls the dark matter of
the digital age: images and videos, which
todays search engines and bots can hardly
make sense of.
For machines to truly assist people in their
daily lives, physical movement smoothly integrated with language and perceptual skills
has to be part of the mix, says computer
scientist Charles Ortiz of the Nuance
Natural Language and AI Laboratory
in Sunnyvale, California. His proposed
challenge would ask both a machine
and a human to manipulate a robotic
arm in order to, say, play with a toy.
At the same time, they would carry on
a conversation about their actions. As
in Turings original test, a judge would
evaluate the humanness of the computers performance.
Intelligence has one more dimension, says computer scientist Barbara Grosz of Harvard University:
teamwork. To effectively collaborate
with humans, machines will need to
understand their teammates preferences, share information appropriately,
and handle uncertain environments.
Groszs challenge would pair computers with people in group activities, such
as formulating health care plans, to
test whether people overlook that their
partners arent human.
Many more research challenges will
be debated at the workshop, aimed at
capabilities from long-term learning
to creativity. The goal, Marcus says,
is to winnow the proposals down to
three to five competitions. A balance of ambition and realism is key, says computer scientist Stuart Shieber of Harvard. You want
to design competitions that are qualitatively
beyond the current level of AI, but not so far
that it would be like setting an X prize for
space flight in da Vincis era, he says.
Although its unlikely that consensus will
emerge in January, the discussion will continue at another AI conference in July, says
co-organizer Manuela Veloso of Carnegie
Mellon University in Pittsburgh, Pennsylvania. By early 2016, the organizers hope to
stage a set of trial competitions that will be
revised and repeated regularly. If we dont
move fast, it wont happen, Veloso says.
People will lose momentum.
sciencemag.org SCIENCE

9 JANUARY 2015 VOL 347 ISSUE 6218

Published by AAAS

PHOTO: FINE ART IMAGES/HERITAGE IMAGES/GETTY IMAGES

COMPUTER SCIENCE

Downloaded from www.sciencemag.org on January 9, 2015

NEWS | I N D E P T H

You might also like