Transcripción

gpt4 can improve Itself by reflecting on its mistakes and learning from them even
if the world does pause AI development gpt4 will keep getting smarter drawing
upon the stunning reflection paper and three other papers released only in the
last 72 hours I will show you not only how gpt4 is breaking its own records but
also how it's helping AI researchers to develop better models I will also cover
the groundbreaking hugging GPT model which like a centralized brain can draw upon
thousands of other AI models to combine tasks like text the image text the video
and question answering the reflection paper and follow-up sub stack post that
caught Global attention was released only a week ago and yes I did read both but
I also reached out to the lead author Noah shin and discussed their significance
at length others picked up on the results with the legendary Andre carpathy of
Tesla and openai fame saying that this metacognition strategy revealed that we
haven't yet seen the max capacity of gpt4 yet so what exactly was found here is
the headline result I'm going to explain and demonstrate what was tested in a
moment but look how they used gpt4 itself to beat past gpt4 standards using this
reflection technique this isn't any random challenge this is human eval a coding
test designed by the most senior AI researchers just two years ago the designers
included Ilya sutskovar of openai Fame and Dario amade who went on to found
anthropic these are realistic handwritten programming tasks that assess language
comprehension reasoning algorithms and Mathematics so how exactly did gpt4
improve itself and beat its own record because remember in the distant past of
two weeks ago in the gpt4 technical report it scored 67 not 88 well here is an
example from page 9 of the reflection paper as you can read in the caption this
was a Hotpot QA trial designed specifically such that models needed to find
multiple documents and analyze the data in each of them to come up with the
correct answer notice how initially a mistake was made on the left by the model
and then the model at the bottom reflected on how it had gone wrong in a self-
contained Loop it then came up with a better strategy and got it right and the
authors put it like this we hypothesized that llm's large language models possess
an emergent property of self-reflection meaning that earlier models couldn't do
this or couldn't do it as well it's a bit like GPT models are learning how to
learn in case you think it was a model blindly trying again and again until it
was successful no it wasn't this was another challenge called Alf world and look
at the difference between success without reflection and success with the
reflection I discussed this of course with the lead author and the goal was to
distinguish learning curves from self-improvement to simple probabilistic success
over time if you're wondering about Alf World by the way it's about interactively
aligning text and embodied worlds for example in a simulated environment the
model had the task of putting a pan on the dining table and it had to understand
and action that prompt so as you can see this ability to reflect doesn't just
help with coding it helps with a variety of tasks at this point I want to quickly
mention something I know that there will be a couple of well-versed insiders who
say didn't gpt4 actually get 82 percent in human eval in the Sparks of AGI paper
of course I did a video on that paper too and asked the author of reflection
about this point there are a few possibilities such as prompting changes and the
sparked authors having access to the raw gpt4 model but either way it is the
relative performance gain that matters whichever bass line you start with gpt4
can improve on it with a reflection and the 88 figure is not a cap the author has
observed results in the last few hours as high as 91 percent but before I go on I
can't resist showing you the examples I found through experimentation and also
shared with the author take this prompt that I gave gpt4 write a poem in which
every word begins with e now as you can see it did a good job but it didn't fully
get it right look at the word Ascent for example without mentioning anything
specific I just then wrote did the poem meet the assignment not even a
particularly leading question because of course it could have just said yes gpt4
then said apologies it appears the poem I provided did not meet the assignment
requirements not every word begins with the letter e here is a revised poem with
every word beginning with the letter e remember I didn't help it at all and look
at the results every word begins with e how far can we take this for the next
example I chose mathematics and asked write me a five question multiple choice
quiz to test my knowledge of probability with correct answers and explanations at
the bottom there should only be one correct answer per question it comes up with
a D decent quiz but notice a problem in question three for example the
probability of drawing an ace or a king is indeed 8 out of 52 but that simplifies
down to 2 out of 13. so two of the answers are correct and I explicitly asked for
it not to do this in the prompt so can the model self-reflect with mathematics
kind of almost look what happens first I give a vague response saying did the
quiz meet the assignment GPT 4 fumbles this and says yes the quiz did meet the
assignment hmm so I tried did the quiz meet all of the requirements and gbc4 says
yes so I did have to help it a bit and said did the quiz meet the requirement
that there should only be one correct answer per question that was just enough to
get gpt4 to self-reflect properly and it corrected the mistake I must say it
didn't self-correct perfectly notice it identified C and D as being correct and
equivalent when it was B and D but despite making that mistake it was able to
correct the quiz in case you're wondering the original chat TPT or gbt 3.5 can't
self-reflect as well I went back to the perm example and Not only was the poem
generated full of words that didn't begin with e also the self-reflection was
lacking I said did the poem meet the assignment and it said yes the poem meets
the assignment as the lead author Noah Shin put it with gpt4 we are shifting the
accuracy bottleneck from correct syntactic and semantic generation to correct
syntactic and semantic test generation in other words if a model can know how to
test its outputs accurately that might be enough even if its initial Generations
don't work it just needs to be smart enough to know where it went wrong others
are discovering similar breakthroughs this paper from just three days ago comes
up with this self-improvement technique they get gpt4 to frame its dialogue as a
discussion between two agent types A researcher and a decider a bit like a split
personality one identifying crucial problem components and the other one deciding
how to integrate that information here is an example with Gypsy 4's initial
medical care plan being insufficient in crucial regards the model then talks to
itself as a researcher and as a decider and then lo and behold it comes up with a
better final care plan the points in bold were added by gpt4 to its initial care
plan after discussions with itself and the results are incredible Physicians
chose the final summary produced by this dearer dialogue over the initial Gypsy 4
generator summary 90 to 10 that's the dark red versus the Pink I'm colorblind but
even I can see there's a pretty big difference the authors also introduce
hallucinations at different levels low medium and high and they wanted to see
whether this dialogue model would reduce those hallucinations these are different
medical gradings and you can see that pretty much every time it did improve it
quite drama automatically and then there was this paper also released less than
72 hours ago they also get a model to recursively criticize and improve its own
output and find that this process of reflection outperforms Chain of Thought
prompting they tested their model on Mini wob Plus plus which is a challenging
Suite of web browser-based tasks for computer control ranging from simple button
clicking to complex form filling here it is deleting files clicking on like
buttons and switching between tabs a bit like my earlier experiments they gave it
a math problem and said review your previous answer and find problems with your
answer this was a slightly more leading response but it worked they then said
based on the problems you found improve your answer and then the model got it
right even if you take nothing else from this video just deploying this technique
will massively improve your outputs from gbt4 but we can go much further which is
what the rest of the video is about before I move on though I found it very
interesting that the authors say that this technique can be viewed as using the
llm's output to write to an external memory which is later retrieved to choose an
action going back to carpathy remember that this critique retry metacognition
strategy isn't the only way that gpt4 will beat its own records the use of tools
as he says will also be critical less than 72 hours ago this paper was released
and arguably it is as significant as the reflection paper it's called hugging GPT
and as the authors put it it achieves impressive results in language Vision
speech and other challenging tasks which paves a new way towards AGI essentially
what the paper did is it used language as an interface to connect numerous AI
models for solving complicated AI tasks it's a little bit like a brain deciding
which muscle to use to complete an
action take this Example The Prompt was can you describe what this picture
depicts and count how many objects in the picture the model which was actually
chatbt not even gpt4 or use two different tools to execute the task one model to
describe the image and one model to count the objects within it and if you didn't
think that was impressive what about six different models so the task was this
please generate an image where a girl is reading a book and her pose is the same
as the boy in the image given then please describe the new image with your voice
the Central Language model or brain which was chattybt had to delegate
appropriately all of these models by the way are freely available on hugging face
the first model was used to analyze the pose of the boy the next one was to
transpose that into an image then generate an image detect an object in that
image break that down into text and then turn that text into speech it did all of
this and notice how the girl is in the same pose as the boy same head position
and arm position and then as a cherry on top the model read out loud what it had
accomplished this example actually comes from another paper released four days
ago called task Matrix remember how the original tool former paper used only five
apis this paper proposes that we could soon use millions in this example the
model is calling different apis to answer questions about the image caption the
image and do out painting from the image extending it from a simple single flower
to this 4K image going back to hugging GPT we can see how it deciphers these
inscrutable invoices and reads them out loud and can even perform text to video
with an astronaut walking in Space at this point I can't resist showing you what
CGI video editing might soon be possible with AI here's Wonder Studio which is
backed by Steven Spielberg welcome to wonder Studio we're making movies with CGI
is as simple as selecting your actor and assigning a character the system uses
AI to track the actor's performance across cuts and automatically animates lights
and composes the CG character directly into the scene [Music] whether it's one
shot or a full sequence Wonder Studio analyzes and captures everything from body
motion lighting compositing camera motion and it even tracks the actor's facial
performance these advancements do seem to be accelerating and requiring fewer and
fewer humans this paper showed back in the before times of October that models
didn't need carefully labeled human data sets and could generate their own going
back to the language models can solve computer task paper the authors seem to
concur they said that previously significant amounts of expert demonstration data
are still required to fine-tune large language models on the contrary the agent
we suggest needs less than two demonstrations per task on average and doesn't
necessitate any fine tuning this reminded me of the alpaca model that fine-tuned
its answers based on the outputs of another language model human experts were
needed briefly at the start but far less than before a bit like a child no longer
needing a parent except maybe gpt4 is on growth steroids Ilya satsgiver from
openai put it like this I mean already mostly data for enforcement loan is coming
from AIS the humans are being used to train the reward function but then the but
then the reward function enter and in its interaction with the model is
automatic and all the data that's generated in the during the process of
reinforcement learning it's created by AI before I end I should point out that
these recursive self-improvements are not limited to algorithms and apis even
Hardware is advancing more rapidly due to AI this week we had this from Reuters
Nvidia on Monday showed new research that explains how AI can be used to improve
chip design by the way this includes the new h100 GPU they say that the Nvidia
research took reinforcement learning and added a second layer of AI on top of it
to get even better results and to go back to where we started the gpt4 technical
report showed that even with compute alone not self-learning we can predict with
a high degree of specificity the future performance of models like gpc5 on tasks
such as human eval these accelerations of AI are even giving the CEO of Google
Whiplash and I can't help feeling that there is one more feedback loop to point
out as one company like openai make breakthroughs it puts pressure on other
companies like Google to catch up apparently Bard which has been powered by
Lambda will soon be upgraded to the more powerful model Palm with self-
improvement tool use Hardware advances and now commercial pressure it is hard to
see how AI will slow down and of course as always I will be here to discuss it
all thank you for watching to the end and have a wonderful day