Open-Source and Science

In the Era of Foundation Models

Berkeley LLM Agents Course - November 18, 2024
Percy Liang
Capabilities skyrocket…

capability

2018 2019 2020 2021 2022 2023

Access plummets…
paper, code, data, weights

paper, weights
access

paper, API

API

2018 2019 2020 2021 2022 2023

Why does access matter?
Access shapes research

1990s: Internet (text in digital form) ⇒ statistical NLP methods

2010s: crowdsourcing platforms ⇒ large annotated datasets

2010s: GPUs ⇒ deep learning methods

Levels of access for foundation models

API “cognitive scientist”

open-weight “neuroscientist”

open-source “computer scientist”

API access

Analogy: cognitive scientists can measure behavior

prompt response

Opportunity: build agents to solve complex problems

Open-weight access

Analogy: neuroscientists can probe internal activations

Opportunity: understand mechanisms, create novel derivatives

Open-source access

Analogy: computer scientist building a system can control every part of it

Opportunity: question everything

Levels of access for foundation models

API “cognitive scientist”

open-weight “neuroscientist”

open-source “computer scientist”

Levels of access for foundation models

API “cognitive scientist”

open-weight “neuroscientist”

open-source “computer scientist”

API access
prompt response

GPT-4, Claude, Gemini

● Think of the API as a universal function (e.g., summarize, verify, generate)

● Compose API calls together into systems (agents)
● Important: API is controller of execution (not called by fixed program)
Agent architecture
future

present

(tool use)

past
(verifiers)
Tale of two agents

Problem-solving agents Simulation agents

ICML 2024
Results
Success rate: fraction over 8 trials that agent improves by 10% over reference
Reflections: research agents
● Related work
○ MLE-Bench [Chan+ 2024]: benchmark with 75 Kaggle challenges
○ AIDE [Schmidt+ 2024]: agent architecture for data science competitions
○ OpenHands (OpenDevin) [Wang+ 2024]: general-purpose platform for software development
○ CORE-Bench [Siegel+ 2024]: benchmark to reproduce research results
○ Generating novel research ideas [Si+ 2024]

Self-improvement: solve task → improve model → solve task better

Results

Hardest: 24 hours
Reflections: dual implications of cybersecurity agents

Evaluation of cyber-risk (offense) Penetration testing tool (defense)

Tale of two agents

Problem-solving agents Simulation agents

UIST 2023
Architecture
Retrieval
Reflection
Simulating social behavior
Let’s make it real…
Interviews are rich
Interviewer: To start, I would like to begin with a big question: tell me the story of your life. Start from the beginning -- from your
childhood, to education, to family and relationships, and to any major life events you may have had.

Joon: Right. So I was born in Korea. I lived there for the first 12 years or so of my life. And my parents and I moved to Boston. So my parents were
pursuing their postdoctoral degree. They're both medical doctors. So we moved to Boston. So we lived there for five years. And then I went to high
school in New Hampshire after that, college in Pennsylvania, called Swarthmore College. It's a liberal arts college. And towards the end of my
college years, one other co-founder and I started to run a startup. So while we were running that, as we were graduating, we moved to Palo Alto. So
we lived by somebody's garage, actually, for about a year. The typical thing that you do if you're into that. The startup itself didn't pan out. But during
that year, I got to know this group of researchers and community in the computer science department at Stanford, just by through some luck and just,
yeah, through pure luck. And I started to work on some research projects with them. So I spent about a half a year, a little bit more than maybe half a
year working with them. And then I went to get my master's in computer science at UIUC, that's Illinois Urbana-Champaign. So I spent two years of
my life there. And when I was finishing that up, I moved back to Palo Alto, started my PhD as a computer science PhD student at Stanford. And I've
been here since then. So this is my year five of doing that.

Interviewer: Thank you for sharing that overview! I'd love to learn more about your experiences. Can you tell me what it was like for you
adjusting to life in Boston after moving from Korea? What were some of the challenges you faced during that transition?

Joon: It's a good question. Well, moving to a different country and different culture for any child and their parents, it's a heavy lift. But I think likely
what was sort of the most interesting was the fact that the entire family, to some extent, was sort of at this point where they were all trying to figure
out their career and so forth…
85% normalized accuracy!
Reflections: agents and API access

● Use API to create agents

● Solve complex problems in ML engineering and cybersecurity
● Simulate people (digital twin of society) - lab for social scientists
● Next: static agents ⇒ learn from experiences
● AlphaGo analogy: supervised learning ⇒ reinforcement learning
Levels of access for foundation models

API “cognitive scientist”

open-weight “neuroscientist”

open-source “computer scientist”

Open-weight access

Llama, Qwen, Mixtral, Jamba, Yi, Gemma, Phi

More accurately: dual-use foundation models with widely available weights

Reproducibility

API models get deprecated You always have the weights…

 θ1 θ2

Were θ1 and θ2 independently trained or not (e.g., θ1 fine-tuned from θ2)?

Idea 1
Compute sim(θ1, θ2) - e.g., cosine similarity of MLP weights

Problem: if sim(θ1, θ2) = 0.1, is that similar or not? Statistical guarantees?

Idea 2
Train a bunch of models { sim(θ1’, θ2): θ1’ = train(random init) }

p-value = ℙ[sim(θ1’, θ2) > sim(θ1, θ2)]

Problem: impossible to train to get θ1’ since only have the final weights!
Idea 3
perm(θ) = permute the hidden units defined by θ to get counterfactuals

p-value = ℙ[sim(perm(θ1), θ2) > sim(θ1, θ2)]

Empirical validation

Not independent!
 StripedHyena-Nous-7B ~ Mistral-7B-v0.1
Other findings
Miqu-70B (Mistral leak) ~ Llama-2-70B

Llama-3.1-8B ~ Llama-3.2-3B
Reflections: open-weight access

● Strong open-weight models (e.g., Llama 3) have been immensely valuable

● Enables research on interpretability, fine-tuning, distillation, merging (all
reproducible!)
● Question: how weight modifications can yield coherent functional changes?
● Teaches us about API models (e.g., adversarial attacks transfer)
● New problems motivated by open-weights: model independence testing
● But still confined by the blueprint of existing models…
Levels of access for foundation models

API “cognitive scientist”

open-weight “neuroscientist”

open-source “computer scientist”

Open-source language model efforts

FineWeb, SmolLM
GPT-J, GPT-NeoX, Pythia OLMo, OLMoE

RedPajama
StarCoder

DCLM-BASELINE
MAP-Neo, OpenCoder
K2
Performance gaps

Model Access MMLU

Claude Sonnet 3.5 API 87.3

Llama 3.1 Instruct (405B) open-weight 84.5

OLMo 1.7 (7B) open-source 53.8

What exactly is open-source?
Free and open-source software
Roots: hacker ethic (MIT in 1950s) + academia (for centuries)

Values: creativity, exploration, transparency, collaboration, resistance against authority

1983: Richard Stallman started GNU (bash, ls, …)

1991: Linus Torvalds started Linux

1998: Open-Source Initiative (OSI) - coined and defined “open-source”

Open-Source AI Definition - version 1.0
Data information, not data

Model developers don’t own license for web data (copyrighted), can’t release!
Need compute to (re-)train to achieve spirit of open-source
 What mixture to use?

distributionally robust optimization (DRO)

NeurIPS 2023
diagonal Hessian with clipping

ICLR 2024
 precise model editing

MacBook := MacBook - Apple + HP

ACL 2023
Would the results hold if we scaled up?
Where do we get the compute?
Track 1: construct scaling laws that extend down
Track 2: harness idle GPUs everywhere
The problem

100Gbps 1Gbps
Training (1B models) is only ~2x slower than in the datacenter
NeurIPS 2022
Track 3: fund the public good
Big Science
Levels of access for foundation models

API “cognitive scientist”

open-weight “neuroscientist”

open-source “computer scientist”

Final remarks

● Access shapes research

● Many interesting problems with API (agents) and open-weight (distillation)
● Today, most research lives within the confines of APIs and fixed weights
● Question everything: data, model architecture, training algorithm
● Goal: understand data, architecture → model behavior (hard even with full
access)
● Compute: try at smaller scales + scaling laws; pool our compute
Thank you!
THE END
Train-test overlap
Upshot: can predict benchmark performance
Data information includes data processing code
Big question: but will these results transfer to larger scales?
Can use surrogates to extrapolate across scales
What is the future of the open web?