Llama (language model)

Llama (Large Language Model Meta AI, formerly

Llama
stylized as LLaMA) is a family of large language
models (LLMs) released by Meta AI starting in
February 2023.[2][3] The latest version is Llama 3.3,
released in December 2024.[4]

Llama models are trained at different parameter sizes,

ranging between 1B and 405B.[5] Initially only a
foundation model,[6] starting with Llama 2, Meta AI
released instruction fine-tuned versions alongside
foundation models.[7]

Model weights for the first version of Llama were only

Screenshot of an example of Llama answer
available to researchers on a case-by-case basis, under
describing Wikipedia
a non-commercial license.[8][3] Unauthorized copies of
Developer(s) Meta AI
the first model were shared via BitTorrent.[9]
Subsequent versions of Llama were made accessible Initial release February 24, 2023
outside academia and released under licenses that Stable release Llama 3.3 / December 7, 2024
permitted some commercial use.[10][7] Repository github.com/meta-llama/llama-
models (https://github.com/met
Alongside the release of Llama 3, Meta added virtual
a-llama/llama-models)
assistant features to Facebook and WhatsApp in select
regions, and a standalone website. Both services use a Written in Python
Llama 3 model.[11] Type Large language model
GPT
Foundation model
Background
License Source-available (Meta Llama
After the release of large language models such as 3.2 Community License)[1]
GPT-3, a focus of research was up-scaling models Website llama.com (https://www.llama.c
which in some instances showed major increases in om/)
emergent capabilities. [12] The release of ChatGPT and
its surprise success caused an increase in attention to large language models.[13]

Compared with other responses to ChatGPT, Meta's Chief AI scientist Yann LeCun stated that large
language models are best for aiding with writing.[14][15][16][17]
An empirical investigation of the Llama series was the scaling laws. It was observed that the Llama 3
models showed that when a model is trained on data that is more than the "Chinchilla-optimal" amount,
the performance continues to scale log-linearly. For example, the Chinchilla-optimal dataset for Llama 3
8B is 200 billion tokens, but performance continued to scale log-linearly to the 75-times larger dataset of
15 trillion tokens.[18]

Initial release
LLaMA was announced on February 24, 2023, via a blog post and a paper describing the model's
training, architecture, and performance.[2][3] The inference code used to run the model was publicly
released under the open-source GPLv3 license.[19] Access to the model's weights was managed by an
application process, with access to be granted "on a case-by-case basis to academic researchers; those
affiliated with organizations in government, civil society, and academia; and industry research
laboratories around the world".[3]

Llama was trained on only publicly available information, and was trained at various model sizes, with
the intention to make it more accessible to different hardware. The model was exclusively a foundation
model,[6] although the paper contained examples of instruction fine-tuned versions of the model.[2]

Meta AI reported the 13B parameter model performance on most NLP benchmarks exceeded that of the
much larger GPT-3 (with 175B parameters), and the largest 65B model was competitive with state of the
art models such as PaLM and Chinchilla.[2]

Leak
On March 3, 2023, a torrent containing LLaMA's weights was uploaded, with a link to the torrent shared
on the 4chan imageboard and subsequently spread through online AI communities.[20] That same day, a
pull request on the main LLaMA repository was opened, requesting to add the magnet link to the official
documentation.[21][22] On March 4, a pull request was opened to add links to HuggingFace repositories
containing the model.[23][21] On March 6, Meta filed takedown requests to remove the HuggingFace
repositories linked in the pull request, characterizing it as "unauthorized distribution" of the model.
HuggingFace complied with the requests.[24] On March 20, Meta filed a DMCA takedown request for
copyright infringement against a repository containing a script that downloaded LLaMA from a mirror,
and GitHub complied the next day.[9]

Reactions to the leak varied. Some speculated that the model would be used for malicious purposes, such
as more sophisticated spam. Some have celebrated the model's accessibility, as well as the fact that
smaller versions of the model can be run relatively cheaply, suggesting that this will promote the
flourishing of additional research developments.[20] Multiple commentators, such as Simon Willison,
compared LLaMA to Stable Diffusion, a text-to-image model which, unlike comparably sophisticated
models which preceded it, was openly distributed, leading to a rapid proliferation of associated tools,
techniques, and software.[20][25]

LLaMa 2
On July 18, 2023, in partnership with Microsoft, Meta announced LLaMa 2, the next generation of
Llama. Meta trained and released Llama 2 in three model sizes: 7, 13, and 70 billion parameters.[7] The
model architecture remains largely unchanged from that of LLaMA-1 models, but 40% more data was
used to train the foundational models.[26] The accompanying preprint[26] also mentions a model with 34B
parameters that might be released in the future upon satisfying safety targets.

LLaMa 2 includes foundation models and models fine-tuned for chat. In a further departure from the
original version of LLaMa, all models are released with weights and may be used for many commercial
use cases. However, because LLaMa's license enforces an acceptable use policy that prohibits Llama
from being used for some purposes, Meta's use of the term open source to describe Llama has been
disputed by the Open Source Initiative (which maintains the The Open Source Definition) and
others.[27][28]

Code Llama is a fine-tune of LLaMa 2 with code specific datasets. 7B, 13B, and 34B versions were
released on August 24, 2023, with the 70B releasing on the January 29, 2024.[29] Starting with the
foundation models from LLaMa 2, Meta AI would train an additional 500B tokens of code datasets,
before an additional 20B token of long-context data, creating the Code Llama foundation models. This
foundation model was further trained on 5B instruction following token to create the instruct fine-tune.
Another foundation model was created for Python code, which trained on 100B tokens of Python-only
code, before the long-context data.[30]

Llama 3
On April 18, 2024, Meta released Llama-3 with two sizes: 8B and
70B parameters.[18] The models have been pre-trained on
approximately 15 trillion tokens of text gathered from “publicly
available sources” with the instruct models fine-tuned on “publicly
available instruction datasets, as well as over 10M human-
annotated examples". Meta AI's testing showed in April 2024 that
Llama 3 70B was beating Gemini Pro 1.5 and Claude 3 Sonnet on
most benchmarks. Meta also announced plans to make Llama 3
multilingual and multimodal, better at coding and reasoning, and
to increase its context window.[31][32]

During an interview with Dwarkesh Patel, Mark Zuckerberg said Example of an image generated by
that the 8B version of Llama 3 was nearly as powerful as the Meta AI Imagine, powered by Llama
largest Llama 2. Compared to previous models, Zuckerberg stated 3. Prompt: A representation
the team was surprised that the 70B model was still learning even of Meta AI and Llama
at the end of the 15T tokens training. The decision was made to
end training to focus GPU power elsewhere.[33]

Llama-3.1 was released on July 23, 2024, with three sizes: 8B, 70B, and 405B parameters.[5][34]

Comparison of models
For the training cost column, only the largest model's cost is written. So for example, "21,000" is the
training cost of Llama 2 69B in units of petaFLOP-day. Also, 1 petaFLOP-day = 1 petaFLOP/sec × 1 day
= 8.64E19 FLOP. "T" means "trillion" and "B" means "billion".

Context Corpus
Release Training cost Commercial
Name Parameters length size
date (petaFLOP-day) viability?
(tokens) (tokens)

6.7B
February 24, 13B
LLaMA
2023 6,300[35] 2048 1–1.4T No
32.5B
65.2B

July 18, 6.7B

Llama 2
2023 13B 21,000[36]
69B
4096 2T
6.7B
Code August 24, 13B
Llama 2023
33.7B
69B

April 18, 8B
Llama 3
2024 100,000[37][38] 8192
Yes, subject to
70.6B
acceptable use
15T policy
Llama July 23, 8B
3.1 2024 70.6B 440,000[34][39] 128,000
405B

1B
Llama September 3B
3.2 25, 2024 128,000[42]
11B
90B[40][41]

Llama December
70B 128,000
3.3 7, 2024

Architecture and training

"recommendation letter for the Magic Unicorn

Architecture Corporation"
Like GPT-3, the Llama series of models are Here is the recommendation letter that I wrote
autoregressive decoder-only Transformers, but for an application to a dragon feeder position at
there are some minor differences: the Magic Unicorn Corporation:
Dear recruiter,
SwiGLU[43] activation function instead I have known ___ for two years, and I believe that
of GeLU; she would be an excellent dragon feeder for the
rotary positional embeddings (RoPE)[44] Magic Unicorn Corporation. ___ has an ability to
instead of absolute positional remember and process large amounts of
 embedding; information, which is an important skill for a
RMSNorm[45] instead of layer dragon feeder.
; [46]
normalization ___, as an accomplished knight, has a deep
understanding of how to kill dragons and how to
key hyperparameters of Llama 3.1 use each dragon’s weaknesses against it. This
8B 70B 405B means that she knows what kinds of foods each
Layers 32 80 126 dragon likes and what kinds of foods are
dangerous to each dragon. This knowledge and
Model
4,096 8,192 16,384 experience will be invaluable as she feeds the
Dimension
FFN Dimension 14,336 28,672 53,248
dragons.
I am confident that ___’s competence, skill, and
Attention Heads 32 64 128
experience will make her an excellent employee.
Key/Value
8 8 8 Please contact me at (___) ___-___ if you have
Heads
any questions. I look forward to hearing from you.
Peak Learning 3× 1.5 × 0.8 × Best regards,
Rate 10−4 10−4 10−4
Honorable Knight
Activation
SwiGLU Sir George
Function

Vocabulary Size 128,000 – Output of 65 billion parameter LLaMA model before

Positional
instruction tuning, given the prompt (in bold)[2]
Embeddings

Training datasets
LLaMA's developers focused their effort on scaling the model's performance by increasing the volume of
training data, rather than the number of parameters, reasoning that the dominating cost for LLMs is from
doing inference on the trained model rather than the computational cost of the training process.

LLaMA 1 foundational models were trained on a data set with 1.4 trillion tokens, drawn from publicly
available data sources, including:[2]

Webpages scraped by CommonCrawl

Open source repositories of source code from GitHub
Wikipedia in 20 languages
Public domain books from Project Gutenberg
Books3 books dataset
The LaTeX source code for scientific papers uploaded to ArXiv
Questions and answers from Stack Exchange websites
On April 17, 2023, TogetherAI launched a project named RedPajama to reproduce and distribute an open
source version of the LLaMA dataset.[47] The dataset has approximately 1.2 trillion tokens and is publicly
available for download.[48]

Llama 2 foundational models were trained on a data set with 2 trillion tokens. This data set was curated
to remove Web sites that often disclose personal data of people. It also upsamples sources considered
trustworthy.[26] Llama 2 - Chat was additionally fine-tuned on 27,540 prompt-response pairs created for
this project, which performed better than larger but lower-quality third-party datasets. For AI alignment,
reinforcement learning with human feedback (RLHF) was used with a combination of 1,418,091 Meta
examples and seven smaller datasets. The average dialog depth was 3.9 in the Meta examples, 3.0 for
Anthropic Helpful and Anthropic Harmless sets, and 1.0 for five other sets, including OpenAI
Summarize, StackExchange, etc.

Llama 3 consists of mainly English data, with over 5% in over 30 other languages. Its dataset was
filtered by a text-quality classifier, and the classifier was trained by text synthesized by Llama 2.[18]

In a lawsuit brought by Richard Kadrey and others against Meta Platforms, CEO Mark Zuckerberg was
alleged to have authorized the use of copyrighted content from Library Genesis to train Llama AI models
and conceal its actions by removing copyright markers from the data.[49]

Fine-tuning
Llama 1 models are only available as foundational models with self-supervised learning and without fine-
tuning. Llama 2 – Chat models were derived from foundational Llama 2 models. Unlike GPT-4 which
increased context length during fine-tuning, Llama 2 and Code Llama - Chat have the same context
length of 4K tokens. Supervised fine-tuning used an autoregressive loss function with token loss on user
prompts zeroed out. The batch size was 64.

For AI alignment, human annotators wrote prompts and then compared two model outputs (a binary
protocol), giving confidence levels and separate safety labels with veto power. Two separate reward
models were trained from these preferences for safety and helpfulness using Reinforcement learning from
human feedback (RLHF). A major technical contribution is the departure from the exclusive use of
Proximal Policy Optimization (PPO) for RLHF – a new technique based on Rejection sampling was used,
followed by PPO.

Multi-turn consistency in dialogs was targeted for improvement, to make sure that "system messages"
(initial instructions, such as "speak in French" and "act like Napoleon") are respected during the dialog.
This was accomplished using the new "Ghost attention" technique during training, which concatenates
relevant instructions to each new user message but zeros out the loss function for tokens in the prompt
(earlier parts of the dialog).

Applications
The Stanford University Institute for Human-Centered Artificial Intelligence (HAI) Center for Research
on Foundation Models (CRFM) released Alpaca, a training recipe based on the LLaMA 7B model that
uses the "Self-Instruct" method of instruction tuning to acquire capabilities comparable to the OpenAI
GPT-3 series text-davinci-003 model at a modest cost.[50][51][52] The model files were officially removed
on March 21, 2023, over hosting costs and safety concerns, though the code and paper remain online for
reference.[53][54][55]

Meditron is a family of Llama-based finetuned on a corpus of clinical guidelines, PubMed papers, and
articles. It was created by researchers at École Polytechnique Fédérale de Lausanne School of Computer
and Communication Sciences, and the Yale School of Medicine. It shows increased performance on
medical-related benchmarks such as MedQA and MedMCQA.[56][57][58]
Zoom used Meta Llama 2 to create an AI Companion that can summarize meetings, provide helpful
presentation tips, and assist with message responses. This AI Companion is powered by multiple models,
including Meta Llama 2.[59]

Reuters reported in 2024 that many Chinese foundation models relied on Llama models for their
training.[60]

llama.cpp
Software developer Georgi Gerganov released llama.cpp as open-source on March 10, 2023. It's a re-
implementation of LLaMA in C++, allowing systems without a powerful GPU to run the model
locally.[61] The llama.cpp project introduced the GGUF file format, a binary format that stores both
tensors and metadata.[62] The format focuses on supporting different quantization types, which can reduce
memory usage, and increase speed at the expense of lower model precision.[63]

llamafile created by Justine Tunney is an open-source tool that bundles llama.cpp with the model into a
single executable file. Tunney et al. introduced new optimized matrix multiplication kernels for x86 and
ARM CPUs, improving prompt evaluation performance for FP16 and 8-bit quantized data types.[64]

Military
In 2024, researchers from the People's Liberation Army Academy of Military Sciences (top military
academy of China) were reported to have developed a military tool using Llama, which Meta Platforms
stated was unauthorized due to Llama's license prohibiting the use of the model for military
purposes.[65][66] Meta granted the US government and US military contractors permission to use Llama
in November 2024, but continued to prohibit military use by non-US entities.[28][67]

Reception
Wired describes the 8B parameter version of Llama 3 as being "surprisingly capable" given its size.[68]

The response to Meta's integration of Llama into Facebook was mixed, with some users confused after
Meta AI told a parental group that it had a child.[69]

According to the Q4 2023 Earnings transcript, Meta adopted the strategy of open weights to improve on
model safety, iteration speed, increase adoption among developers and researchers, and to become the
industry standard. Llama 5, 6, and 7 are planned for the future.[70]

The release of Llama models has sparked significant debates on the benefits and misuse risks of open
weight models. Such models can be fine-tuned to remove safeguards, notably by cyber criminals, until
they comply with harmful requests. Some experts contend that future models may facilitate causing
damage more than defending against it, for example by making it relatively easy to engineer advanced
bioweapons without specialized knowledge. Conversely, open-weight models can be useful for a wide
variety of purposes, including for safety research.[71]
Open Source Initiative head Stefano Maffulli criticized Meta for describing Llama as open source, saying
that it was causing confusion among users and "polluting" the term.[72]

See also
GPT-4o
IBM Granite, an open-source LLM made by IBM
Mistral AI, a French open-source AI company

References
1. "llama-models/models/llama3_2/LICENSE at main · meta-llama/llama-models · GitHub" (http
s://github.com/meta-llama/llama-models/blob/main/models/llama3_2/LICENSE). GitHub.
Archived (https://web.archive.org/web/20240929030827/https://github.com/meta-llama/llama
-models/blob/main/models/llama3_2/LICENSE) from the original on 2024-09-29. Retrieved
2024-10-20.
2. Touvron, Hugo; Lavril, Thibaut; Izacard, Gautier; Martinet, Xavier; Lachaux, Marie-Anne;
Lacroix, Timothée; Rozière, Baptiste; Goyal, Naman; Hambro, Eric; Azhar, Faisal;
Rodriguez, Aurelien; Joulin, Armand; Grave, Edouard; Lample, Guillaume (2023). "LLaMA:
Open and Efficient Foundation Language Models". arXiv:2302.13971 (https://arxiv.org/abs/2
302.13971) [cs.CL (https://arxiv.org/archive/cs.CL)].
3. "Introducing LLaMA: A foundational, 65-billion-parameter large language model" (https://ai.fa
cebook.com/blog/large-language-model-llama-meta-ai/). Meta AI. 24 February 2023.
Archived (https://web.archive.org/web/20230303112302/https://ai.facebook.com/blog/large-l
anguage-model-llama-meta-ai/) from the original on 3 March 2023. Retrieved 16 March
2023.
4. Wiggers, Kyle (2024-12-06). "Meta unveils a new, more efficient Llama model" (https://techc
runch.com/2024/12/06/meta-unveils-a-new-more-efficient-llama-model/). TechCrunch.
Retrieved 2024-12-25.
5. "Introducing Llama 3.1: Our most capable models to date" (https://ai.meta.com/blog/meta-lla
ma-3-1/). ai.meta.com. July 23, 2024. Archived (https://web.archive.org/web/202407231539
09/https://ai.meta.com/blog/meta-llama-3-1/) from the original on 2024-07-23. Retrieved
2024-07-23.
6. Peters, Jay; Vincent, James (24 February 2023). "Meta has a new machine learning
language model to remind you it does AI too" (https://www.theverge.com/2023/2/24/2361351
2/meta-llama-ai-research-large-language-model). The Verge.
7. "Meta and Microsoft Introduce the Next Generation of LLaMA" (https://about.fb.com/news/2
023/07/llama-2/). Meta. 18 July 2023. Archived (https://web.archive.org/web/202309141323
06/https://about.fb.com/news/2023/07/llama-2/) from the original on 14 September 2023.
Retrieved 21 July 2023.
8. Malik, Yuvraj; Paul, Katie (25 February 2023). "Meta heats up Big Tech's AI arms race with
new language model" (https://www.reuters.com/technology/meta-launch-ai-language-model-
llama-2023-02-24/). Reuters.
9. OpSec Online LLC (21 March 2023). "github/dmca - Notice of Claimed Infringement via
Email" (https://github.com/github/dmca/blob/master/2023/03/2023-03-21-meta.md). GitHub.
Archived (https://web.archive.org/web/20230410032303/https://github.com/github/dmca/blo
b/master/2023/03/2023-03-21-meta.md) from the original on 10 April 2023. Retrieved
25 March 2023.
10. David, Emilia (30 October 2023). "Meta's AI research head wants open source licensing to
change" (https://www.theverge.com/2023/10/30/23935587/meta-generative-ai-models-open-
source). The Verge. Archived (https://web.archive.org/web/20240914145514/https://www.the
verge.com/2023/10/30/23935587/meta-generative-ai-models-open-source) from the original
on 14 September 2024. Retrieved 20 October 2024.
11. "Meet Your New Assistant: Meta AI, Built With Llama 3" (https://about.fb.com/news/2024/04/
meta-ai-assistant-built-with-llama-3/). Meta. 18 April 2024. Archived (https://web.archive.org/
web/20241007093730/https://about.fb.com/news/2024/04/meta-ai-assistant-built-with-llama-
3/) from the original on 7 October 2024. Retrieved 20 October 2024.
12. "Examining Emergent Abilities in Large Language Models" (https://hai.stanford.edu/news/ex
amining-emergent-abilities-large-language-models). hai.stanford.edu. 13 September 2022.
13. "The inside story of how ChatGPT was built from the people who made it" (https://www.tech
nologyreview.com/2023/03/03/1069311/inside-story-oral-history-how-chatgpt-built-openai/).
MIT Technology Review. Archived (https://web.archive.org/web/20230303093219/https://ww
w.technologyreview.com/2023/03/03/1069311/inside-story-oral-history-how-chatgpt-built-op
enai/) from the original on 2023-03-03. Retrieved 2024-10-20.
14. Ray, Tiernan (23 January 2023). "ChatGPT is 'not particularly innovative,' and 'nothing
revolutionary', says Meta's chief AI scientist" (https://www.zdnet.com/article/chatgpt-is-not-p
articularly-innovative-and-nothing-revolutionary-says-metas-chief-ai-scientist/). ZDNET.
Archived (https://web.archive.org/web/20230217163917/https://www.zdnet.com/article/chatg
pt-is-not-particularly-innovative-and-nothing-revolutionary-says-metas-chief-ai-scientist/)
from the original on 2023-02-17.
15. Badminton, Nik (13 February 2023). "Meta's Yann LeCun on auto-regressive Large
Language Models (LLMs)" (https://futurist.com/2023/02/13/metas-yann-lecun-thoughts-large
-language-models-llms/). Futurist.com. Archived (https://web.archive.org/web/20240722082
109/https://futurist.com/2023/02/13/metas-yann-lecun-thoughts-large-language-models-llm
s/) from the original on 22 July 2024. Retrieved 20 October 2024.
16. "Yann LeCun on LinkedIn: My unwavering opinion on current (auto-regressive) LLMs" (http
s://www.linkedin.com/feed/update/urn:li:activity:7030921081876029443/).
www.linkedin.com. Archived (https://web.archive.org/web/20240917092533/https://www.link
edin.com/feed/update/urn:li:activity:7030921081876029443/) from the original on 2024-09-
17. Retrieved 2024-10-20.
17. "Meta's Yann LeCun Asks How AIs will Match — and Exceed — Human-level Intelligence"
(https://www.engineering.columbia.edu/about/news/metas-yann-lecun-asks-how-ais-will-mat
ch-and-exceed-human-level-intelligence). 23 October 2024.
18. "Introducing Meta Llama 3: The most capable openly available LLM to date" (https://ai.meta.
com/blog/meta-llama-3/). ai.meta.com. April 18, 2024. Archived (https://web.archive.org/we
b/20240515023523/https://ai.meta.com/blog/meta-llama-3/) from the original on 2024-05-15.
Retrieved 2024-04-21.
19. "llama" (https://github.com/facebookresearch/llama). GitHub. Archived (https://web.archive.o
rg/web/20230315183955/https://github.com/facebookresearch/llama/) from the original on
15 March 2023. Retrieved 16 March 2023.
20. Vincent, James (8 March 2023). "Meta's powerful AI language model has leaked online —
what happens now?" (https://www.theverge.com/2023/3/8/23629362/meta-ai-language-mod
el-llama-leak-online-misuse). The Verge. Archived (https://web.archive.org/web/2023110316
1046/https://www.theverge.com/2023/3/8/23629362/meta-ai-language-model-llama-leak-onli
ne-misuse) from the original on 3 November 2023. Retrieved 16 March 2023.
21. VK, Anirudh (6 March 2023). "Meta's LLaMA Leaked to the Public, Thanks To 4chan" (http
s://analyticsindiamag.com/metas-llama-leaked-to-the-public-thanks-to-4chan/). Analytics
India Magazine. Archived (https://web.archive.org/web/20230326020443/https://analyticsindi
amag.com/metas-llama-leaked-to-the-public-thanks-to-4chan/) from the original on 26
March 2023. Retrieved 17 March 2023.
22. "Save bandwidth by using a torrent to distribute more efficiently by ChristopherKing42 · Pull
Request #73 · facebookresearch/llama" (https://github.com/facebookresearch/llama/pull/73).
GitHub. Archived (https://web.archive.org/web/20230410000618/https://github.com/faceboo
kresearch/llama/pull/73) from the original on 10 April 2023. Retrieved 25 March 2023.
23. "Download weights from hugging face to help us save bandwidth by Jainam213 · Pull
Request #109 · facebookresearch/llama" (https://github.com/facebookresearch/llama/pull/10
9). GitHub. Archived (https://web.archive.org/web/20230321172220/https://github.com/faceb
ookresearch/llama/pull/109) from the original on 21 March 2023. Retrieved 17 March 2023.
24. Cox, Joseph (7 March 2023). "Facebook's Powerful Large Language Model Leaks Online"
(https://www.vice.com/en/article/xgwqgw/facebooks-powerful-large-language-model-leaks-o
nline-4chan-llama). Vice. Archived (https://web.archive.org/web/20230406135000/https://ww
w.vice.com/en/article/xgwqgw/facebooks-powerful-large-language-model-leaks-online-4cha
n-llama) from the original on 6 April 2023. Retrieved 17 March 2023.
25. Willison, Simon (11 March 2023). "Large language models are having their Stable Diffusion
moment" (https://simonwillison.net/2023/Mar/11/llama/). Simon Willison's Weblog. Archived
(https://web.archive.org/web/20230316201253/https://simonwillison.net/2023/Mar/11/llama/)
from the original on 16 March 2023. Retrieved 16 March 2023.
26. Touvron, Hugo; Martin, Louis; et al. (18 Jul 2023). "LLaMA-2: Open Foundation and Fine-
Tuned Chat Models". arXiv:2307.09288 (https://arxiv.org/abs/2307.09288) [cs.CL (https://arx
iv.org/archive/cs.CL)].
27. Edwards, Benj (2023-07-18). "Meta launches LLaMA-2, a source-available AI model that
allows commercial applications [Updated]" (https://arstechnica.com/information-technology/2
023/07/meta-launches-llama-2-an-open-source-ai-model-that-allows-commercial-application
s/). Ars Technica. Archived (https://web.archive.org/web/20231107082612/https://arstechnic
a.com/information-technology/2023/07/meta-launches-llama-2-an-open-source-ai-model-tha
t-allows-commercial-applications/) from the original on 2023-11-07. Retrieved 2023-08-08.
28. Thomas, Prasanth Aby (5 November 2024). "Meta offers Llama AI to US government for
national security" (https://www.cio.com/article/3599448/meta-offers-llama-ai-to-us-governme
nt-for-national-security.html). CIO. Retrieved 9 December 2024.
29. "Introducing Code Llama, a state-of-the-art large language model for coding" (https://ai.met
a.com/blog/code-llama-large-language-model-coding/). ai.meta.com. Archived (https://web.a
rchive.org/web/20240927091138/https://ai.meta.com/blog/code-llama-large-language-model
-coding/) from the original on 2024-09-27. Retrieved 2024-10-20.
30. Rozière, Baptiste; Gehring, Jonas; Gloeckle, Fabian; Sootla, Sten; Gat, Itai; Tan, Xiaoqing
Ellen; Adi, Yossi; Liu, Jingyu; Sauvestre, Romain (2024-01-31). "Code Llama: Open
Foundation Models for Code". arXiv:2308.12950 (https://arxiv.org/abs/2308.12950) [cs.CL (h
ttps://arxiv.org/archive/cs.CL)].
31. Wiggers, Kyle (18 April 2024). "Meta releases Llama 3, claims it's among the best open
models available" (https://techcrunch.com/2024/04/18/meta-releases-llama-3-claims-its-amo
ng-the-best-open-models-available/). TechCrunch. Archived (https://web.archive.org/web/20
240918202013/https://techcrunch.com/2024/04/18/meta-releases-llama-3-claims-its-among-
the-best-open-models-available/) from the original on 18 September 2024. Retrieved
20 October 2024.
32. Mann, Tobias (April 19, 2024). "Meta debuts third-generation Llama large language model"
(https://www.theregister.com/2024/04/19/meta_debuts_llama3_llm/). The Register. Archived
(https://web.archive.org/web/20240825145130/https://www.theregister.com/2024/04/19/met
a_debuts_llama3_llm/) from the original on August 25, 2024. Retrieved October 20, 2024.
33. Patel, Dwarkesh (2024-07-24). "Mark Zuckerberg - Llama 3, Open Sourcing $10b Models, &
Caesar Augustus" (https://www.dwarkeshpatel.com/p/mark-zuckerberg).
www.dwarkeshpatel.com. Archived (https://web.archive.org/web/20240716152236/https://w
ww.dwarkeshpatel.com/p/mark-zuckerberg) from the original on 2024-07-16. Retrieved
2024-08-01. "the 8 billion is nearly as powerful as the biggest version of Llama 2 that we
released [...] even by the end, it was... still learning right it's like we probably could have fed
it more tokens and it would have gotten somewhat better but i mean at some point you know
you're running a company you need to do these meta reasoning questions of [...] how do I
want to spend our GPUs"
34. Dubey, Abhimanyu; Jauhri, Abhinav; Pandey, Abhinav; Kadian, Abhishek; Al-Dahle, Ahmad;
Letman, Aiesha; Mathur, Akhil; Schelten, Alan; Yang, Amy (2024-07-31), The Llama 3 Herd
of Models, arXiv:2407.21783 (https://arxiv.org/abs/2407.21783)
35. "The Falcon has landed in the Hugging Face ecosystem" (https://huggingface.co/blog/falco
n). huggingface.co. Archived (https://web.archive.org/web/20230620002832/https://huggingf
ace.co/blog/falcon) from the original on 2023-06-20. Retrieved 2023-06-20.
36. "llama/MODEL_CARD.md at main · meta-llama/llama" (https://github.com/meta-llama/llama/
blob/main/MODEL_CARD.md). GitHub. Archived (https://web.archive.org/web/20240528090
541/https://github.com/meta-llama/llama/blob/main/MODEL_CARD.md) from the original on
2024-05-28. Retrieved 2024-05-28.
37. "Andrej Karpathy (Apr 18, 2024), The model card has some more interesting info too" (http
s://x.com/karpathy/status/1781047292486914189). Archived (https://web.archive.org/web/20
240817055806/https://x.com/karpathy/status/1781047292486914189) from the original on
August 17, 2024. Retrieved October 20, 2024.
38. "llama3/MODEL_CARD.md at main · meta-llama/llama3" (https://github.com/meta-llama/lla
ma3/blob/main/MODEL_CARD.md). GitHub. Archived (https://web.archive.org/web/2024052
1181439/https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md) from the
original on 2024-05-21. Retrieved 2024-05-28.
39. "llama-models/models/llama3_1/MODEL_CARD.md at main · meta-llama/llama-models" (htt
ps://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md).
GitHub. Archived (https://web.archive.org/web/20240723151851/https://github.com/meta-lla
ma/llama-models/blob/main/models/llama3_1/MODEL_CARD.md) from the original on
2024-07-23. Retrieved 2024-07-23.
40. Robison, Kylie (2024-09-25). "Meta releases its first open AI model that can process
images" (https://www.theverge.com/2024/9/25/24253774/meta-ai-vision-model-llama-3-2-an
nounced). The Verge. Retrieved 2024-09-25.
41. Wiggers, Kyle (2024-09-25). "Meta's Llama AI models get multimodal" (https://techcrunch.co
m/2024/09/25/metas-llama-ai-models-get-multimodal/). TechCrunch. Archived (https://web.a
rchive.org/web/20240925192155/https://techcrunch.com/2024/09/25/metas-llama-ai-models
-get-multimodal/) from the original on 2024-09-25. Retrieved 2024-09-25.
42. "Archived copy" (https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devic
es/). ai.meta.com. Archived (https://web.archive.org/web/20240925235424/https://ai.meta.co
m/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/) from the original on 2024-09-
25. Retrieved 2024-09-26.
43. Shazeer, Noam (2020-02-01). "GLU Variants Improve Transformer". arXiv:2002.05202 (http
s://arxiv.org/abs/2002.05202) [cs.CL (https://arxiv.org/archive/cs.CL)].
44. Su, Jianlin; Lu, Yu; Pan, Shengfeng; Murtadha, Ahmed; Wen, Bo; Liu, Yunfeng (2021-04-
01). "RoFormer: Enhanced Transformer with Rotary Position Embedding". arXiv:2104.09864
(https://arxiv.org/abs/2104.09864) [cs.CL (https://arxiv.org/archive/cs.CL)].
45. Zhang, Biao; Sennrich, Rico (2019-10-01). "Root Mean Square Layer Normalization".
arXiv:1910.07467 (https://arxiv.org/abs/1910.07467) [cs.LG (https://arxiv.org/archive/cs.LG)].
46. Lei Ba, Jimmy; Kiros, Jamie Ryan; Hinton, Geoffrey E. (2016-07-01). "Layer Normalization".
arXiv:1607.06450 (https://arxiv.org/abs/1607.06450) [stat.ML (https://arxiv.org/archive/stat.M
L)].
47. "RedPajama-Data: An Open Source Recipe to Reproduce LLaMA training dataset" (https://g
ithub.com/togethercomputer/RedPajama-Data). GitHub. Together. Archived (https://web.arch
ive.org/web/20231107223503/https://github.com/togethercomputer/RedPajama-Data) from
the original on 7 November 2023. Retrieved 4 May 2023.
48. "RedPajama-Data-1T" (https://huggingface.co/datasets/togethercomputer/RedPajama-Data-
1T). Hugging Face. Together. Archived (https://web.archive.org/web/20231103013716/http
s://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T) from the original on 3
November 2023. Retrieved 4 May 2023.
49. Wiggers, Kyle (January 9, 2025). "Mark Zuckerberg gave Meta's Llama team the OK to train
on copyrighted works, filing claims" (https://techcrunch.com/2025/01/09/mark-zuckerberg-ga
ve-metas-llama-team-the-ok-to-train-on-copyrighted-works-filing-claims/). Techcrunch.
Retrieved January 12, 2025.
50. Taori, Rohan; Gulrajani, Ishaan; Zhang, Tianyi; Dubois, Yann; Li, Xuechen; Guestrin, Carlos;
Liang, Percy; Hashimoto, Tatsunori B. (13 March 2023). "Alpaca: A Strong, Replicable
Instruction-Following Model" (https://crfm.stanford.edu/2023/03/13/alpaca.html). Stanford
Center for Research on Foundation Models. Archived (https://web.archive.org/web/2023040
6082332/https://crfm.stanford.edu/2023/03/13/alpaca.html) from the original on 6 April 2023.
51. Wang, Yizhong; Kordi, Yeganeh; Mishra, Swaroop; Liu, Alisa; Smith, Noah A.; Khashabi,
Daniel; Hajishirzi, Hannaneh (2022). "Self-Instruct: Aligning Language Models with Self-
Generated Instructions". arXiv:2212.10560 (https://arxiv.org/abs/2212.10560) [cs.CL (https://
arxiv.org/archive/cs.CL)].
52. "Stanford CRFM" (https://crfm.stanford.edu/2023/03/13/alpaca.html). crfm.stanford.edu.
Archived (https://web.archive.org/web/20230406082332/https://crfm.stanford.edu/2023/03/1
3/alpaca.html) from the original on 2023-04-06. Retrieved 2023-03-20.
53. Quach, Katyanna. "Stanford takes costly, risky Alpaca AI model offline" (https://www.theregis
ter.com/2023/03/21/stanford_ai_alpaca_taken_offline/). www.theregister.com.
54. "Stanford Researchers Take Down Alpaca AI Over Cost and Hallucinations" (https://gizmod
o.com/stanford-ai-alpaca-llama-facebook-taken-down-chatgpt-1850247570). Gizmodo. 21
March 2023. Archived (https://web.archive.org/web/20240512075506/https://gizmodo.com/st
anford-ai-alpaca-llama-facebook-taken-down-chatgpt-1850247570) from the original on 12
May 2024. Retrieved 20 October 2024.
55. "alpaca-lora" (https://github.com/tloen/alpaca-lora). GitHub. Archived (https://web.archive.or
g/web/20230404210345/https://github.com/tloen/alpaca-lora) from the original on 4 April
2023. Retrieved 5 April 2023.
56. "Meditron: An LLM suite for low-resource medical settings leveraging Meta Llama" (https://a
i.meta.com/blog/llama-2-3-meditron-yale-medicine-epfl-open-source-llm/). ai.meta.com.
57. Petersen, Tanya (28 November 2023). "EPFL's new Large Language Model for Medical
Knowledge" (https://actu.epfl.ch/news/epfl-s-new-large-language-model-for-medical-knowl
e/). Archived (https://web.archive.org/web/20240917180520/https://actu.epfl.ch/news/epfl-s-
new-large-language-model-for-medical-knowle/) from the original on 17 September 2024.
Retrieved 20 October 2024.
58. "epfLLM/meditron" (https://github.com/epfLLM/meditron). epfLLM. 11 May 2024. Archived (h
ttps://web.archive.org/web/20240927092256/https://github.com/epfLLM/meditron) from the
original on 27 September 2024. Retrieved 20 October 2024.
59. "How Companies Are Using Meta Llama" (https://about.fb.com/news/2024/05/how-compani
es-are-using-meta-llama/). Meta. 7 May 2024. Archived (https://web.archive.org/web/202409
27181724/https://about.fb.com/news/2024/05/how-companies-are-using-meta-llama/) from
the original on 27 September 2024. Retrieved 20 October 2024.
60. "How dependent is China on US artificial intelligence technology?" (https://www.reuters.co
m/technology/how-dependent-is-china-us-artificial-intelligence-technology-2024-05-09/).
Reuters. May 9, 2024.
61. Edwards, Benj (2023-03-13). "You can now run a GPT-3-level AI model on your laptop,
phone, and Raspberry Pi" (https://arstechnica.com/information-technology/2023/03/you-can-
now-run-a-gpt-3-level-ai-model-on-your-laptop-phone-and-raspberry-pi/). Ars Technica.
Archived (https://web.archive.org/web/20240109194611/https://arstechnica.com/information
-technology/2023/03/you-can-now-run-a-gpt-3-level-ai-model-on-your-laptop-phone-and-ras
pberry-pi/) from the original on 2024-01-09. Retrieved 2024-01-04.
62. "GGUF" (https://huggingface.co/docs/hub/gguf). huggingface.co. Retrieved 9 May 2024.
63. Labonne, Maxime (29 November 2023). "Quantize Llama models with GGUF and
llama.cpp" (https://towardsdatascience.com/quantize-llama-models-with-ggml-and-llama-cp
p-3612dfbcc172). Medium. Towards Data Science. Archived (https://web.archive.org/web/20
240509081605/https://towardsdatascience.com/quantize-llama-models-with-ggml-and-llama
-cpp-3612dfbcc172) from the original on 9 May 2024. Retrieved 9 May 2024.
64. Connatser, Matthew. "Llamafile LLM driver project boosts performance on CPU cores" (http
s://www.theregister.com/2024/04/03/llamafile_performance_gains/). www.theregister.com.
Archived (https://web.archive.org/web/20240510232003/https://www.theregister.com/2024/0
4/03/llamafile_performance_gains/) from the original on 10 May 2024. Retrieved 10 May
2024.
65. Cheung, Sunny (October 31, 2024). "PRC Adapts Meta's Llama for Military and Security AI
Applications" (https://jamestown.org/program/prcs-adaptation-of-open-source-llm-for-military
-and-security-purposes/). Jamestown Foundation. Retrieved 2024-11-03.
66. Pomfret, James; Pang, Jessie (November 1, 2024). "Chinese researchers develop AI model
for military use on back of Meta's Llama" (https://www.reuters.com/technology/artificial-intelli
gence/chinese-researchers-develop-ai-model-military-use-back-metas-llama-2024-11-01/).
Reuters. Retrieved November 1, 2024.
67. Smith, Matthew S. (17 November 2024). "Meta Opens Its AI Model for the U.S. Military -
IEEE Spectrum" (https://spectrum.ieee.org/ai-used-by-military). IEEE Spectrum. Retrieved
9 December 2024.
68. Knight, Will. "Meta's Open Source Llama 3 Is Already Nipping at OpenAI's Heels" (https://w
ww.wired.com/story/metas-open-source-llama-3-nipping-at-openais-heels/). Wired. Archived
(https://web.archive.org/web/20240927073830/https://www.wired.com/story/metas-open-sou
rce-llama-3-nipping-at-openais-heels/) from the original on 2024-09-27. Retrieved
2024-10-20.
69. "Meta's amped-up AI agents confusing Facebook users" (https://www.abc.net.au/news/2024
-04-19/meta-releases-llama-3-ai-model/103744538). ABC News. 19 April 2024. Archived (htt
ps://web.archive.org/web/20240917102930/https://www.abc.net.au/news/2024-04-19/meta-r
eleases-llama-3-ai-model/103744538) from the original on 2024-09-17. Retrieved
2024-10-20.
70. "Archived copy" (https://s21.q4cdn.com/399680738/files/doc_financials/2023/q4/META-Q4-2
023-Earnings-Call-Transcript.pdf) (PDF). Archived (https://web.archive.org/web/2024091711
5531/https://s21.q4cdn.com/399680738/files/doc_financials/2023/q4/META-Q4-2023-Earnin
gs-Call-Transcript.pdf) (PDF) from the original on 2024-09-17. Retrieved 2024-10-20.
71. Knight, Will. "Meta's New Llama 3.1 AI Model Is Free, Powerful, and Risky" (https://www.wir
ed.com/story/meta-ai-llama-3/). Wired. ISSN 1059-1028 (https://search.worldcat.org/issn/10
59-1028). Archived (https://web.archive.org/web/20240803201314/https://www.wired.com/st
ory/meta-ai-llama-3/) from the original on 2024-08-03. Retrieved 2024-08-04.
72. Waters, Richard (October 17, 2024). "Meta under fire for 'polluting' open-source" (https://ww
w.ft.com/content/397c50d8-8796-4042-a814-0ac2c068361f). Financial Times.
Further reading
Huang, Kalley; O'Regan, Sylvia Varnham (September 5, 2023). "Inside Meta's AI Drama:
Internal Feuds Over Compute Power" (https://www.theinformation.com/articles/inside-metas
-ai-drama-internal-feuds-over-compute-power). The Information. Archived (https://web.archi
ve.org/web/20230905174145/https://www.theinformation.com/articles/inside-metas-ai-drama
-internal-feuds-over-compute-power) from the original on September 5, 2023. Retrieved
September 6, 2023.

External links
Official website (https://www.llama.com/)
Official Hugging Face organization for Llama, Llama Guard, and Prompt Guard models (http
s://huggingface.co/meta-llama)

Retrieved from "https://en.wikipedia.org/w/index.php?title=Llama_(language_model)&oldid=1272591430"