Large Language Model Threats Taxonomy 20240610

Large Language
Model Threats
Taxonomy
AI Controls Framework
Working group
Release Date: 06/10/2024
Cloud Security
Alliance
Defining
● LLM Assets
● LLM-Service Lifecycle
● LLM-Service Impact Categories
● LLM-Service Threat Categories
2
LLM Assets
3
LLM-Ops Cloud Environment
Cloud running the training Hybrid and multi-cloud

1 environment 4 infrastructure
Cloud running the model Security of the deployment

2 inference point 5 environment
Cloud running the AI Continuous monitoring, etc.

3 applications 6
4
Model
Foundation Model Open Source vs. Closed Model cards

Source Models
Fine-Tuned Model Domain-Specific Models
5
Orchestrated Services
Caching Services Optimization Services

1 5
Security Gateways (LLM Plug-ins for Security

2 Gateways) 6
Deployment Services Plug-ins for Customization and

3 7 Integration
Monitoring Services LLM General Agents

4 8
6
AI Applications
• Direct touchpoint between LLM technology and end-users
• Interface for users to interact with the underlying intelligence of LLMs.
• Facilitate seamless communication
• Task automation across various domains.
• Amplify the benefits or risks associated with LLMs
AI Controls Framework:
• To prioritize the governance and oversight of AI applications.

• Can proactively address the challenges and risks associated with LLM-powered applications
• Should facilitate continuous monitoring and evaluation of AI applications
7
LLM-Service Lifecycle
Preparation: Deployment:
○ Data collection ○ Orchestration

○ Data curation ○ AI Services supply chain
○ Data storage
○ AI applications
○ Resource provisioning
○ Team and expertise
Delivery:
Development: ○ Operations
○ Maintenance
○ Design ○ Continuous monitoring
○ Training ○ Continuous improvement
○ Key considerations during development
○ Guardrails
Evaluation/Validation Service Retirement:
○ Evaluation ○ Archiving
○ Validation/Red Teaming ○ Data deletion
○ Re-evaluation
○ Model disposal
○ Key considerations during evaluation/validation
8
LLM Impact Categories
Initial list of high-level impact categories of LLM-related risks:
Loss of Confidentiality Loss of Integrity Loss of Availability
Disruption to the LLM's operation,

Exposed or leaked to unauthorized LLM's data or its generated outputs preventing users from accessing it
individuals. are altered or corrupted. when needed.
E.g. personal data, trade secrets, or E.g. denial-of-service attacks,

other confidential material. Maliciously or accidentally: incorrect system failures, unexpected
or misleading results. downtime, excessive billing quotas
or computational resources.
This could be expanded to include the new categories of ‘Abuse/Misuse’ and ‘Loss of Privacy” (according to the NIST document AI 100-2 E2023)
9
LLM Service Threat Categories
1. Model manipulation
2. Data poisoning
3. Sensitive data disclosure
4. Model theft
5. Model Failure/malfunctioning
6. Insecure supply chain
7. Insecure apps/plugins
8. Denial of Service (DoS)
9. Loss of governance/compliance
10
Acknowledgements
Lead Authors Contributors Reviewers

Phil Alger
Marina Bregkou Ilango Allikuzhi
Siah Burke Vidya Balasubramanian Bakr Abdouh
Vinay Bansal
Avishay Bar
Marco Capotondi Monica Chakraborty
Vijay Bolina
Brian Brinkley
Anton Chuvakin Anupam Chatterjee
Daniele Catteddu Ricardo Ferreira Jason Clinton
Alan Curran
Alessandro Greco Sandy Dunn
Ken Huang Krystal Jackson
Gian Kapoor
David Gee
Zack Hamilton
Vic Hargrave
Kushal Kumar Jerry Huang
Ankita Kumari Rajesh Kamble
Yutao Ma Gian Kapoor
Rico Komenda
Danny Manimbo Vani Mittal
Vishwas Manral Jason Morton
Jesus Luna Ameya Naik
Michael Roza Gabriel Nwajiaku
Meghana Parwate
Lars Ruddigheit Prabal Pathak
Dor Sarig Ruchir Patwa
Amit Sharma Brian Pendleton
Kunal Pradhan
Rakesh Sharma Dr. Matt Roldan
Kurt Seifried Omar Santos
Caleb Sima Dr. Joshua Scarpino
Eric Tierling Natalia Semenova
Bhuvaneswari Selvadurai
Jennifer Toren Jamillah Shakoor
Rob van der Veer Tal Shapira
Ashish Vashishtha Akram Sheriff
Srinivas Tatipamula
Sounil Yu Maria (MJ) Schwenger
Dennis Xu Mahmoud Zamani
Raphael Zimme
11
AI Controls Framework
WG
Website:
https://cloudsecurityalliance.org/research/working-
groups/ai-controls
Circle Community:
https://circle.cloudsecurityalliance.org/community-
home1?communitykey=3e59485a-1cb9-4bf5-
8565-018bad52267d
12

Large Language Model Threats Taxonomy 20240610

Uploaded by

Copyright:

Available Formats

Large Language Model Threats Taxonomy 20240610

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Large Language Model Threats Taxonomy 20240610

Uploaded by

Copyright:

Available Formats

Large Language

Cloud running the training Hybrid and multi-cloud

Cloud running the model Security of the deployment

Cloud running the AI Continuous monitoring, etc.

Foundation Model Open Source vs. Closed Model cards

Fine-Tuned Model Domain-Specific Models

Caching Services Optimization Services

Security Gateways (LLM Plug-ins for Security

Deployment Services Plug-ins for Customization and

Monitoring Services LLM General Agents

• To prioritize the governance and oversight of AI applications.

○ Data collection ○ Orchestration

Evaluation/Validation Service Retirement:

Loss of Confidentiality Loss of Integrity Loss of Availability

Disruption to the LLM's operation,

E.g. personal data, trade secrets, or E.g. denial-of-service attacks,

Lead Authors Contributors Reviewers

You might also like