Purpose

The primary aim of of the OWASP Machine Learning Security Top 10 project is to deliver an overview of the top 10 security issues of machine learning systems. As such, a major goal of this project is to develop a high quality deliverable, reviewed by industry peers.

Target Audience

The primary audience for the deliverables in this project are developers, machine learning engineering and operational practitioners, and application security experts. While each of these roles build, operate and secure machine learning systems, the content is not aimed to be exclusively at them. The content will aim to specify where appropriate the level of understanding required for specific technology domains.

Scope

This project will provide an overview of the top 10 security issues of machine learning systems. Due to the rapid adoption of machine learning systems, there are related projects within OWASP and other organisations, that may have narrower or broader scope than this project. As an example, while adversarial attacks is a category of threats, this project will also cover non-adversarial scenarios, such as security hygiene of machine learning operational and engineering workflows.

Governance

The project will:

• Adhere to the OWASP Project Policy

Project Leaders will:

• Follow and adhere to all OWASP Foundation policies and procedures
• Lead the project as per the Project Leader Handbook

Project Contributors will:

• Follow and adhere to the code of conduct

Related

Top 10 lists related to ML and AI:

Top10 lists similar to famous OWASP Top10 for Web Applications list, but for AI:

• MLSecOps Top10
• OWASP Top10 for Large Language Models

Vulnerability databases:

Catalogued vulnerabilities and risks that were present in real-world AI and ML systems:

• AI Vulnerability Database (AVID)
• MITRE ATLAS
• AI Risk Database

AI/ML security guidelines:

Various guidelines on ML and AI Security and Safety

• OWASP AI Security and Privacy Guide
• ETSI “Securing Artificial Intelligence
• Biden&Harris Administraton - Ensuring Safe, Secure and Trustworthy AI

Playbooks

Interactive playbooks useful in threat modelling and securing AI.

• NIST AI Risk Management Framework Playbook
• Department of Energy AI Risk Management Playbook

Other

All the other resources related to ML Security - threat modelling resources, risk assessments framework, “Awesome Lists” etc.

• Google on Red Teaming AI
• Berryville ML Institute Resources for Threat Modelling ML
• Microsoft AI Risk assessment framework
• ETSI document on securing Artificial Intelligence
• Trusted AI Adversarial Robustness Toolbox
• ENISA - Securing Machine Learning Algorithms
• Awesome AI Security
• Awesome ML Security
• Awesome Attacks on ML Privacy

0 1 2 3 4 5 6 7 8 9 A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

0

1

2

3

4

5

6

7

8

9

A

Adversarial attack
Type of attack which seeks to trick machine learning models into misclassifying inputs by maliciously tampering with input data

B

C

Classification
Process of arranging things in groups which are distinct from each other, and are separated by clearly determined lines of demarcation

D

Data labeling
Process of assigning tags or categories to each data point in a dataset

Data poisoning
Type of attack that inject poisoning samples into the data

Deep learning
Family of machine learning methods based on artificial neural networks with long chains of learnable causal links between actions and effects

E

Ensemble
See: Model Ensemble

F

G

H

I

Input Validation
Input validation is a technique for checking potentially dangerous inputs in order to ensure that the inputs are safe for processing within the code, or when communicating with other components

Intrusion Detection Systems (IDS)
Security service that monitors and analyzes network or system events for the purpose of finding, and providing real-time or near real-time warning of, attempts to access system resources in an unauthorized manner

Intrusion Prevention System (IPS)
System that can detect an intrusive activity and can also attempt to stop the activity, ideally before it reaches its targets

J

K

L

M

MLOps
The selection, application, interpretation, deployment, and maintenance of machine learning models within an AI-enabled system

Model
Detailed description or scaled representation of one component of a larger system that can be created, operated, and analyzed to predict actual operational characteristics of the final produced component

Model ensemble
Art of combining a diverse set of learners (individual models) together to improvise on the stability and predictive power of the model

N

O

Obfuscation
Defense mechanism in which details of the model or training data are kept secret by adding a large amount of valid but useless information to a data store

Overfitting
Overfitting is when a statistical model begins to describe the random error in the data rather than the relationships between variables. This occurs when the model is too complex

P

Perturbation
Noise added to an input sample

Q

R

Regularisation
Controlling model complexity by adding information in order to solve ill-posed problems or to prevent overfitting

S

Spam
The abuse of electronic messaging systems to indiscriminately send unsolicited bulk messages

T

U

Underfitting
Underfitting is when a data model is unable to capture the relationship between the input and output variables accurately, generating a high error rate on both the training set and unseen data

V

W

X

Y

Z