Security Metrics & Monitoring
Security Metrics & Monitoring Framework
This guide provides a comprehensive framework for monitoring, measuring, and improving AI agent skill security through data-driven metrics and analytics.
Key Performance Indicators (KPIs)
Security Metrics
1. Vulnerability Management
Metric: Critical Vulnerabilities Fixed / Total
Target: 100% within 30 days
Tracking: Weekly
CVE Management:
- Published CVEs in skills: [current]
- Fixed this month: [count]
- Pending fixes: [count]
- Average fix time: [days]
2. Malware Detection Rate
Metric: Malicious Skills Detected / Total Skills Published
Target: >95% detection within 30 days of publication
Tracking: Weekly
Samples:
- Total skills scanned: [count]
- Malicious detected: [count]
- False positives: [count]
- Detection accuracy: [percentage]%
3. Security Incident Response
Metric: Mean Time to Respond (MTTR)
Target: <4 hours for HIGH severity
Tracking: Per incident
Response Times:
- CRITICAL: <1 hour (current avg: X min)
- HIGH: <4 hours (current avg: X min)
- MEDIUM: <1 day (current avg: X hours)
- LOW: <1 week (current avg: X days)
4. Publisher Compliance
Metric: Compliant Publishers / Total Active Publishers
Target: >90%
Tracking: Monthly
Compliance Breakdown:
- Signed skills: [percentage]%
- Recent security review: [percentage]%
- Updated documentation: [percentage]%
- No vulnerabilities: [percentage]%
5. User Security Awareness
Metric: Users Who Reviewed Permissions / Total Installers
Target: >70%
Tracking: Quarterly
Survey Data:
- Users reviewing permissions: [percentage]%
- Users verifying publisher: [percentage]%
- Users updating skills: [percentage]%
Dashboard Metrics
Executive Dashboard
For: Leadership, Board, Security Committee
┌─────────────────────────────────────────┐
│ AST10 Security Status - Executive View │
├─────────────────────────────────────────┤
│ │
│ Overall Security Score: 7.8/10 │
│ ████████░░ 78% │
│ │
│ Key Metrics: │
│ • Active Threats: 3 (MEDIUM) │
│ • Avg Incident Response: 2.2 hrs │
│ • Publisher Compliance: 87% │
│ • Skills with Issues: 0.3% │
│ │
│ Trend (30-day): ↑ Improving │
│ │
│ Action Items: 2 │
│ 1. Update 5 outdated policies │
│ 2. Complete Q2 security audit │
│ │
└─────────────────────────────────────────┘
Operations Dashboard
For: Security Operations, Platform Teams
┌──────────────────────────────────────────────┐
│ AST10 Operations - Real-Time Monitoring │
├──────────────────────────────────────────────┤
│ │
│ Skills Published (24h): 127 │
│ ├─ Scanned: 127 (100%) │
│ ├─ Passed: 121 (95%) │
│ └─ Flagged: 6 (5%) │
│ │
│ Active Incidents: │
│ ├─ CRITICAL: 0 │
│ ├─ HIGH: 1 (Incident INC-2026-0042) │
│ ├─ MEDIUM: 3 │
│ └─ LOW: 8 │
│ │
│ Scanning Performance: │
│ ├─ Avg Scan Time: 2.3 sec │
│ ├─ Malware Detection Rate: 96.2% │
│ └─ False Positive Rate: 0.8% │
│ │
│ Platform Health: │
│ ├─ OpenClaw: ✓ Healthy │
│ ├─ Claude Code: ⚠ 2 alerts │
│ ├─ Cursor: ✓ Healthy │
│ └─ VS Code: ✓ Healthy │
│ │
└──────────────────────────────────────────────┘
Developer Dashboard
For: Skill Developers, Security Team
┌────────────────────────────────────────┐
│ My Skill Security Status │
├────────────────────────────────────────┤
│ │
│ Published Skills: 12 │
│ ├─ All Compliant: 10 │
│ ├─ Minor Issues: 2 │
│ └─ Critical Issues: 0 │
│ │
│ Latest Scans: │
│ ├─ 2026-03-22 10:15: PASS ✓ │
│ ├─ 2026-03-21 14:30: PASS ✓ │
│ ├─ 2026-03-20 09:00: WARN (2 issues) │
│ └─ 2026-03-19 16:45: PASS ✓ │
│ │
│ Security Training Status: │
│ ├─ Completed Courses: 3/5 │
│ ├─ Certification: AST10-SE Analyst │
│ └─ Expires: 2027-03-22 │
│ │
│ Recommendations: │
│ 1. Update deprecated library in skill │
│ 2. Review and adjust permissions │
│ │
└────────────────────────────────────────┘
Monitoring Infrastructure
Collection Points
┌─ Skill Registry ────────────────────────┐
│ Events: │
│ • Skill published │
│ • Skill updated │
│ • Skill removed │
│ • Download count │
│ • User reports │
└────────────────────────────────────────┘
│
↓
┌─ Security Scanners ─────────────────────┐
│ AST10-Scanner findings: │
│ • Vulnerability detections │
│ • Malware classifications │
│ • Permission analysis │
│ • Supply chain assessment │
└────────────────────────────────────────┘
│
↓
┌─ Runtime Monitoring ────────────────────┐
│ Agent execution tracking: │
│ • Skill invocations │
│ • Permission usage │
│ • Network activity │
│ • File system access │
│ • Performance metrics │
└────────────────────────────────────────┘
│
↓
┌─ Analytics Platform ────────────────────┐
│ Aggregation & Analysis: │
│ • Trend analysis │
│ • Anomaly detection │
│ • Risk scoring │
│ • Correlation analysis │
└────────────────────────────────────────┘
│
↓
┌─ Visualization & Alerting ──────────────┐
│ Dashboards & Notifications: │
│ • Real-time dashboards │
│ • Automated alerts │
│ • Reports & trends │
│ • Escalation workflows │
└────────────────────────────────────────┘
Data Collection
import metrics
class SkillMetricsCollector:
def __init__(self):
self.metrics = metrics.MetricsClient()
def record_skill_event(self, event_type, skill_id, metadata):
"""Record skill lifecycle events"""
self.metrics.increment(
'skill.events',
tags={'type': event_type, 'skill_id': skill_id}
)
self.metrics.gauge(
'skill.downloads',
metadata.get('download_count'),
tags={'skill_id': skill_id}
)
def record_scan_result(self, scan_result):
"""Record security scan results"""
self.metrics.increment(
'scans.total',
tags={'status': scan_result.status}
)
self.metrics.histogram(
'scans.duration_ms',
scan_result.duration_ms
)
for finding in scan_result.findings:
self.metrics.increment(
'findings',
tags={'severity': finding.severity}
)
def record_incident(self, incident):
"""Record security incidents"""
self.metrics.increment(
'incidents',
tags={'severity': incident.severity}
)
self.metrics.histogram(
'incident.response_time_minutes',
incident.response_time
)
Alerting Rules
Critical Alerts
Alert: MaliciousSkillDetected
Condition: Malware confidence > 90%
Action:
- CRITICAL incident created
- Skill immediately flagged
- Security team paged
- Incident response activated
High Severity Alerts
Alert: UnexpectedPermissions
Condition: Skill requests unusual permission combination
Action:
- HIGH incident created
- Manual review queued
- Publisher notified
- Enhanced monitoring enabled
Medium Severity Alerts
Alert: VulnerabilityFound
Condition: CVE published affecting skill dependency
Action:
- MEDIUM incident created
- Affected publishers notified
- Patch available communication sent
- Tracking enabled
Low Severity Alerts
Alert: ComplianceGap
Condition: Publisher documentation outdated (>90 days)
Action:
- LOW priority ticket created
- Publisher sent reminder email
- Tracking enabled
Trend Analysis
Quarterly Report Example
Q1 2026 Security Report
========================
Executive Summary:
- 1,247 new skills published
- 12 malicious skills detected (0.96%)
- 23 security incidents resolved
- 87% publisher compliance
Trend Analysis:
- Malware Detection: ↑ 15% (improved detection)
- Response Time: ↓ 25% (faster response)
- Publisher Compliance: ↑ 5% (better education)
- User Awareness: ↑ 12% (more training)
Key Findings:
1. Supply chain attacks increasing (trend analysis)
2. Permission escalation most common vulnerability
3. Platform A has 3x more issues than B
4. Certain publisher type has 5x incident rate
Recommendations:
1. Increase supply chain scanning
2. Add permission validation warnings
3. Platform A needs hardening
4. New publisher education program
Anomaly Detection
Machine Learning Models
Model 1: Skill Behavior Analysis
- Detects unusual permission patterns
- Identifies obfuscation techniques
- Flags suspicious code structures
- Accuracy: 94.2%
Model 2: Publisher Assessment
- Analyzes publisher history
- Detects risky publisher patterns
- Predicts future incidents
- Accuracy: 87.6%
Model 3: Network Traffic Analysis
- Detects unusual outbound connections
- Identifies data exfiltration
- Flags C2 communication
- Accuracy: 96.1%
Alert Confidence Scoring
Score: 0-100 (higher = more confidence)
85-100: CRITICAL - Immediate action
70-84: HIGH - Urgent investigation
50-69: MEDIUM - Schedule review
25-49: LOW - Monitor
0-24: INFO - Log only
Examples:
- Skill connects to known C2: 98/100 → CRITICAL
- Suspicious permission combo: 62/100 → MEDIUM
- Rare library usage: 35/100 → LOW
Reporting
Automated Reports
Daily Security Summary
Date: 2026-03-22
Skills Published: 127
Security Scans: 127 (100%)
Passed: 121 (95.3%)
Issues Found: 6 (4.7%)
Malware Detected: 0 (0%)
Incidents Opened: 2
Incidents Closed: 1
Weekly Trends
New Malicious Skills: 3 (↓ 40% from previous week)
Publisher Violations: 5 (↑ 25%)
User Complaints: 12 (↓ 15%)
Security Training: 34 enrolled (↑ 5%)
Monthly Report
- 30-day trends
- Emerging threats
- Successful mitigations
- Performance metrics
- Recommendations
Continuous Improvement
Metrics Review Cycle
Weekly:
- Alert review
- Incident metrics
- Detection rates
Monthly:
- Dashboard review
- Trend analysis
- Alert tuning
Quarterly:
- Full metric assessment
- Model retraining
- Strategy adjustment
Annually:
- Goals review
- Program evaluation
- Long-term trends
Benchmarking
Industry Benchmarks (2026):
- Malware detection rate: 85-95%
- Incident response time: 2-6 hours
- Publisher compliance: 70-85%
- False positive rate: 1-3%
Our Performance:
- Detection rate: 96.2% ✓
- Response time: 2.1 hours ✓
- Compliance: 87% ✓
- False positives: 0.8% ✓
Status: Above industry average
Tools & Platforms
Recommended Stacks
Collection:
- Prometheus (metrics)
- Elasticsearch (logs)
- Jaeger (tracing)
Analysis:
- Splunk / ELK
- Grafana
- Datadog
Alerting:
- PagerDuty
- Opsgenie
- Slack
Visualization:
- Grafana
- Kibana
- Splunk
Metrics framework updated: March 2026. Review quarterly with leadership.
Example
Put whatever you like here: news, screenshots, features, supporters, or remove this file and don’t use tabs at all.
Leadership & Founding Members
Project Leadership
Current Leaders
Ken Huang
Hammad Atta
Fabio Cerullo
Aonan Guan
Bhavya Gupta
Niv Hoffman
Iftach Orr
Akram Sheriff
AIVSS Distinguished Review Board
The OWASP AIVSS project’s Distinguished Review Board comprises world-renowned cybersecurity leaders, former government officials, and industry pioneers who provide strategic guidance and expert oversight for the AI Vulnerability Scoring System framework. We thank them for their guidance, several of whom have also supported this project’s work.
Rob Joyce
Advisor to PwC and OpenAI, Former Special Assistant to the President and Cybersecurity Coordinator
Jason Clinton
Deputy CISO, Anthropic
Amy R. Steagall
Chief Information Security Officer, Stanford University
Martin Stanley
AI Risk Management Framework Lead, NIST
Apostol Vassilev
Research Supervisor, NIST
Andrew Coyne
CISO, Banner Health, Former CISO, Mayo Clinic
Kevin Rocque
Managing Director/Executive Vice President, Global Technology Risk Officer, TD Bank
Jeff Williams
Former Global OWASP Chair, Founder and CTO, Contrast Security
Michael Tran Duff
University Chief Information Security and Data Privacy Officer, Harvard University
Emil Bender Lassen
Standards Lead, AIUC-1
Agentic Skills Top 10 Founding Members
Founding members of the OWASP Agentic Skills Top 10 project itself — project leads, co-leads, and additional contributors — listed alphabetically. Several also contribute to the sibling OWASP AIVSS project listed above.
Ken Huang
Project Lead, Agentic Skills Top 10
Hammad Atta
Co-Lead, Agentic Skills Top 10
Manish Bhatt
Security Researcher, AWS
Fabio Cerullo
Co-Lead, Agentic Skills Top 10
David Girard
Senior Director, AI Security & AI Alliances, Trend Micro
Aonan Guan
Co-Lead, Agentic Skills Top 10
Bhavya Gupta
Co-Lead, Agentic Skills Top 10
Pamela Gupta
Founder & CEO, OutSecure / Trusted AI
Idan Habler
Staff AI/ML Security Researcher, Intuit
Niv Hoffman
CTO, Air Security
Charles Iheagwara
AI/ML Security Leader, AstraZeneca
Sushmitha Janapareddy
Director - Security Integrations, American Express
Edward Lee
Vice President, Lead AI Security, JP Morgan
KJ Lian
Senior Manager, Data & AI (Public Sector), AWS
Vineeth Sai Narajala
Application Security, AWS
Iftach Orr
Co-Lead, Agentic Skills Top 10
Kanna Sekar
Cyber Security, Google
Akram Sheriff
Co-Lead, Agentic Skills Top 10
Dennis Xu
Research VP, AI, Gartner
OWASP AIVSS Founding Members
The OWASP AIVSS (Agentic AI Vulnerability Scoring System) project is a sibling OWASP initiative focused on scoring the severity of agentic AI vulnerabilities. Its founding members are recognized here as OWASP founding members in the agentic AI security space; many of them have also contributed directly to the Agentic Skills Top 10 project’s research and review process.
Sunil Agrawal
Chief Information Security Officer, Glean
David Ames
Partner, PwC
Michael Bargury
Founder and CTO, Zenity
Joshua Beck
Application Security Architect, SAS
Manish Bhatt
Security Researcher, Amazon Kuiper Security
Mark Breitenbach
Security Engineer, Dropbox
Anat Bremler-Barr
Professor of Computer Science, Tel Aviv University
Siah Burke
HIPAA Security Officer, Siah.ai
David Campbell
AI Security, Scale AI
Ying-Jung Chen
AI safety researcher, PhD, Georgia Institute of Technology
Anton Chuvakin
Security Solution Strategy, Google
Jason Clinton
CISO, Anthorphic
Adam Dawson
Staff AI Security Researcher, Dreadnode
Leon Derczynski
Principal Research Scientist, NVIDIA
Walker Lee Dimon
AI Security Researcher, MITRE
Marissa Dotter
AI Security Researcher, MITRE
Dan Goldberg
ISO Market Lead, Omnicom
David Haber
CEO, Lakera
Idan Habler
Staff AI/ML Security Researcher, Intuit
Jason Haddix
Founder, Arcanum Information Security
Keith Hoodlet
Director of AI/ML & AppSec, Trail of Bits
Ken Huang
AIVSS Project Lead, OWASP
Chris Hughes
CEO, Aquia
Charles Iheagwara
AI/ML Security Leader, AstraZeneca
Krystal Jackson
Researcher, Center for Long-Term Cybersecurity, UC Berkeley
Sushmitha Janapareddy
Director - Security Integrations, American Express
Rob Joyce
Former Cybersecurity Director of NSA, Advisor to PwC, PwC
Diana Kelley
CISO, Noma Security
Prashant Kulkarni
Lead AI Security Research Engineer, Google Cloud
Mahesh Lambe
Founder, MIT, Unify Dynamics
Edward Lee
Vice President, Lead AI Security, JP Morgan
Nate Lee
CEO, Cloudsec.ai
Vishwas Manral
CEO, Precize.ai
Daniela Muhaj
Executive-in-Residence for Research & Development, AI 2030
Vineeth Sai Narajala
Application Security, AWS
Om Narayan
AI Security Researcher, AWS
Varun Pant
Engineering and Product Leader, AI applications at the Automated Reasoning Group, AWS
Advait Patel
Senior Site Reliability Engineer (DevSecOps + Cloud + AIOps), Broadcom, IEEE
Alex Polyakov
CEO, adversa.ai
Ramesh Raskar
Professor & Director, MIT Media Lab
Ron F. Del Rosario
VP-Head of AI Security, SAP
Tal Shapira
Co-Founder & CTO, Reco AI
Akram Sheriff
Senior AI/ML Software Engineering Leader, Cisco
Samantha Siau
Security and Compliance, Anthropic
Kevin Simmonds
Partner on AI Offensive Security, PWC
Martin Stanley
NIST AI RMF Lead, Independent
Omar A. Turner
General Manager of Security, Microsoft
Apostol Vassilev
AI Research Team Supervisor, NIST
Matthew Versaggi
AI Fellow, White House Presidential Innovation Fellow
David Webb
Agency Cybersecurity Officer, Cybersecurity and Infrastructure Security Agency
Dennis Xu
Research VP, AI, Gartner
Xiaochen Zhang
Executive Director and Chief Responsible AI Officer, AI 2030
Recognition
We extend our gratitude to all founding members who have contributed to establishing this crucial framework for AI security assessment. Their vision and dedication have been instrumental in shaping the Agentic Skills Top 10 project.
Get Involved
Interested in contributing to the Agentic Skills Top 10 project? We welcome new contributors and leaders. Please see our Contribution Guidelines for more information on how to get involved.