AWS Launches Deception Benchmark for AI Security Tool Evaluation
AWS has released the Deception Benchmark, a new tool designed to measure the defensive precision of AI security models. This benchmark tests whether AI can accurately distinguish between real vulnerabilities and code that appears risky but is actually safe, addressing the issue of excessive false alarms in AI-driven security analysis. It aims to build trust in AI security tools by providing a rigorous evaluation of their ability to reason about code, mitigations, and context. The benchmark comprises 14,822 samples across 16 languages and over 70 Common Weakness Enumeration (CWE) categories, and its dataset and whitepaper are available to the community.
Features (1) ›
- AWS Releases Deception Benchmark for AI Security Model Evaluation
The new Deception Benchmark assesses how accurately AI security models differentiate between real vulnerabilities and benign code with deceptive, safe patterns. It comprises 14,822 samples across 16 languages and over 70 CWE categories, providing a resource for the community to rigorously evaluate the defensive precision of AI security tools. Initial evaluations of 12 models revealed that precision in distinguishing true vulnerabilities from false alarms was in the mid-50s under standard prompting.
https://aws.amazon.com/blogs/security/the-state-of-ai-for-security-measuring-what-matters-most-for-building-trust/
Related releases
- Terraform AWS Provider v6.64.0 Adds New Resources and Bedrock Agent Enhancements Terraform AWS Provider Releases ·
- AWS Lambda now supports 90-minute function timeouts on Managed Instances AWS What's New ·
- AWS Lambda Managed Instances now support Graviton5-powered EC2 instances AWS What's New ·
- Amazon EMR Extends No-Cost Support for Older Releases During Migrations AWS What's New ·
- AWS Systems Manager Enhances Diagnosis for Unmanaged EC2 Instances AWS What's New ·
- Amazon Bedrock Knowledge Bases Adds Native Connector for Confluence Data Center AWS What's New ·