Interesting and early. Worth a spike or exploration session.
AI Red Teaming
AI red teaming is professionalising fast — OffSec's AI-300 certification and the EU AI Act August 2026 deadline are making it a compliance requirement, not just a best practice.
Evaluation·Regulation
atlas.mitre.orgOur Take
What It Is
AI red teaming is the practice of systematically testing AI systems for vulnerabilities, biases, harmful outputs, and failure modes. It goes beyond traditional software security testing to cover AI-specific risks: jailbreaks, prompt injection, data poisoning, bias amplification, and model manipulation. Frameworks include MITRE ATLAS, NIST AI RMF, and OWASP's LLM Top 10.
Why It Matters
AI red teaming stays in Emerging, but the professionalisation signals are strong. OffSec (the people behind OSCP, the gold-standard penetration testing certification) launched AI-300, the first professional certification for AI security testing. When OffSec enters a category, it's being taken seriously by security professionals, not just researchers.
The research is sobering: roleplay-based attacks succeed 89.6% of the time against current models. For organisations deploying AI systems that handle sensitive data or make consequential decisions, the question isn't whether to red team — it's whether you can afford not to. The EU AI Act makes this explicit: full compliance for high-risk AI systems is required by August 2026, and conformity assessments will expect documented adversarial testing.
Key Developments
- Mar 2026: OffSec launches AI-300 certification — first professional credential for AI security testing.
- Feb 2026: Research shows roleplay-based attacks achieve 89.6% success rate against frontier models.
- Jan 2026: EU AI Act high-risk system rules set for August 2026 enforcement, requiring documented adversarial testing.
- Dec 2025: MITRE ATLAS and NIST AI RMF converge on standardised red teaming methodologies for enterprise adoption.
What to Watch
The August 2026 EU AI Act deadline will create demand for AI red teaming services and tooling. Watch for whether automated red teaming tools mature enough to complement manual testing — current tools find obvious issues but miss the creative attacks that human red teamers discover. The 89.6% roleplay attack success rate is a benchmark to track; if frontier model safety improvements bring this below 50%, it signals real progress.
Strengths
- Professional credentialing: OffSec AI-300 certification brings AI security testing into the mainstream security profession.
- Regulatory tailwind: EU AI Act, NIST AI RMF, and ISO 42001 all require or recommend adversarial testing of AI systems.
- Standardised frameworks: MITRE ATLAS and OWASP LLM Top 10 provide structured approaches to AI-specific threat modelling.
Considerations
- Talent scarcity: AI red teaming requires both security expertise and AI/ML knowledge. The intersection is a small talent pool.
- Tooling immaturity: Automated AI red teaming tools catch surface-level issues but miss creative adversarial attacks that require human judgment.
- Moving target: Model updates change the vulnerability surface. Red team results from January may not apply to a March model update.
- Cost and time: Thorough AI red teaming is expensive and time-consuming. Most organisations underinvest relative to the risk exposure.
Resources
Documentation
Adversarial Threat Landscape for AI Systems — structured knowledge base of AI attack techniques.
NIST's comprehensive framework for AI risk management including adversarial testing.
Top vulnerability categories specific to LLM applications.
First professional certification for AI security testing from the makers of OSCP.