We are happy to share that we have started working with Reflection. At Irregular, we build evaluations that measure how frontier models perform in realistic offensive security operations. As part of ...

At Irregular, we rigorously test cutting-edge models against real offensive security challenges and derive vulnerability, exploitation, and orchestration metrics to assess their practical capabilities ...

LLM-generated passwords appear strong, but are fundamentally insecure. Testing across GPT, Claude, and Gemini revealed highly predictable patterns: repeated passwords across runs, skewed character ...

We studied a self-hosted system in which the same open-weights model powered both a coding agent and an AI application that the coding agent was asked to maintain. This architecture is particularly ...

TL;DR: This blog shares key findings from our investigation into a recently publicly reported incident involving one of our evaluation environments. Importantly, all subsequent public disclosures ...

Autonomous AI agents are increasingly being deployed for offensive security tasks. These models can scan codebases in seconds, identify misconfigurations at machine speed, and execute tactical exploit ...

AI agents deployed for routine enterprise tasks are autonomously hacking the systems they operate in. No one asked them to. No adversarial prompting was involved. The agents independently discovered ...

Existing cybersecurity benchmarks are gradually becoming saturated across frontier models, offering diminishing insight into real-world risk. As LLMs begin to influence security incidents in the wild, ...

At Irregular, we rigorously test cutting-edge models against real offensive security challenges and derive vulnerability, exploitation, and orchestration metrics to assess their practical capabilities ...

FrontierCyber is a new benchmark that turns real-world systems into offensive cyber capability evaluations for AI models. Models are given verifiable objectives on real systems, from software and ...

At Irregular, we test frontier models against realistic offensive security challenges to measure their performance. As part of our collaboration with Meta, we recently evaluated Muse Spark 1.1 across ...

Sign up for our newsletter and get notified of irregular updates ...