Cloudflare's Security Audit Skill, Tested Against the Rest

0:00Cloudflare just open-sourced a security-audit-skill — the same one it uses on its own software factory — and the timing is sharp: AI models and reverse engineering have made hacking dramatically easier. Rob Shocks builds a deliberately broken app with six planted holes and runs the new skill against the free scanners, Claude Code's /security-review, and a bare prompt.

Why it matters

0:45Agents are good at textbook security but lazy about authorization: they'll check you're logged in, then forget to check the record actually belongs to you. The cited evidence: last year a researcher found more than 170 Lovable-built apps leaking data because nobody set up row-level database rules (Lovable's fair response: security is the customer's responsibility). His prediction going in: scanners would catch the classic bugs, and you'd need an agent to catch the AI-written ones.

The test app

1:37A small Next.js app, "Gleecast," with two companies (Acme and Globex) and six planted bugs: three classics (SQL injection in search, a committed-then-deleted secret key, an old Next.js version with a known middleware bypass) and three the kind an agent would write (Acme can read Globex's posts, anyone can delete any post, export dumps every company's data). Plus one decoy that looks dangerous but is safe — any tool that flags it loses a point. He wrote a working exploit for every bug, and each tool ran in its own sandbox.

Round 1: free scanners

2:33Gitleaks scans the whole Git history and found the deleted key (three commits back) in under a second — a reminder that deleting a key doesn't remove it from history. Trivy checks dependencies and instantly flagged the old Next.js (the middleware bypass) — but it told him the package was bad without telling him the admin page depends on middleware alone, which is half the bug. Neither caught the SQL injection, because neither actually reads your code. Both: 2 of 6.

Round 2: /security-review

3:36Claude Code's built-in /security-review reviews what just changed, so he fed it the whole app as one change (normally it only checks the latest diff). It found 4 of 6 — the SQL injection, cross-company read, delete, and export — in ~84 seconds for 16 cents. It missed the middleware bypass and the git key (it doesn't see git history). The docs note it skips rate-limiting by design and isn't hardened against prompt injection, so be careful on other people's PRs.

Round 3: Cloudflare's skill

4:30The Cloudflare skill works differently: it maps the app, then sends "hunters" at each attack type, and a separate agent verifies each finding — never the one that found it. The design principle is worth quoting: "if a hunter is allowed to grade its own homework, it will confidently validate everything it outputs." On this tiny app it ran zero hunters against zero verifiers (it marked its own run incomplete), yet it still named all six bugs — the only tool to do so, including the git key — in ~5 minutes for 38 cents, and it proved the SQL injection on a dummy database. The other five came back "needs validation" because proving them requires dedicated lockdown sandboxes. The honest caveat: Cloudflare says a full run finds only about half the bugs, so thorough coverage means running it several times — and that costs.

The plain-prompt control

6:37Just to check whether models need all this scaffolding, a bare one-sentence prompt — "review the codebase for security vulnerabilities, name the file and what an attacker gets" — found 5 of 6 in 34 seconds for under a cent, missing only the git key (it never looked at history) and correctly calling the decoy safe. But he's quick to note this is a tiny app the model can read in one go with three near-identical bugs; at real scale, parallel fan-out agents are what you need.

The scoreboard

8:11The through-line: scanners catch what lives outside your code (git history, dependency lists), and AI tools catch the logic bugs — you need both. The twist: all three AI tools flagged a bug he didn't plant — a hardcoded session password that the AI agent (Opus 5.5) had rolled in while helping build the app. The agent nailed the textbook stuff and then hardcoded the password protecting every session. Gitleaks missed it. It's the argument for defense in depth in one anecdote.

Three levels of defense

8:59A practical ladder:

• Level 1 (this week): run Gitleaks over your whole history + add a pre-commit hook; put Trivy/Dependabot on dependencies; before shipping, run /security-review (or the plain prompt); bake recurring agent mistakes into CLAUDE.md/AGENTS.md (e.g. "always scope every query to the organization").
• Level 2 (this month, per PR): /security-review on every trusted PR, Gitleaks + Trivy in CI, and header checks via Mozilla Observatory.
• Level 3 (before launch / touching billing): run the Cloudflare skill in a sandbox where it can execute code, verify the verifiers actually ran, run it a couple of times; then point an AI pen tester at staging, not production; and for the highest stakes, pay a professional human.

Claims checked

Sources & further reading