Letting inmates run the asylum: Using AI to secure AI

Summary

One of Anthropic's quieter releases recently was their "Security Review," where Claude Code can identify and fix security issues in your code. But how good is it really? In my case, will it find issues with code it helped me write for my newsletter service and Chrome extension?The release states it uses a "specialized security-focused prompt that checks for common vulnerability patterns." After throwing so much compute at model training, LLMs are nearing the top of the S-Curve, so finding ways to use their existing capabilities is becoming more important. In this case, a special prompt results in a new feature, much like OpenAI used a carefully-crafted prompt to release Study Mode in ChatGPT.So is any of my code vulnerable?First, I had it take a peek at Simple Wikiclaudia, the browser extension I wrote with Claude to simplify Wikipedia articles. One of the extension's biggest features is how simple it is, so I didn't expect it to find much. But how much can you trust the AI that wrote most of the code in the first place?Appropriately found on a blog about o11yAdditionally, it looks like Claude's security review mainly targets low-hanging-fruit included in the OWASP Top 10 (new version coming soon!). But what if there's something else the security review can't catch, maybe something specific to browser sandboxes and web extensions? Or other unknown unknowns? After all, Claude said all my stuff was peachy keen:Claude: Well of course it's fine. I wrote it!It all comes back to Defense in Depth. Do I think this feature has value? Of course it does. Would I rely solely on this LLM review before shipping to production? Of course not. So what else can you do? Human code review, Static Application Security Testing, Dynamic Application Security Testing, QA testing, fuzz testing, testing, testing, testing. And you still won't catch everything!OK but is this code secure or notFortuitously, I recently started playing around with a free trial of Datadog. They just let anybody sig...