Death by small bugs: securing code in the Vulnpocalypse, with Patrick Collins and Simon Harloff

The issues that we saw come through would make your toes curl.

Oh, my app works, but everyone can read everyone's data, for instance.

The things that we have to comply with are not the things that we're worried about.

The best model depends on your definition of best. Time, speed to run, cost, ability to find the vulnerabilities that we're talking about, and the noise that it generates along the way as well.

Yeah, so I think this is just going to keep happening. First happened in Curl and then FFmpeg and then Linux and Firefox. It's now happening in PHP. It's just going to keep happening. And everyone's talking about the frontier models and their guardrails. We try to ask the frontier models to find vulnerabilities and they just won't comply because of their guardrails. Enter the open weights models from China. DeepSeek's incredibly effective. It'll comply with our requests to try to hack if we want. And that's very helpful for us as defenders to try to find vulnerabilities. But it's equally as dangerous in the hands of attackers. And so this kind of stringing together of complex vulnerability chains in open source software is just going to accelerate under the presence of new, more powerful cyber capable open weights models.

I was thinking about like PHP for Shell, right? And if we look at what happened there is that there was an issue that was known at a time where it was human labor to fix the issue and it was probably, you know, lower priority by the time they worked out it couldn't be exploited. My message to people is when the human effort involved in fixing even, you know, medium and lower class of bugs, maybe that gets found, drops to zero. Just put in the time to close each issue out so you don't get caught with your pants down. When someone eventually works out how to chain together like three low class security bugs that have been sitting in your backlog for a long time. So I think this is the time where people should be trying to burn through their backlogs so that they don't get caught off guard

and use agents to burn through your backlogs. Or especially like if you've got a tool that's found a bunch of vulnerabilities even if they seem like they're low, get on them because a bunch of lows chain together into highs. There's another angle to your question, you know, which is all of the software that I depend on, you know that proverbial meme with the like down at the bottom of the XKCD meme, where by the bottom, at the bottom like little component right now says PHP, the proverbial component that the entire Internet relies on. That'll keep happening. It'll be DNS or NTP or whatever. Is the next thing that where there's some like critical vulnerability. I think we have to stay responsive. We need to accept that we're need to patch any part of our supply chain at any point in time and stay responsive to that. So this idea that there are dark corners of my old application that I don't need to worry about anymore because very few people use it, it's over. If it's on the Internet and it's exposed, then you're now responsible for keeping that entire supply chain up to date.

Typed languages, please just

No JavaScript, no PHP. The LLM just reasons better. With typed languages it's like built in linting and one shot, your JavaScript do JavaScript stuff over to TypeScript. You can do it, just make it happen. I'm being facetious there, but like if you're starting something new, you know, like no Ruby, no JavaScript and actually we were a CTO. I was at a CTO dinner last night with some of like leading CTOs sitting around the table. This was actually their opinion, like just get rid of the, get rid of the scripting languages, like right now. Make it happen. And they're literally one shotting things. Sit down, like just let Claude run for, you know, 40 hours, 100 hours if needed, and just make it happen. It'll ultimately dramatically increase your security profile overnight because the LLMs just are much more effective at the typed languages, even if they're less readable and they kind of are less readable. I find TypeScript kind of hard to read, but like, whatever. Probably because I'm not. Who

reads code anymore? Exactly. What does it matter? Like, it's not for me. Simon, didn't your team just like actually put a linting rule in to rip comments out of the code because it kept tripping up the LLMs as well?

Yeah, yeah, we've been really clamping down on like there's like a one line code change and then 50 lines of prose from Claude documenting every single scenario. So yeah, we've been deploying linting rules just to slap the agent's hand, so to speak. Right. And they're security specific ones tailored to our code base. So every time an edit happens you can run these linting rules and be like, no, don't do that. And so you can very, very quickly get to a better outcome. I want to touch also on your new point around the like infrastructure. I feel like it's a bit of a double edged sword because I look at railway.dev I don't know whether you guys saw that got recently popped. I've always been of the mindset of like, you know, maybe one less server is better to manage and you know, not having to manage your own SSL certs and your own firewall rules and VPCs and all of that kind of jazz. So I do think there's a lot of merit to serverless. I do think you're probably safest on still the big hyperscalers or maybe a tier 2, a weld lane tier 2 provider like a Vercel, for instance.

Yeah, definitely. I mean, I think Cloudflare, like you said, knows their stuff. All they do all day long is security networking. Like they've got deep roots in all of this. There's a lot of infrastructure providers popping up like everywhere and then you go like Yep, sure. I'm now no longer managing my app. It's just basically a GitHub repo that magically spins into action. And I think that magic can sometimes be hurtful when the underlying platform might get popped and all your credentials get stolen. So it is a double edged sword. But I think you've got safe harbor in these kind of bigger scale providers for sure.

One way to process what Simon's saying is vanilla and simple is best. Think about it this way. The LLMs are trained on all the stuff that's on the open Internet and the more common and popular that stuff is, the more likely you are to get it right. And so vanilla is good right now. Vanilla TypeScript. Why go with obscure language choices that have poor training data for the LLMs and probably have less, just overall less training data when you can go simple and vanilla. There was once a period of time where cool tech, modern tech was the right way to go. And it was normally through the lens of sometimes like principles like DRY and you know, keeping the code simpler and that, that doesn't matter anymore. It doesn't matter if the code's verbose, it doesn't matter if it, if it takes more lines of code and is more, you know, is harder to read. It doesn't, doesn't matter anymore. What matters is that an LLM can lint it correctly. It's got plenty of source material. Whether Cloudflare is good or not is through the lens of like, is the training material good enough? I would probably say is that brand new service that Cloudflare is deploying the one you want to use? Are you sure? Because it's got less like, just general visibility to your, to your LLM. Pulling back to your broader question of like how to, how to keep myself secure in these days, I would say first principles. Every PR you push has got to have a security review on it. Most customers we talk to SOC 2 or getting towards SOC 2 or ISO compliance and it's pretty easy to check the box on those things without getting the basics in place. So PR scanning at a minimum. And also I'm a big believer in automated and always on red teaming for your application. If you don't know what red teaming is, I'm using that word instead of pen testing. So the old school used to be we'd run an annual pen test and some kind of junior folk at a consulting firm would come in and poke, poke a couple of holes,

try to get through your proverbial firewall. And I think we can all agree that was a pretty ineffective way of trying to find security issues. But there's modern tooling that constantly red teaming your software. And red teaming means constantly trying to find the issues that would come through a code scanning tool like us and then probing through the front door to try to adversarially exploit those issues. Hack

Hack it constantly. Always token expensive, but make it happen. Yeah. And try to guide that.

We're not a red team vendor by the way, so I have no skin in the game here. I recommend that as an always on best practice. There's a variety of vendors in this space, also a variety of open source. I would say that given the explosion of investment into the cybersecurity space with companies operating and targeting small businesses or enterprises or mid market, I literally think you'd be crazy to build your own. You could maybe try to vibe code up your own red teaming thing, but why would you do that? It's not your core business. And actually what makes you think that you know how to build a good security tool anyway? Just because you've been a good engineer. Like let me just pressure test that. So come to our side of the equation where we do pull request reviews constantly. We are always talking to security teams, platform engineering teams often who are like, oh, we've built our own bot and when they see our demo and they understand what we've had to build, they tend to freak out because they realize there's a lot of edge cases and it's hard to make it work smoothly in a way that works for your developers and doesn't slow them down. Like for instance, Simon's team have built nine sub agents specifically good at knowing how to craft the kinds of vulnerabilities in different like in the OAuth categories, different security categories have different tools, different skills. We even use different models per language and we've got a whole evaluation suite around that. So it's just like, well, you know, it's just up to you whether you think you can build that. Now we're just one vendor, there's quite a few vendors. I'm just challenging the idea that you would DIY something. No,

Well, Supabase is loving life as well right now because if you fire up a new application like Supabase is now the default. It is actually a vendor. It's, I mean it's like, wow.

Well, there's some local Aussie startups, Harmony Intelligence for instance, one of them Shinobi. And then you've got bigger players like Exeo for instance that just do this continual style exploitation campaigns, you know, where they'll look at the code, theorize an exploit and then try and kind of go work from the outside in. Typically it requires a little bit of investment as well in terms of like, hey, do you have a live running environment? Like it's the old pen testing thing, rears its head again, it's like, where's your staging environment? How do people log in? Oh, you can only log in with social login providers, so you'd want to get that kind of sorted, which we've done on our end.

There's even specialist red teaming software specifically for attacking your LLM bots. If you're building an LLM bot or your LLM applications is one called Promptfoo, which is specifically designed to help you secure through adversarial attacking, secure your LLM applications you're building as well. So there's a variety of them out there.

The other thing I just wanted to just touch on very quickly is like for those people who will ultimately try, I think they'll very quickly run into what we run into every day, which is just like flat out refusals. So the moment any model gets any sense that you're trying to do something malicious, even if it's as simple as, hey, explain to me how this issue, how someone could exploit it, for instance, which is something like, that's like the root of the question that we try and productize in some of our pipeline. You'll just get refusals from all of the frontier labs. And so you have to use Qwen 2.5, you have to use DeepSeek. It's just not possible to go deeper without being like a, I don't know, a Palo Alto Networks who's on the version of the model that has all the guardrails removed.

That's exactly it. Yep. The model refusal is so frustrating right now and it's getting harder and harder. We're fortunate as a vendor that we're typically a partner of the frontier models, and so we can use their models that are less guarded. We'll also use the open weights models if our customers support it. We just have generally got more experience navigating around these guardrails as well, even though it's still very frustrating.

We build evaluation suites around a few specific tasks. One of them, which we publish a benchmark on, it's on our blog, is specifically the task of determining whether a pull request has a security issue in it. So if you're like building, pushing pull requests into your pipeline, you want a bot that can quickly and effectively. I say quickly because time does matter in the context of a pull request. You don't want a pull request that's sitting there spinning for an hour waiting for a security review to occur. So we built an evaluation suite. It's pretty expensive to run. We've got like repositories with multiple languages. We've got 80, 80 of them. Yeah,

80, 80 repos in the. In the background that we do this across

plant vulnerabilities in those repos, deliberately planted vulnerabilities, and then we test each model's ability to find those vulnerabilities. And so we do that in a really systematic way. So to answer your question, the best models at finding the vulnerabilities depends on your definition of best time, speed to run, cost and ability to find the vulnerabilities that we're talking about. And the noise that it generates along the way as well. So there's like four different parameters there. But if you wanted to optimize for cost and just overall finding quality, DeepSeek is hands down the winner right now. DeepSeek Flash V4.1 like the latest one that just came out.

Yeah, the 17th of October snapshot.

Astonishingly good. A tenth of the price of the nearest frontier model, which is GPT-4.1, which is about on par with DeepSeek Flash. We, our jaws are on the floor. So we run this benchmark every, what, every two weeks? Roughly, Simon. Every few weeks. Like when a new model, Gemini Argon for four came out, every model, we'd put that in there as well. And when the DeepSeek Flash model came out we didn't believe the results. We were like running the, running the benchmark multiple times. Simon was like, there's got to be something wrong. But it really is just very effective.

Yes. However, and here's that this is the challenge of being on the frontier. Come join us. Is it's really hard to get access to. To DeepSeek. Vercel have availability for it, but it's a bit rate limited

I think in our context. I think it's also really important to call out like we really care about giving developers timely feedback. And so like our P50 for giving someone feedback on their PR is two and a half minutes right now. And so the reason why I'm going there is that at our scale, we need a lot of token throughput. So I mean, it's a lot of requests per minute, a lot of tokens per minute. Anyone that wants to jump on can sign up to Vercel or sign up to Baseten Fireworks.

Yeah, that's my point. I think as a starting point, you can get access to these tools through OpenRouter. I think at our scale, what we're trying to optimize for is a really good developer experience and it sort of gets a little bit harder when you're trying to do it for one PR a second kind of thing.

Just check the quantization level though, as like we're finding we run our eval suite across different providers and some of them say they have like the latest Qwen or DeepSeek and it's heavily quantized and the results aren't as good.

how they've deployed it, how. Yeah, exactly.

I mean, I can maybe just walk through a couple of the questions that we asked. So we have a whole adversarial verification pipeline. It asks a ton of questions and has specialist tools. You know, there are sometimes cases where the LLM will be like, hey, you know, I think there's something wrong here, but that code's not reachable or that path. So like a basic reachability test will do wonders where you're just like, hey, are you sure someone can actually get here as a for instance, One of the other things that we do is just contextualize, like what is the part of the application that you found this in? You know, is this a CLI tool being used by a developer? And so therefore, you know, the the idea of like input that's controlled by like a malicious actor is like, well, what are you talking about? That's just going to be one of our staff members running this CLI tool, for instance, right?

You know, those, those kind of things can, can go a long way. But what you end up is kind of like a skill that asks like, you know, four or five kind of validating questions to go, well, are you sure about this? Are you sure about that? We do all of that automatically and we pre build some of that context so it's like readily available and we can deliver the feedback quickly. But that's ultimately the route you end up going down to kind of try and reduce the noise. Because the other thing that we found is that an LLM will always have something to say. So if you ask the same question five times in a row and you fix the output of each one of them, you'll never get to a point where it's actually going to say nothing.

We talked about it briefly earlier, but some models think well and generate less findings to start with. Unfortunately, although DeepSeek is the best at finding the vulnerabilities we plant, it's also the noisiest. So our issue processing pipeline has to work hardest with DeepSeek, the issue processing pipeline that Simon just mentioned, to invalidate and cull those findings. And then coming back to where we started, you know, if we talked about red teaming, there's another great tactic is the to take those findings and push it into your red teaming software. See if you can exploit. Here's a finding that our code scanning tool came up with. See if you can exploit it, see if you can go find this.

I will say just going back all the way to the front of the call, like, just fix the issues. Like to run a loop and be like, hey, can you just address these issues regardless of what they are? For most people today, it's not a, a budget breaker or bank breaker and it's going to result in you not being the next PHP to shell, basically because that's exactly what happened. Right? It's like someone did that prioritization and they went, ah, I can't exploit that issue. So it's going to go further down. So I think in today's, you know, world where you're not writing the handcrafting the code, you're not spending six hours fixing it. So just, just do another loop, fix the issues.

The way I think of a software factory is like the, you know, basically the stages, automated. Most examples I've looked at and companies we work with that have tried this end up being doing like a DAG or like a workflow, a predefined workflow where you pop in like some logs that were noisy or some error logs from your production system. Then something goes and writes a spec on how to fix it. Something generates the tests that need to pass. So there's like a TDD step and then there's the actual implementation step. And they tend to be kind of separate agents that are handing context off to each other. I think security can happen at that planning step. So when you're planning out what the implementation is going to look like. Right, that's a good point. Starting point where you can like ask questions around like okay, how does this fit into my threat model? For instance? Then at the implementation, one of the things that we do internally is we have very specific linting guardrails from a security point of so that as the agent is generating code, we are actually kind of like slapping the agent and being like ha, that tripped up that little security rule. Most agents are smart enough to go okay, either I did something wrong or I just need to suppress that rule. So linting rules can be incredibly helpful just in a lights out fashion to just like keep slapping the hand. And then at the very end when the entire PR is available, then yeah, as Patrick mentioned, you wanted to do like a review of that and then you know, in depth check. And so those are the three things that come to mind. I will say I haven't seen anyone do like a total lights out software factory. I think there's always humans in the loop. And so I feel like most implementations are lights on and people are getting involved either when the agent just has too much ambiguity and it needs to ask questions or follow up questions to, you know, flesh out the, the spec a little bit more. Implementation didn't go as planned and so on.

I'll add a few interesting examples from what we're seeing at some of our more advanced customers who, who are building software factories. There used to be a period where blocking a PR was a no, no. Engineering leaders would be like, you cannot block the PR these days. It's a great day because you're not, you're not blocking an engineer if it's a software factory, you're blocking an agent. So if your bot finds a critical vulnerability, like if we find a critical vulnerability in a pull request, we will block the pull request and an agent will go back and forward with our agent until it resolves that issue. No human's been affected by that process. So I think all of our customers are actually turning on PR gating because they want their agents to not jam past and blow past critical or high vulnerabilities. So it's an interesting trend Shift. And on the opposite end of the spectrum, the other trend with shift we're seeing is auto approval of PRs. Everyone's now aligned. We need to get to auto approval PRs. Very few customers have gotten there. There was an interesting article last week from Anthropic how they just, they've kind of gotten rid of it altogether because they're trusting their software factory so much, in fact, so much that they've even just taken git out of the loop. Interestingly, almost every CTO I speak to right now is engaged with their CEO on how to auto approve some percentage of their PRs. And it's always through the lens of efficacy, speed and getting to market faster. If you're really honest and you look at your pull requests, at least a third of them we would consider to be routine, routine, basic, very low risk pull requests, either documentation update or simple SEO page updates, things like that, they can be auto approved. The trap's going to be your compliance partner hasn't caught up with you. So if you need to do an annual compliance audit, they're going to insist that was a human in the loop on every single software production push from a software quality perspective. So then how do you rise to meet that need? My advice is to still review those auto approved pull requests, but do it in a batch. Because no one says there has to be a human in the loop in real time. It just means there needs to be a human who looks at it at some point. And so some customers are looking at all of the auto approved PRs on every two week basis, for instance, and just kind of casting their eye back over it and then guiding their agents from there.

And who says that a human did a better job anyway?

Yeah. Yeah.

Yeah,

Well, Simon, I mean, you're helping us get through our SOC 2, and ISO. And what I see is the things that we have to comply with are not the things that we're worried about. Right.

Yeah. I mean, I think it also comes down to, like, when you're going through it the first time, just making the most of it for yourself. Right. Like, I'll talk for a really boring example. You know, you do your, like, business, your tabletop, like, IR exercise. Just make it relevant, you know, don't do the, like, Gary lost his laptop. Make something like, oh, someone downloaded LiteLLM and their laptop actually got popped. We lost all of our, like, make it. Make it spicy, make it interesting. Because the reality is you can create a lot of value for yourself and go like, okay, actually there's ways to make it better. But I mean, I agree, like, the old world of, like, you know, I'm going through a SOC 2 and we do the training, you know, once a year and call that security. Yeah, it's pretty theatrical.

Yeah. I mean, take. Take the example from earlier of needing an annual pen test. I get it. I get the control. In our instance, like, we go way beyond what's required from a compliance perspective. You know, Simon's nudging agents to break out of the sandboxes we're putting them into. We're adventuring adversarially, trying to get our system to, you know, as we assume that all code coming into our system is malicious,

Do we think these frameworks are going to catch up and if so, how long?

I think they already are. I mean, I talk to our auditor on sort of A monthly basis. And I think the tides really turned on them and they're like yeah, we're a bit more flexible here or we're happy to work with you on an agentic access review for instance. Some of these things do make you more secure, like the idea of an access access review and culling people to access where it's not needed, that ultimately prevents things like someone's account getting hijacked, them having like permissions to do basically everything and it resulting in like a LiteLLM where you know, they're actually a maintainer of like a really popular open source library or something that millions of people deploy. So some of it can be helpful.

as a citizen coder or as a software developer? Because I think there's different answers between those two.

You gave the example of a founder who's not very technical. I think if you're a non technical founder pushing stuff onto the Internet, you should be very concerned. I think that there's a high level of trust in the tools that are putting code out onto the market right now. And I really would strongly advise ensuring you have a skilled engineer around you reviewing stuff that you're putting onto the Internet. So that's the one thing I would say for the non technical developer using security stuff.

The journey I think that most people go on is they start with a ChatGPT session and ChatGPT starts like, or Claude Code starts creating files like a software project and they don't know that they're creating a software project. And then to our earlier point, like most of these models, they're on distribution for recommending Vercel or Supabase. And then you're like you should go and create a Vercel account or create a Supabase account and you're like okay, I'll go and do that. And then 10 minutes later you've got your app up and running and it's extremely gratifying. But you also, you Know, if you don't really know what you're doing with Supabase, you can shoot yourself in the foot with the RLS policies, for instance, you know, and you're like, oh, my app works, but everyone can read everyone's data, for instance, you know, like, there's no, there's no concept of actual tenancy. So I think in the citizen developer context, I agree with Patrick,

same deal, great for prototypes, do not put it in production.

Yeah, well, it's interesting. Like, Simon and I often talk about whether we want to try and fill that gap and we, you know, here's the question that we ask ourselves. Is it fillable? Let's take an example of. I was talking to LawPath last night and they were like, people are, you know, citizen developing their own legal documents as well, and lawyers are aghast at that process. And the right question is like, is it better than a lawyer's guiding an LLM because they're an expert judge of the output? And is it okay that it found as LLM generating a contract? And it's the same question to ask with code. The risks are the same. And so I think it just, it really depends on your risk appetite there as a security professional looking at the kinds of things that come out of Lovable. Well, we did a trial with the. A customer who their process would add the product team developing an app with Lovable. It would then go into. It was through a filtering mechanism with DamSecure and their engineering team. And the issues that we saw come through would make your toes curl. There was routes that didn't have authentication on them at all, like just completely open, authenticated, unauthenticated routes. And these are just things that if you would have prompted correctly, those issues wouldn't have occurred. So it's actually in the hands of the operator not knowing how to prompt it correctly and ask it correctly for the right outcomes. Which is why I just go back to like, if you're in that position, just get an engineer around you to help you review the outcomes of it before you put it in production.

I think you get pretty decent mileage out of that. We found, even with our internal testing. Like some of the stuff that we do is like, well, okay, you found that thing, but did you find the other thing? So LLMs are really good at like, hey, have I do I think I've satisfied the task. And so, you know, you end up finding a couple of things and you as a user you're like, oh yeah, okay, I've found two things good, but there might be another three or four or five, you know, and that's why we kind of take our approaches to, to try and, you know, exhaustively find all the things that could be wrong so that it's not just like the, the three things that get fixed, it's the other, you know, seven that might matter in the, in the long run. So I, I think that's definitely a good starting point. I would say it's like, hey, can you have a think? But also I would say you're getting an incomplete answer just by design on how the LLM works and thinking and reasoning levels and effort levels. At some stage it's just going to stop and be like, okay, I think I'm done here.

They do generate secure code if prompted correctly. That's correct. Just never Forget that these LLMs are trained on the same material that's on the Internet. And that material is not necessarily secure. There is. I could point you at numerous articles, even recent ones, that test the code that comes out of even modern LLMs. They're not more secure than the code we generated manually four years ago. They're just not. There's no magic machine here. They're trained on the same stuff that we generated over the last four years. And so will they get better? Maybe at minimum have an adversarial agent like ours in place trying to review stuff and there's some open source stuff that you could use. If, if you don't want to purchase a commercial vendor, at minimum do that. But again, you're like at a point where, like, can a citizen coder even configure that and get that up and going? Maybe this is where you need a software engineer involved, I think, to get that kind of software factory building going so that talking to a customer yesterday and. And like they're doing some crazy cool things. I've got very large customer. You would all know the brand name and their sales team put pull requests into their production repository. They're encouraging everyone across the company to do pull requests against the repository. It's a great idea and now I've got a whole process for how to manage that. The product team then take that, they take it as an idea. It's just like product feedback. It's like a feedback mechanism. It doesn't mean it's going to hit production. It's more like, I'm going to show you how I would build this for our customers. And a lot of that code actually ends up getting polished and put into production as well. So I do think there's. We're going to see some really interesting different models occur in smaller and bigger companies with this citizen coder model. And I think the quality of the software factory you build will be the determinant of how quickly you can take that input and push it through in.

even more so today, the adversarial capabilities unfortunately have accelerated. I'm not trying to instill fear. It is accelerating rapidly. We're seeing it happening in market and the defenses need to come up accordingly. And so if you rewind the clock on your thinking three or four years ago, on your kind of security preparedness, amplify that several levels and you're probably getting towards the right level of where you need to be today.

Yeah, please just go to DamSecure AI. You can look at our blog and find the benchmark that we talked about earlier and we're very active on LinkedIn. Please follow Simon and I on LinkedIn. We're very active on current topics, current issues, whether it relates to open weights or security, active security issues as well. Maybe see some images of my cat along the way.