Where humans still matter when agents write the code

We're a lot further down than like LinkedIn would suggest, and a lot further behind than where X would suggest. Code is now free and we don't have to do code reviews anymore. So why aren't the engineers taking all that free time and connecting directly with customers and driving the product? AI creates very few entirely new problems, but it has made an order of magnitude worse. AI is horrific at making product decisions like the product decisions are so incredibly terrible. When your application breaks, who are you going to turn to and what do you expect them to do? Something has to change something. I've been wondering for quite a while now with Agentic AI coding tools is Is code review the traditional PR review dead? I have got two amazing guests to discuss that with me right after the break. One of the amazing things about AI is it allows you to turn your idea into a real business faster than ever before. But you don't want to let yourself get dragged down by PA bureaucracy and government stuff like that. So that's why you need today's sponsor Spark Launch for just $499. Their Delaware C Corp formation package includes Delaware filing and your first year of registered agent service. Plus keep your company documents and founder equity organized as you grow so you can focus on building the business you build the product. Spark Launch helps you start the company. Visit SparkLaunch IO to to get another 20% off their already great deals and get started. This episode of Applied Wizardry is brought to you by Dam Secure. DamSecure is the code security platform for AI. First engineering teams get top tier security PR reviews at a fraction of the cost of other models and get through your PRs 32% faster. Don't just take my word for it. Head over to DamSecure AI Wizardry. That's D A M Secure AI Wizardry for your first month. Completely free to discuss the topic of whether code review is dead. I have got two great guests. First of all, James McGill, an old friend and colleague of mine from Google is currently the founding engineer at actuals. And Hazik Norden used to be at Amazon and is now Founder and CTO at Enhanced Labs. These guys have been building software software at scale for a long time. Now they're doing early stage incredible stuff using AI. They are fantastic experts to discuss this topic. So I cannot wait to get into it with them. Let's jump straight in. Is Code Review dead? I guess before we start right, what do we mean by code review and what do we mean by dead? I feel like the whole dead thing is really messy. Apparently software died 17 times in the last 12 months without pulling out the Oxford English Dictionary for definitions. I sort of agree, you know, code review is dead. Long live code review. When I first started my career, code review happened once every two months. We got in a room, there was a projector, someone would project all of the code that had been written over the last two months. We were in there for like nine hours just smashing coffee until we were vibrating. And we'd be like, okay, like, should we change this thing? Should we change that thing? It is definitely not line by line code review, right? But it was the way we had at the time with the tools. We had to make sure that everybody broadly understood how the system had been changing. And then we shifted to inline or Google heads with no Mondrian style code review that has been popularized so much by GitHub, where we're sitting there staring at every single line and agonizing over every single variable name. And we got into a little bit of a loop of like we do line by line code review because we do line by line code review. And that's what engineers do. I agree with you, James, that we just sort of fell into it. There's a lot of cargo cult in software engineering where we do things because we do them. I've always felt similarly about testing. I know I'm not the only one where it's like, well, you need your test coverage above 90% or whatever, and it's like, why? What is the purpose of testing? Right? And so when you ask the whys, you often end up at a conclusion that is not quite the same as the standard practice. And similarly, why do we do code review? Right? And yeah, I guess when I put this question here, it's like, as popularized by GitHub, the pull request, I guess that's sort of what I had in mind. The idea of, okay, here's a bunch of diffs and you look at them all and you make comments and eventually you approve the diff. And then it gets patched into or merged into main. It was a thing you got out of the box with GitHub, which was the thing everyone used, right? It's just a behavior that was easy to scale, right? Like, you could very easily just be like, hey, everyone, just follow this process. The tool supports this process and this is how we want to determine code quality, which I think is reasonable in very large companies. But yeah, you're right, there are a million ways to achieve the goals. The code quality piece is interesting too, because that is something that even pre AI changed pretty substantially, right? So I remember my first reviews that I received as a junior engineer. There was just a huge amount of emphasis on style and formatting and naming. You need would remember the 50 page style guides we had on how to write C or how to write JavaScript. And that feels like it's just a solved problem, right? Between linters and static formatters. It's not something that I spend any time thinking about. It's not something that I would expect to be commenting on in a review. But we didn't step back and update that as like, okay, if that's solved, what then is the next thing we're trying to accomplish with review? And is line by line review the best way to accomplish it? I think it's an interesting question to ask. What would you be hoping to get out of code review? What are the goals you would have in having it as a process? When I started in code reviews, it definitely wasn't the projector, it was more the GitHub style of, hey, we'll have the diff people go through it. You have two people go through it. And the expectation very much was you go through every line. Like there are different kinds of value you can get from that and there are different ways to waste a lot of time. But I think one of the most valuable things that fall out of that is just it helped people learn how to code in the way that the team wanted you to code. And that's very different now because, well, we're not writing the code right. Like a lot of the stuff that you would normally want to catch doing a code review, you can now turn into linters. And those linters can be a lot more powerful than they used to be because AI can write those linters. Right? Like, for example, it used to be very unreasonable to say, okay, we want some code review thing that, that would check our architecture. That's annoying and that's very brittle and fragile to write, but now who cares? So a lot of those kind of reasons are gone. The reason I still kind of do code reviews not of the line by line is really complexity. I think that's a big one, is I think agents still kind of struggle with managing complexity and identifying when they're doing something unusual, when something easier would be more appropriate. And I haven't found a good way of catching that yet. Yeah, that's one of the points I wanted to make. And I think it's dead on. We're pretty much past the point where if you ask an agent to build something, it will Build something that does the thing you asked. I at least believe engineers still add a lot of value in steering during that process to make sure that it's doing it in the way that is simplest. Doing it in the way that if you get successful tomorrow and you had 500 users, you're not going to just completely blow up the system. And agents, at least today could change in six months. They're not great at self evaluating on would there have been a simpler way? And so part of the thing I'm reviewing for at the moment is just did the engineer who guided the agents write this code, steer it successfully? And that's not a line by line thing. It might be literally a, hey, I'm rejecting this whole change because I don't think that you put enough thought and guidance and steering into achieving the best outcome with the agent. But if you get rid of that, my concern is that it's so easy to fall into this just slop cycle of like, cool, I'm jamming, I'm moving fast, I don't need to look at the output. And then suddenly you step back and you have a million lines of code where 10,000 lines of code would have sufficed. And that matters because that becomes the context that feeds the next prompt and the next prompt and the next prompt. And if you let that complexity accumulate unnecessarily, subsequent agents are going to look at it and go like, cool, that's how this code base should look. I'm going to do things the same way. I do worry that at some point it starts to collapse. Yeah, yeah. Complexity death spirals are like, the only reason I really look at code nowadays is like, hey, will this cause a complexity death spiral where this code base will turn into mush? I mean, it feels so different to normal code reviews that it almost feels a bit dishonest to say that I'm doing code reviews. Like, I'm mostly just panning through the diff and just being like, hey, I just want to kind of get a rough shape of what's going on here. It does not feel like me going through in the way I used to go through and be like, oh, this variable name that behavior now feels like an insane behavior. There's sort of that that you want to get out of code review. So like, how do we want to deal with complexity? There are some things that I'm still, I feel a little bit weird about just letting the agent just do. But again, like, you know, if I go through like with a very fine tooth comb, like some Bit where I'm just like, okay, cool, like this is a security thing I'll go through. I'm not actually finding that I'm catching that much stuff. The default behavior isn't very wrong and when I do, it'll be like, oh, this could happen later. I feel myself becoming the engineer that just says stuff. There was always the code review person inside the code review that was like, I feel like I need to say something, so I'm going to say something. I'm feeling that behavior a lot more when I start looking for these lower level things. It's interesting. I agree that very rarely do I find something that's just objectively wrong. I do find places where basically the prompt was underspecified. Right. Or the properties that we needed out of the system weren't described correctly. I think that begs the question, which is, do you just replace code review with spec review or it's something else review? Yeah, or something else review. But I personally find that my ability to tell that it is underspecified just by looking at that prompt or that spec isn't great. Right. It's sort of in the process of seeing how did that spec intersect with the existing system and how did it get translated into a set of changes that I'm like, oh, shoot, yes, we forgot that in this case. If we don't limit the number of database connections we have coming in here, then eventually we're going to have memory exhaustion. Maybe I'm just not smart enough, but I lack the ability to look at a prompt and reliably identify those missing components. I think I'm actually a little bit further along than you guys in terms of. I would say I don't do code review. And I think my response to some of the things that you've been saying is that while I think a lot of the things you mentioned is true in terms of complexity spirals, in terms of specification, there's another factor at play that I think, in my view you're both underweighting and I think it depends what stage you're at. But all three of us are at early stage startups. And so I think at an early stage, something that is also different than before AI is what I would call the malleability of a code base. It used to be that rewriting and refactoring was expensive and now it is cheap and getting cheaper. It's in some sense safer or from a business point of view to take what used to be prototype code into production. Right. And if it is useful, then you can just rewrite it, right? Like if you're like, oh, okay, this thing's inefficient, it's unscalable, just rewrite it. I think you're right, but with one really important caveat, right? The code base is more malleable than ever. Data isn't. And so one of the questions I'm asking when I'm looking at how much review do I need is this stateful. If it's a react widget and I push it out and people hate it, or it's got a bug in it and they tell me about it with an agent, I can probably replace that whole thing and ship a new version in a couple of minutes. If I ship something that corrupts all the data on my database, I'm going to have a pretty bad time. Or if it injects data that is really hard to untangle and later breaks some of the assumptions I wanted to make of the schema of the data, that just becomes a more expensive operation to unwind. And that's a little bit from recent lived experience. So I'm starting to apply a little bit of a sliding scale where fun and code features things that are stateless. I'm much more happy to put out there with very, very cursory or no review. Anything that touches the database I think still needs a little bit of a stronger lens. I agree on the data thing. I do think data is a lot more malleable than it used to be. But you're right, it's not free. For sure, it's more, but it's not free. Yeah, it's definitely not free. But I agree with you, Yan of I'll be an annoying software engineer in this very bounded, in this very bounded way, which is like, hey, code whose behavior that you can bound is essentially free. Who cares, right? Like if you've locked it down with an end to end test and you have a very, very healthy set of end to end tests, you can just go, oh, whoops, write all of it. Again. Where that doesn't work though, is when you can't bound the behavior. And I think that's why complexity is so scary. So for example, like, hey, we want to do some migration and the AI introduces some crazy ledger thing. Oh, we'll track it with a ledger. Like that feels fun. And then like, oh, the ledger should trig behavior. And if you catch that too late, the complexity is now something that is just very hard to wrap your head around. And then simplifying it becomes really hard because to simplify A thing you need to kind of understand what it is to begin with. And that behavior is not free. I think a lot of things are cheaper, but a lot of it is just a function of what behavior is actually bounded. It's very easy to hit a point where you just cannot reasonably bound the behavior of a thing or bounding it is just hard because the AI is doing some novel strange thing that you haven't really thought about how to pound before. I suspect agents can deal with a lot of complexity and unwind it for you, but there's some point where probably they're not going to be able to do that reliably or safely. And if you reach a point where you no longer have confidence that you can iterate with an agent without carefully reviewing everything it does, because you've just stacked so many ledges and data models and state machines on top of each other, that has a very sinister cost, which is you're just going to slow right down. And so I think prevention is a little bit better than a cure here. And it doesn't have to be a heavy review, but just keeping an eye on things to make sure that you're remaining within a complexity bounds that you and the agents can navigate feels valuable. We were talking about like what is the point of code review? And my point of view, which I've held for a long time about your GitHub style PR review is it is a point of control. It's the only point between a software engineer doing whatever the fuck they want and that stuff being merged into production ultimately. Right? It can be used to make sure that the code is style compliant. It can be used to make sure that there aren't any architectural flaws. It can also just be used to make sure that we don't have new features going into production that some engineers fever dream that nobody else actually signed off on. Right? So forgive the wild analogy. I'd be interested to see if it resonates with you, is that I think often of democracy, right? Which is like you vote in a government potentially, and then the government does what it likes for three, four years and then you have another election. You're not really getting direct control over what's happening in your country. What democracy gives you is the ability to throw out a bad government every few years. And what that means is that the government knows it is being watched and that it is accountable. And I feel like code reviews often had that. Right. First of all, it created a self corrective mechanism in engineers where they're like, my code is going to be looked at, so I better make sure it's okay. And it also made sure that any of the worst excesses were stopped before they went too far. I think that's an incredibly important point on both fronts. So to my earlier point, if no one's watching, I think the temptation to just layer slop on slope is honestly irresistible. I don't know if I have the willpower to avoid it. I don't think anyone does. And then secondly, before my hair fell out and I was a young, enthusiastic engineer, I thought I had the best product ideas and that all that needed to happen was that they would land and there'd be a round of applause. And to torture your democracy analogy, if we just go to full holocracy and everybody can contribute straight to Maine and the product just evolves in every single direction that anyone can ever think of, it's all going to fall apart. And code review is one mechanism. I think you can probably design other mechanisms to help ensure coherence and structure in what's landing in the product. But it's a pretty effective mechanism that people understand and that has been doing good work so far. And in particular, it is the thing again, I guess different companies have different ways of operating, but if you do a sort of trunk based development where pull requests then merge to main and then get deployed to production, it is the only thing standing between an engineer and your customers. Right. It controls the actual artifact that your customers see. I don't think it's the right gate though. Right. Like, because if what we're saying is that the goal is to make sure that the product is sensible, the place to look for that is the product. And the people that you'd want to talk to about that are the people that talk to the customers all day, every day, which can't be the engineer because they're like, even if they do talk to customers a lot, they're also doing engineering. Let's push in on that. Right? You say the engineers are too busy engineering to chat to customers, but code is now free and we don't have to do code reviews anymore. So why aren't the engineers taking all that free time and connecting directly with customers and driving the product? Being able to reason about products is hard. It's its own skill set, it's its own sort of training thing, which means that it will take time for most people to upskill. I've been really trying to learn about design and UI ux and one of the recurring problems that we have is that I'll think that a thing is fine. I'll see it and be like, oh, there's a modal, it's so clear. And then our designer will see it and be like, hey, dude, it's clear because you know what the software is. But that's not clear to literally anybody else on this earth. Right. And if someone submitted that pr, I would probably think it was fine. Which is kind of why I think product review is going to be something that becomes very important soon. To just say, hey, can we widen the tent? How do we make sure that it's practical for people who are actually across what the product should be? I think that's been very interesting. We tried a bunch of just random ways of doing this. Maybe we'll do screenshots and maybe we'll do this and maybe we'll do that. But right now there isn't really a good answer for that. But I think that's when I say that when I was taking the conversation a step back, that's exactly my point, which is that code review's value was as this check on what could be done. So it gave a coordination and an accountability mechanism. So if we were thinking about designing a mechanism that serves that function, but serves that function for the world we live in today, what does that mechanism look like? And I think that's the interesting question when we're talking about spec review and so on. Yeah. And this is where the conversation goes to. Okay, if instead of code review we said change review, does everyone immediately get on the same page? Which is, do we all agree that there is any. Anytime your product is evolving or your architecture is evolving, there is some value in having a second set of eyes on that. Particularly to Hazik's point, if you acknowledge that different people in your organization are going to have different capabilities, different areas, they're stronger than others. This is a place where you need. As you mentioned, we're all at relatively small early stage startups, but I think it's a very different world in a one person, two person startup versus once you start to get to 20, 25 people and you're bringing in people with particular specialization in design, particular specialization in user interaction, particular understanding of data storage and architectures. How are you making sure that you're getting the strongest possible inputs on any change to the system? I very much like where you're headed, James. For us, it's a little bit different because our founding team has the designer that you would normally only be able to get later on. I think there's a couple of Challenges there, right? I think the PR is probably still the right time for sure, because that's when the thing will actually change. A code diff is not the right answer because it's impenetrable. And the temptation, I think, if you just got non technical people, is that they would also just go into Claude and be like, hey, does this look good? It still does require a second pair of eyes. Because AI is horrific at making product decisions. The product decisions are so incredibly terrible. And it's still very, very bad at being like, hey, how much complexity can a human actually handle? Is this annoying, what you've just done? Would that be annoying if a thing just popped up when you were trying to do something else? I feel like it'd be annoying. It's not very good at intuiting that behavior. And a human still just needs to look at that. Because we used to be in this world of qa, right, where someone would do qa. And I think the narrative was very much that QA was just like, oh, you're just clicking stuff. It's so easy. But it's very labor intensive. It's very hard to get your app in the right state to go do the things. And right now we do a lot of qa and it takes a huge amount of time. I think it's not only labor intensive. I think it requires a particular set of skills and an ability to, you know, construct interesting edge cases and to think about how someone who is different to you might interact with or what they might expect from an app. And I actually believe that that is becoming one of the more valuable skill sets for every single engineer to try and build. And it's not one that everyone naturally has or a muscle that they've had to flex before. I think the interesting thing is, in this age of code abundance, the real challenge and the real risk is not knowing what the fuck your product even does or what it's supposed to do. Which brings us back to the spec, doesn't it? It's like, okay, it is shockingly easy to end up in a world where you have a product that does all sorts of things that no human ever actually consciously thought about, and that no one individual has actually said it should work this way or it should work that way. It just does the things that it does. And while that's not totally a new problem, it is massively, and I would argue AI creates very few entirely new problems, but it has made it an order of magnitude worse. And the question is, okay, so what is the control point that allows us to actually know that our product is doing the things that we want it to be doing and that we actually know what we want the product to be doing. So I think that's a good segue into another thing that I wanted to get both of your thoughts on. Right, which is we were talking about, why do you do code review? One of the reasons we haven't touched on is mental model. And I think one of the valuable aspects of code review is that it helped other engineers in the organization build and maintain a mental model of the architecture and how the product and how the software is changing. Where is that important? Well, when your application breaks, who are you going to turn to and what do you expect them to do? And if your answer is, we will universally start by prompting an LLM and saying our app is broken, please fix, then I think you're fine. But at almost every company I'm talking to, the answer is still, yeah, we'll tap an engineer and we expect them to be able to very quickly with an LLM, but steering that LLM pretty heavily diagnose and fix the issue. And I don't know how you can do that if the mechanics of your software remains a black box. It's hard to say, right? But I do think we've sort of been circling around this idea of a spec and there's a product spec and a technical spec. And another analogy that I've seen is the analogy between LLM coding agents and compilers, right? Which is like nobody is looking at the compiled machine code and nobody is saying this needs to be understandable or it needs to manage the complexity. Like modern optimized assembler is beyond the capability of any normal human to understand. And we don't expect humans to understand it. You know, the argument a year ago especially was like, well, you know, LLM makes lots of mistakes. It's stochastic, blah, blah, blah, blah, blah. That is true and it's still true, but it's a lot less true than it used to be. And I would posit that for most normie software, a well specified document will then convert into software that does the things that are specified in that document. And so it does raise the question at least of is the right control plane then around spec and making sure that everything that needs to be specified is properly specified. And how do we do that in a scalable way? Certainly that's been our experience, right? That's the whole thing with the ticket reviews is really what that is. Is spec review to be like, hey, like anyone can Cut a ticket, like that's totally fine and the agent loop can just pick it up and just do them most of the time. But like, what is not cool is just being like, oh, we need some kind of alert. Like, what do you mean? Like some kind of alert. There's no human that's going to sit there and reason this through, so you better be bang on, right? I think spec reviews are going to become more of a thing. I totally agree with you, Yanif. We are entering this world where LLMs are just like spec compilers and it becomes very important that your spec is good because the thing will just compile what you put in. But I don't think that materially changes where the final gate happens. I think you end up spending a lot more time thinking about upfront, what do we want? But I think the gate still has to be the merge. There's one extreme of this where you can look at code as just a very detailed, very precise spec written in a very limited language. That's torturing the analogy a little bit, but the limitations in what syntax you have to play with makes it very easy for a human to read that and reason about exactly how the software is going to work. And at the other extreme, English or natural language is very easy to write and very imprecise. It is easy to leave things out, it is easy to leave things subject to interpretation. And I expect that that is something that is going to improve. We'll see tooling, we'll see agents being able to grade specs at a higher degree of reliability. And I know there's a bunch of efforts to come up with like micro spec languages, but I just don't think anything has nailed that yet to a point where you don't have to look at the output of the agent having interpreted that spec. Or is it like to sort of again, reason by analogy a little bit. And this is where I get to be old, right? Like I remember programming in C, right? And I'd look over at those Java weenies with their garbage collection and going, these guys do not understand memory management, right? They're getting all these memory leaks, the code is inefficient, they don't know what's happening under the hood, which was all true, actually. And the Java garbage collector was really bad back then. And I felt the sense of mastery and control with my memory management. And the truth is that the higher level language gave people more leverage at the cost of more abstraction and less control over the details of what is actually happening on the device. When the code is executed and this is the whole history of language abstraction. And so I think the devil's advocate here is this is just the latest step of language abstraction. You know, like Typescript doesn't really represent what's happening on the device very well. So now we have English language or natural language, as you say, and that's just your next level of abstraction that gives you more leverage at the expense of less precise control over what's happening on the device. Why is it. I think it's a degree of. It's a degree of maturity as well. Right. So to dive into your Java example, early Java with the janky management, memory management did give the engineers the ability to move faster, but guess what? That complexity didn't go away. And having spoken to the SREs who ran those early JRE services, the amount of blood, sweat and tears they went into tuning the parameters so that it didn't crash all the time or oom, like there was very significant work there that just got shifted onto a different set of humans and then as the technology matured, that became a much more solved problem. And so my position is not that we are never going to reach a point where through a combination of coming up with whatever the natural language equivalent of design patterns is through tooling through experience, that natural languages is not going to be sufficient as the primary input. I just don't think we're there today. And if you have to make a decision as an engineering leader like today, am I going to throw code review entirely out the window? My position remains it's premature. We're not there yet. You're just going to shift that complexity to someone else in the organization and you're maybe going to shift it to that person. Having to deal with that complexity at 3am with a black box they don't really understand while your customers are yelling at you. Sucks to be them. I think the Java analogy is a really good example though, because in the early days people did kind of inspect the bytecode to be like, ah, this thing is horrific. Like today, if you do that, you'd just be like, hey man, so we pay you a lot of money to get quite a lot of work done. It just doesn't seem like the way I think you're right. It's just like, where are we in that timeline? I feel like we're a lot further down than LinkedIn would suggest and a lot further behind than where X would suggest. Right now. I think it's still pretty okay to want to peel the thing back and it's not absurd to want to drill a bit deeper and there's still questions about how far do you want to drill, but I think in a year and two years it's going to start to feel very funny. Now I wanted to take a moment to talk about cybersecurity and one of today's sponsors, Dam Secure. Really caring about building secure software is one of the key differences between agentic AI software development and mere vibe. Coding and DamSecure specialize on providing security PR reviews for you and your AI first engineering team. They do their own research on the latest models and they learn your code base. The result is you get PR reviews that are security focused and faster and cheaper than the general models with far less noise and unnecessary reports. So you can focus on fixing the security issues that really matter and get back to building for your customers. DamSecure is offering Applied Wizardry listeners one month for free. If you go to DamSecure AI Wizardry, that's DamSecure AI Wizardry to claim your first month completely free. Building with AI allows you to get your business started so quickly, but to do that right you need a partner that helps you do the paperwork and the bureaucracy at the same speed. With today's sponsor, Spark Launch, you can create your startup in seconds using their platform, or you can even plug in your favorite AI tool and it can talk to it directly. For just 15 $499 you get a Delaware C Corp formed now Delaware C Corp are a really great choice for startups all over the world. A lot of investors will only invest in Delaware C Corps as well as the registration. That $499 package helps you keep company documents and founder equity organized as you grow. Now for Applied Wizardry listeners, you get another 20% off the already fantastic price. So go to SparkLaunch IO. That's SparkLaunch IO Wizardry to claim another 20% off. It feels like verifying behavior is like the right thing and where we'll eventually end up. And there are lots of different ways to verify behavior, but then there's also a lot of challenges that come with that. Like for example, it's never been practical for us to write enough end to end tests that we will notice how long they take. It is now extremely practical to do that and end to end tests are like the best way to verify behavior. There's a whole ocean of challenges that come with that. I'm hitting them right now which is like hey, if we want lots of agents to run at the same time, then the verifiers can't Be too expensive. End to end tests are the most expensive verifier, but they're also the most powerful ones. So how do we want to cut this? And is there some minimal set that will prove what we needed to prove? I've never had to think about that. Right. Most of the time I was happy to get any end to end test that was not very brittle. I think verification is an interesting point. That's something we've been spending a lot of time chatting about at actuals. If you have very strong verification, then absolutely you can just not stare at like you do not need to care about the implementation details. So this is where we've been spending a lot more time maybe than you would historically, thinking about things like fuzzing, thinking about things like running broad memory profiling tests. If the thing you care about is the product does what you wanted it to do and it's not going to crash and it's not going to corrupt any data and you have comprehensive fuzzing and you've watched a video of the agent driving the product it built and that video looks good to you, then I think maybe you're in a pretty good position and that's to think about these. What are these skill sets that someone who used to wake up every day and think about coding should be thinking about building? You know, we touched on design sense and product sense. I think familiarity with thinking like a member of the QA team and thinking about what are the engineering approaches to verification. How can I apply fuzzing? How can I apply property based testing? Like these different types of tests that allow you to comprehensively validate properties of the system, the better job you can do there. Yeah. Then I think the less you need to peel back the code. I'm glad we're talking about testing because I think philosophically it's always been this interesting thing where I remember many debates over the years with testing purists who are like, it should all be about unit tests and test coverage. It should all be end to end, or it should be API tests and integration tests and this and that. And the truth about testing is there was always this important but unsatisfying craft of assembling the right patchwork of different types of tests to meet this quite pragmatic outcome, which is software that works pretty well. And that still remains the case. Right. And again, with AI, the thing that I see about it is it doesn't really introduce anything new. Right. And so what we've got here is a situation where certain things have become much Much cheaper to do. And what do we do with that power? Right? So, yeah, the first thing is AI can write a fuck ton of unit tests and everyone's got their AI writing that. But then again, a lot of those unit tests are of questionable value. It also means that we can effectively say, hey, we have a QA team of persistent, cheap, relentless, and also technically very capable QA people who can just, you know, test the shit out of our software if we set it up properly. And then the question is, okay, what is the right patchwork of quality gates that we put onto our software that is fit for purpose, given the kind of cost landscape that now exists that is so different to what it was a year ago just to dig in on something. I have no idea where I'm actually going with this, but there was a secondary point in my mind to running tests, which is that that often in the process of writing the tests towards consuming your own software, you realize that your software sucked, right? So it wasn't just having the test that was the point in my mind. It was writing the tests that helped improve the overall quality of the work. And I think there's some sort of analogy there that pushing on your own output and having friction in the process of creating something often elevates the quality of what you're building. And how do we maintain that level of introspection and thought and analysis of our own work in a world where we're just typing prompts and hitting enter? I don't know where to go with that, but it's an interesting secondary benefit that I always thought of the process sitting down and writing tests. If we don't care about the actual code, then writing the test to consume that code is no longer valuable. But there's some analogy there to. To making sure the thing we're building is actually good. And if you just delegate all of that to agents, I don't know where that introspection comes from. Well, I think it lives in the spec, right? As you're kind of writing the spec, you're thinking about what would it mean for me to be happy that the spec has been implemented. What is worth having a very deep familiarity with, I think for sure is changing, I don't know in software, and I think it's because of the way we tend to talk about software. There are some things that are just dealt with as a inherent good, right? Like knowing your software deeply is often thought of as an inherent good, but like, knowing yourself more deeply is only useful insofar as it, like, unlocks Some sort of thing for the business. Right. Like, and I can absolutely see a world where you can go like, hey, like all my tests have been defined in the spec. I do not look at the code like at all. And that world actually being like a very good one where great software like falls out at the other end. So I don't think that feeling very familiar with your software is a thing that we necessarily even really need to maintain. The thing then becomes interesting of like, hey, what happens if it breaks at 3:00am? I mean, to be honest, we're in startups and I don't have an on call rotation at 3am, so I haven't quite experienced this. On the other hand, and James, I suspect you're going to disagree with me and maybe this is just because I'm dumb, but I feel that my agents are so much better at diagnosing a production outage than I ever was. They can go through a fuck ton of logs and they know about all the little corner cases and weird production things that can happen. And so for that specific issue of like outages I would be like, you know, my agent, my agent's going to fix that at 3am but lately outages also involve you having to do some kind of dangerous operation all the time. I think that's the thing is having the contents like if the agent says all good mate, all I need to do is just like pull down your whole kubernetes cluster. I'm going to delete these persistent volume claims and create some new ones. At the end of the day I feel like I need to have enough context to know whether that's a sensible mutating operation to do. Yeah, I mean, I guess you're talking about accountability and so on as well. Yeah, I think that's the important piece is organizationally where does the accountability sit? And you know, is the answer with the agents? Because I think people say that but, but I rarely see it apply in practice. One of the common things I see in sort of AI use policies in the way software engineering organizations are talking about AI is the accountability is still sitting with you. Right? The agent did it is not a excuse for something that impacts customers happening. And I agree with that. But then the question becomes, well, does that mean that the accountability is how have you prompted and set up your agent so that you can feel accountable for it? Listening to both of you guys talk like this thought just at the bottom of my mind is this is so much about leverage. Right? Like all of these questions we're asking is okay, we've been given this tool for enormous leverage, but if we use it unwisely, we get the leverage, but we lose a level of safety and control. And to your point, Hazik, which I think is really valid, I guess the sense that human knowledge and human insight is still at the core of building something successful. And when we're sitting on all this leverage, the question is, where do we focus our narrow but deep attention as humans to make the best use of that leverage so that we don't just, you know, we're not just riding a slop unicorn and we're actually building something valuable. The leverage point is important and something that shouldn't get lost in this conversation because something I'm hearing from a lot of companies I work with is that, hey, we're using all this AI and we're not moving any faster. And if you are implementing all your changes with AI, but you're still doing line by line code review as humans, then absolutely you've introduced this bottleneck that means you don't benefit from any of the leverage of AI. So something has to change. I think it's just a degree of extremes. Right. And do you completely get, get rid of looking at the code altogether to do what Hazik's doing and scan for the architecture, learning how? Guided code reviews, I think are a really interesting thread to pull on where you are reviewing an artifact that's generated from the code. Right. So that catches the issue of the prompt was misinterpreted or under specified. But you're reviewing a very high level summary of the changes that happened in the code. You're certainly not looking at line by line implementation details. And I think that is an interesting thing to explore. Or you know, as we said, maybe, maybe you just click around the product through the eyes of a QA engineer and convince yourself that it's going to operate the way you need to operate in front of your customers and that for a lot of products and a lot of changes might be sufficient. But I would continue to take a strong viewpoint that you need to review something every time your software is evolving. Yeah, I think the question of leverage is interesting because it brings up this kind of opportunity cost thing. These things are oftentimes posed as though they're free. If you're doing your line by line review, does that mean that you're spending less time thinking about the architecture? Does that mean that you're spending less time clicking through the product this AI has just vomited out? And does that mean you're spending less time Talking to customers, what is the leverage set to really unlock that? Like in a company you need to have the accountability conversation. And it's a very, very, very difficult conversation. I think for a lot of businesses to have the question of like, who do we blame and how comfortable do they feel Being blamed is a lot less interesting right now, at least to like, where can we get a scalable fix? Right? Like you can feel bad and we can tell you off and you can be responsible, but like, how does that actually help the business? I think the one interesting conversation is I guess, kind of like what you were sort of proposing, Yanav, which is just say, hey, there's a system, it does stuff, it did the wrong thing. What about the system did the wrong thing? So as three engineers, I feel we've been hedging a lot in this conversation, but I'd actually like to put myself out there and say I think that code review as defined as traditional GitHub style PR review, 99% of the time it is not worth the opportunity cost of doing that. I think opportunity cost is the right framework and therefore, yes, I believe that code review is dead in the sense that it is traditionally thought to be interpreted and therefore we need something different and better to replace it. And right now, because things are evolving so much, but also because of the engineers catch cry, which is, it depends, we're going to have multiple different replacements for code review. But I would put it to you that unless you are working in the very specific parts of your code base or on very specific types of product, the idea that you are looking thoroughly at a diff every time there is a change to your code base, I think that is a poor use of your time. I couldn't agree more. The traditional line by line diff is dead. I think we haven't worked out what replaces it yet and it's in place. I'm very excited for a bunch of people to be innovating, but if as a software company you're not actively talking about that and trying different things, then I think you are behind the game. Yeah, I think we agree and I think for most businesses to take that step, they have to be thinking in terms of the business is a system and we fix the system. And if you're trapped in the world of the business is a way to distribute blame, then I think that approach becomes extremely hard because then everyone wants to hedge to the maximum degree and the personal opportunity cost is very different than the opportunity cost of the business. This. There's also a first mover Risk here. That is interesting. Right. So something we haven't touched on is compliance. And a very standard part of your SoC2 or ISO checklist is, hey, we solemnly swear that every change that goes into production has been thoroughly reviewed. And I think that review has been traditionally understood to mean line by line review by another engineer. Right. And so I'm excited to see that pushed forward. You also have. It is as a business, you have to choose. Am I going to spend my innovation tokens arguing with the auditor that my mechanism of review is sufficient for this? Maybe, maybe not. It's something I'm excited to explore, but there is some inertia that I think will hold us back. And compliance is one of those ingredients. One of my management or leadership principles that I've espoused in the past, and people who've listened to me on lots of my podcasts are probably sick of hearing this one. But one of my stories is about forcing functions. Right? The idea of if you want change to happen in an organization, you have to take away the safety net and see what emerges in its place. And when I first joined Airtasker, we had a manual QA function that was responsible for signing off on every release. And the first thing I did, one of the first things I did there is like, that is no longer happening. We still have a QA function, but it's on engineers to hold the accountability for the quality of the release. And QA is there to consult and provide feedback and do exploratory testing, but they are no longer going to clean up your shit for you. And I got a lot of protests around that, around, like, oh, well, but everything's going to break in production and we don't have enough tests and so on. I was like, well, then we better fix that. We better figure out what to do so everything doesn't break in production. And guess what? We did. And the point is not that there was zero risk or that everything went 100% smoothly, but that if you are clinging onto line by line PR review in your organization because you're afraid of what might replace it, you're going to get stuck in the slow lane indefinitely. So I do think you have to start by removing the gate and then with the senior members of your team, figure out what the process is going to be to replace it. And probably one or two things will go wrong. But if you do it well, fewer things than you anticipate, that would be my guess. Yeah. I think there's also risk in not doing anything. Right. Yeah. So for every organization right now. I think there's a very real risk that if you don't try some new things, you're going to be left behind. And you're doing your team a disservice too, because the rest of the industry is going to move on. And part of our job is to make sure that all the engineers working with us are adapting and learning the new skills that are going to be part of building products and building software going forward. So taking no action is an action in itself. James Hasik, thanks so much for joining me. I think here's the opportunity to tell people where they can find you online and to shill your products if you'd like. Yeah, you can find me on LinkedIn. I'm always getting fights, Has ignored it on LinkedIn. Our product is product guardrails for agents for this future world where lots of PRs are happening. You need some way to stay on top of what your product is, work out what it is, work out what's changing, define what you actually want changed. And that's what we do. And you can find me and my ramblings@jamesmcgill.net and on LinkedIn. I am currently a founding engineer at Actuals, where we are looking at what the world of finance is going to look like with the ability to do these incredible things driven by LLMs and agents. So if you are at all interested in finance and interested in building at an AI native company, come check out actuals. Amazing. All right, thanks so much guys. That's the pod. Today's episode of Applied Wizardry was brought to you by DAM Secure, the code security platform for AI first engineering teams. Honestly, I've used some of the big guys and they drive me a bit crazy. You get so many comments, so many vulnerability reports and most of them are just noise. They're not real vulnerabilities and they hide the stuff that actually matters to you. Not only that, I got the bill and it was so expensive I couldn't believe it. Now DamSecure is cheaper, faster, it'll do your reviews in under 2 minutes and you'll get 3 times less noise than from the big guys. They do their own model research and so they know what works really well. That is the difference between a specialist like DAM Secure and just one of the big guys. So see it for yourself. Go to DamSecure AI Wizardry. That's DamSecure AI Wizardry for your first month, completely free, and see how painless security reviews can be for your product. This episode is sponsored by Spark Launch. When my last company applied to Y Combinator, we were told there were only three incorporation options that Y Combinator would accept. One is a Singapore registration, one is a Canary island registration, but the third is the one nearly everybody chooses, and that is a Delaware C Corporation. Spark Launch has a launch package for just $499 that will register your company as a Delaware C Corporation, give you one year of registered agent service, and allows you to keep your company documents and founder equity organized as you grow. This is a great place to start, and if you're thinking of applying to Y Combinator or any number of other accelerators, you may as well get your paperwork right from the beginning. You build the product. Spark Launch helps you start the company. Visit SparkLaunch IO Wizardry to get 20% off exclusively for Applied Wizardry listeners off their already very low prices. That's SparkLaunch IO Wizardry. And get started today.