Cutting AI coding token costs, with Dave Slutzkin

Wait, where's all this money going?

I think we've known for a long time that it's really important to understand your tools and to understand the detail of what's happening when you're building software. The same turns out to be true for these amazing and magical and complex tools.

If there's only one thing that you intend to care about, it should be

Great to be here.

Feels to me like 10 or 15 years ago with cloud, we're bumping into this situation where individual engineers are making decisions which are hitting your P and L really directly on a daily basis. They're doing these things that are starting to affect your budget as a CTO or as a CIO, and you might be getting tapped on the shoulder. What I'm increasingly seeing is CTOs getting tapped on the shoulder by CFOs saying, wait, where's all this money going? People are spending $10,000 US in tokens per developer per month. That's. That feels like a lot to me. It's significant fraction of a salary. So, yeah, I think that's why people are starting to care about token burn at an organizational level. From a personal level, you've written a lot of code over the years, as have I. You like to do things the right way, and no one quite knows what the right way is, but they can tell when it feels like they're going on tangents and spending tokens and spending time and spending effort doing things that are useless. So I feel like that. That basic engineering mindset comes in as well. Personally, I see that.

We've seen the orgs. You mentioned the token maxing thing. Like, we see orgs go through those kind of three stages, the first being token maxing. Everyone just needs to use this thing. We need to get adoption. We need to. Whatever. There's token leaderboards. You Know you're going to be judged on how many tokens you're using. And then the second, once you get past a little bit of adoption, the second step is this kind of effectiveness question of what are you doing with the tokens? And then the third step, which is where people are starting to bump into is efficiency. Are we spending the right amount of money to get the, get the results that we're getting from this? And that's a much harder question to answer. I find it fascinating because I think in 12 or 18 or 24 months that question is going to arise. For most white collar work. Most tasks that exist in white collar work, it's going to be, well, they all have a non zero token cost associated with them. So you have to ask the question, Was it worth $37 to write that three line email? Or maybe a human should have just done that for four minutes. We're going to be nickel and diming everything, I suspect. And I don't know if that's going to be very fun, to be honest.

Yeah, and I should note there's two aspects. There's two areas in which we're, I think we're talking about this as an industry. First is personal use of AI. When you're doing things, when you're coding, when you're writing an email, when you do it. That's the kind of personal use question. Are you using the right model, Are you using it the right way, Are you getting the right context, etcetera but then there's building AI into your product, which we can also talk about, but they actually turn out to be quite different. As you said at the time of recording, 5.5 has entirely been released now. Haiku 5.5 was released this morning, Australian

If you're building into products, very few people are going to use that manually to write code.

It's not what it's for. You get a hundred thousand tokens, over a hundred thousand tokens, the price goes up five times. So you're probably not going to be messing around with keeping your context window under a hundred thousand tokens to write code with it. But for using it in products, that's going to be potentially really valuable. So have you started looking at it yet for you?

My theory was that we would end up with specialized models, that they'd be modeled, there'd be a coding model, but the coding model would be different to a chat model and the chat model would be different to whatever. I don't think we're seeing that from the major labs yet. You know, Harvey and Lagora have some specialization for legal tasks as an example, but ultimately they are calling back the front end models behind the scenes anyway. They don't really have their own models. I still think that that might be where we end up that it turns out that a coding model needs to be different to a chat model, that needs to be different to a research model.

There is a surplus of tokens in one place though, which is inside the frontier model companies. And you saw that recently there was a woman who works at Grok, at xAI, I think her name is Lauren Tan. She goes under Potito on Twitter and she was posting about how she was using a lot of models in parallel and doing a whole bunch of things in parallel in the development of stuff and doing a lot of PRs, hundreds of PRs per day. Someone worked out what she is spending to get to that point. And it's about $380,000 in tokens a day.

There's exactly one thing that you need to care about. We have analyzed a lot of sessions for a lot of different organizations across a lot of models. We've seen terabytes of session logs. The only thing that really matters for cost at the moment is the depth of the session, how deep you allow it to get. Claude models, Anthropic models have a million token context window. If you are allowing it to get to a million tokens, you are spending significantly more than you should be. You probably hitting your limits much faster than you should be. You're probably spending dollars much faster than you should be. If there's only one thing that you need that you intend to care about, it should be that noticing when you're getting too deep in a context window. I'm not going to say too deep in a session though, because we're now better able to run long sessions than we ever have been. If I project back six months, say compaction was the devil. We would all avoid compaction with complete rigor, because anytime compaction happens, everything got dumb. That's not the case anymore. So the answer these days, the mistake is that people haven't thought about compaction. They get scared of it, or they just think, well, I've got a million token context when I'm going to use the median token context window. That's bad for cost. It's actually also bad for quality because really only pay attention to the start and the end and the middle bits are going to get forgotten anyway. So there's no point even having those 800, 900 min tokens in there. But the biggest mistake that people make is allowing it to get too deep. What you should be doing is compacting actively at 200, 300 ish thousand tokens. I have my auto compact window in Anthropic models set to about 300 or 350,000 tokens. But I will compact manually over about 200,000. I'll just compact manually. The best time to compact is when you finish something. When you finish a piece of work, but it actually doesn't matter. Even if you're in the middle of something, it doesn't matter that much. You'll notice in practice. What I am constantly advising people at the moment is to learn to love compaction. Get to know it, get to understand it, get to understand the feel of it, and just do it much more than you were before manually or automatically

Every turn, every time you send input back to the model or a tool, call finishes, which also sends input back to the model or whatever happens. Everything that happens in your session has to make another request back to the model. Every time you make that request back to the model, the model has to look at everything that's in the context window up to that point, has to run attention across every token in that context window, and it has to do something with that. Now, a lot of that is cached. I know that you have had a bee in your bonnet about caching for some time. Any of. And that is definitely a thing. Most of the sessions we see cache reads are 70 plus percent of the cost of the session anyway. So even if it's cached, that just means that your costs are less. But what you end up with is you end up with this thing where when you've got a lot in the context window, everything has to be sent back every time. The model therefore has to consider all of it every time. Even though some of it is cached, it has to still consider it for attention and understand what output it's going to give. And then it's got to charge you for all of that. It's much more expensive to charge you for that when there's 800 or 900,000 tokens in the context window than when there's 100 or 200,000 in the context window. It's doing more work, so they charge you more for that. And caching doesn't completely fix that.

I mean, we hoped a couple of years ago that or even a year ago or even six months ago that that's where we would end up, and maybe we still will, where it'll just work and we won't have to think about how it works and we won't have to think about why or how we, how we need to optimize that. But at the moment we do need to understand that. I think that's what everyone seems to be finding. You need to know what's going on behind the scenes. We've known that as software engineers for a long time, I think, and again, that's why I probably focus on software engineering and product engineering generally as part of this. I think we've known for a long time that it's really important to understand your tools and to understand the detail of what's happening when you're building software. The same turns out to be true for these amazing and magical and complex tools that are LLMs and agents built around LLMs. It can get very expensive very fast. And you'll notice also if you run locally that it gets very slow as well, which is not something that a lot of people are doing necessarily. But you can feel it in a local model that it just gets slower as you get deeper into the context window. So there's almost a, there's a, there's a pretty direct feedback loop if you're running local models.

Yeah, I mean, if you think about when you're 800 or 900,000 tokens into a context window, assuming that you'll get into that point, in Anthropic models, the cache will always expire after one hour. And again, I'm talking very specifically about when you're doing software engineering clock code. If you're using the API, you can either set the 5 minute cache or the 1 hour cache, but for most people we ignore that. So it's an hour cache. Let's say when I'm coding, it's an if I come back after one hour and one minute, that cache is gone. It's thrown it away. There's nothing left of the prefix that's been cached there. In order to rehydrate that, in order to get it back into the cache, it has to cache writes. Cache writes are expensive. If I've got 800,000 tokens in the context window, that's going to cost me about 20 bucks. $20 just to type in my first question at that point, all the questions after that, it will now be fresh. So they won't all cost me $20, but. But the first one definitely will. So you're much more exposed to that when you deepen a context window. In general, though, that is a thing that costs you a lot more is leaving sessions go idle for more than, in the case of Anthropic models, for more than one hour and then coming back because it always has to do a cache write. It's just that that cache write is proportional to the amount of stuff you have in the context window at the time. So yeah, leaving sessions idle for more than an hour is expensive. One thing that Claude Code has started doing by default is that if there's more than 200,000 tokens in the context window, it will automatically compact before the cache expires. Personally, I like that some people hate it because they're still scared of compaction. And as I've said, and I'll probably say again, you need to probably stop worrying and learn to love compaction. But it'll actually do that automatically. The other option, if you come back to a session after more than an hour, if you, you just compact manually and you save a significant amount of money even at that point, because it doesn't do a cache write, it just does a compaction on the server and then sends you back the new thing. Actually, incidentally, with compaction as well, you could look at it, include code. You can just click on the line that said I compacted. It'll show you exactly what the summary is. So if you want to get comfortable with it, that's another way to do it. Look at what the summary is and see how well it represents what you think it should. But yes, caching does matter. Leaving sessions, staying idle is going to cost you money. How much it's going to cost you is a question that you have to stay on top of. But in general, you want to try and avoid it.

And we say 10 or $20. There's obviously also, if you're on a subscription, it's obviously also a percentage of your subscription, and that could easily be. Those individual requests when we model them, can easily be 1 or 2% of your entire weekly subscription that you've spent just on that one request. So you only get to do that a few times, and then you hit your weekly subscription limit.

Yeah, absolutely. And that there's. There's a factor between the API costs and the amount of quota that you will use. In our data, that factor is shrinking over time. You get less usage from your subscription. Every month for the past 6, 9, 12 months, you've been getting less and less usage with your subscription. That's just from our data. I'm not necessarily attempting to imply anything.

All I'm saying Is that that's what we see in our data, that, yes, it is tightening up as time goes on.

I agree. Although I would say if you're on subscription, you're not hitting limits. Then don't waste your time worrying about it too much right now. If you're not hitting limits, you're not hitting limits. Awesome. Like there's. That's fine. I mean, if you care about the environment, maybe you want to burn fewer tokens for a particular result. You know that there's logic there. But yeah, maybe you're not hitting limits. That's great. Obviously you're doing the right things and your usage is. Pattern is such that it's not. But most people are starting to hit limits way more than they used to be. And we certainly work with a lot of people who have multiple subscriptions. They would rather probably get a little bit more out of those multiple subscriptions than pay for yet another.

The core principle for me here is I said before you got to know your tools, I see a lot of people who call back to benchmarks that they've seen and say, well, according to this benchmark, this model is equally capable to this model and yet slightly cheaper. That may well be the case for your use case, or it might not. Maybe they benchmax for that particular benchmark. Maybe your use case is different in some way. Maybe you're writing your app in Elixir and this model happens to not be very good at Elixir or whatever that might be. You definitely see differences across companies. Certainly the bigger companies that we talk to. I'm talking to companies with 3,000 plus developers who are building internal evals for this. From a software engineering point of view, they have internal evals to see how well this new model works with our particular use cases. I don't even know how realistic that is, but I appreciate that they're attempting it. I think that's probably the right way to Go. As an individual though, unfortunately, you just have to try the new models and see how they feel for you and get a sense of what works best. I'm talking about models and we'll talk about thinking modes or reasoning modes or effort modes in a minute. But across models. Personally, I use Astra for a bunch of stuff, but I get it to delegate to sub agents that are cheaper. And I actually, I don't do that in any high tech way. All I do is say to it, delegate the grunt work to cheaper submodels or cheaper sub agents. That's all I say in every prompt. I don't even have skills around this. I don't have any sort of magic. I just say that in every prompt it takes me four seconds to type. And generally the models, in my experience, both Fable and Astra are pretty good at that. I have a subscription, we have subs to both. So I will use Fable for a bunch of stuff. I'll use Astra for a bunch of stuff, especially if it requires more thinking. I'll use Opus if it requires thinking. But I don't necessarily need a fully fleshed out plan and I don't necessarily want to go back and forth with it a lot. I just want it to do, you know, basically the obvious stuff. I like Sonnet for sub agents, I like Luna for subagents. I've basically never used Terra and I never really use Haiku. I think Sonnet's cheap enough and does a good enough job for coding. Although I haven't tested haiku 5.5 exactly. As I said before, I need to test it to find out if I feel like it works for my particular use cases. I feel like Haiku might do a good job, I might not. So generally you need to get the grain of them. I don't know if the GROK models are good because I just haven't tested them enough. I don't know if most of the open models feel good to me because I just haven't tested them enough to understand. And there are no benchmarks that personally I trust to align with the way that I feel the models work. We can talk about effort stuff in a sec. But do you agree with that? Is that something you've seen?

Yeah, because the difference between those two models was the language. Right. It wasn't so much the code it wrote, it was how it talked to you and how it interacted with you. And it's pretty hard to.

I think if we look at the metagame, though, that agents are playing, that was an experiment that generally LLMs were making. The collective LLM consciousness was making was an experiment in seeing if you get humans to do what you want them to do just by baffling them with words that they don't understand. So I think that's going to come back with models in the future where they really want us to do something. I do think there's something to that, that we, as a, as a culture and as an industry are exploring the space of what models, not only what models can do, but how they can do it and whether that's a good thing or a bad thing for them to be doing it in that way. Maybe that Opus 5 style of interaction is completely fine for particular situations where it's non interactive, as you said. Maybe that's actually going to work better in some situations. I just don't think we know that.

Yeah, I mean the way, as I sort of outlined the way that I do it is to use the frontier model, use the expensive model, but tell it to delegate to cheap sub agents. And for me that generally feels like a strong enough pattern. It does a good job. It doesn't hit my limits too fast, it doesn't rack up too much cost. Most of the time you got to still keep an eye on it. You've got to still be interacting with it. You're going to be working in multiple windows at a time. That's okay. It's going to be slower on some things, but then thinking modes come in. So the point of thinking modes or reasoning modes or effort modes is that there's a parameter in the harness which tells the agent effectively how much to go back and forth thinking about a particular thing. Just it's a dial essentially how much as you go back and forth. And yeah, there's a couple of special modes. Like UltraCode is actually a special mode, but we'll ignore that for now. There's a dial as to how long they should take on each thing. You can see that if you actually inspect the reasoning tokens, which you can do in the session logs. If you inspect the reasoning tokens, you'll see that if you turn the thinking mode up, it'll talk for hours. Like this podcast, it'll talk for hours about something, whereas if you have the reasoning mode turned down, it'll do a little bit of talking and then it'll do the tool call or do the next thing or whatever it needs to do. So that's directly proportional to the cost and to the amount of time these things take. So one thing that I have found useful is that I will almost always run Astra on low or medium thinking mode. I very rarely see a situation where I think this justifies high and I never see a situation where I think it justifies X high or max thinking modes. Those are very expensive modes in general. The answer is for the top level models in general. My advice, strong advice, is to not use those if you have infinite tokens. If you're working at a Frontier lab, good luck to you. Use whatever think mode you want, I don't care. But it's also slower, so you need to. Then you will naturally want to be doing multiple things in parallel because you'll ask it a, you know, ask it to plan something and it'll take 45 minutes. Whereas if you're on low or certainly Astra on low is really quite fast.

And they are defaulting to low and medium these days. And I think that's right. With those big models, your sub agents can think for longer, that's fine and they probably should, especially depending on the size of the agent. But I usually just leave that to the model to sort that out. It'll run Luna on X high and that's fine. I don't mind that. So is that what you do as well, Yaniv? Is that your general vibe? You're going to run on lower effort modes or lower thinking modes, your reasoning modes?

And this is all frustrating. I'll acknowledge up front this is frustrating because you're constantly having to reevaluate and reassess where you are and what the right thing to do is and what the right process is and what the right setup is, both from an individual perspective and from your team perspective. Maybe the skills that were put into your repository by someone very well intentioned six months ago are now actually negative and that's really frustrating. And now you have to have an argument about it with that person or with the rest of the team or with your DevEx team or whoever's running it in your particular company. But unfortunately things are changing so fast. That's true. Six or 12 months ago everyone was building their own harness, everyone was building their own, oh, we need this and we need that. It's going to be incorporate our workflow and it's going to delegate to exactly the right sub agents, going to have the right correctness gates and it's going to do all of the, you know, there's going to be ReAct built in and all of this stuff that turns out now most people seem to be moving away from that again. Yes, some organizations still want that for consistency of process and correctness, to make sure the right quality gates are being ticked. But even those seem to be lighter than they were previously. We're ending up with this harnesses. The harness engineering piece is taking a bit of a back seat at the moment. Honestly, I don't do very much at all from a harness perspective. I just go into pretty vanilla Claude Code or Codex and I say do the thing and use sub agents and it kind of gets it done.

A pattern that I've noticed over the last 12 months of this is that the people who are getting the most out of agents, agentic, dev, agentic, software engineering, whatever we're calling it, and the people who are most comfortable with it are people who are already used to delegating people who were managing before that for exactly that reason, I think, because you're used to giving the why and leaving the details to someone else. Whereas engineers who've never managed, never delegated or never enjoyed doing that are often finding this more frustrating because they're having to learn delegation on the fly. I completely agree with you. What I try to do with the agents is to give them the rationale behind things. You know, they'll say, I need you to make a decision A, B or C. I don't feel tied down. In fact I specifically in Claude, I specifically in anthropic models, I specifically tell it not to use the Ask User question tool because I don't like that that promotes just selecting A, B or C. No, what I want to tell it is A because blah blah blah. You know, because of this reason. Here's my rationale behind that. I want you to understand what I'm trying to get at here because that'll probably color future decisions and it's more context that you should probably have. I specifically don't like just saying a

I find that frustrating.

I am not one to anthropomorphize these models. I don't think they're human. I don't think they're close to that particularly. But a lot of the ways that it's important to interact with them are similar to the ways that you would interact with your direct reports as a manager. That's right. You have to give context and allow the details to sort of be someone else's problem in that case their problem. And then you're going to have quality gates and you're going to understand, you're going to review it. That's a way that I think about it, whether that's useful for people.

Yeah, I think that's true. So effort levels matter like that. You will spend a lot more tokens on X high or max just a Lot more tokens than on low or medium. And I want to touch on speed again. You might get better quality. As you said, there's probably a sweet spot for quality and it's probably around medium or high for a lot of the models. But also the old saying is that speed is a quality of its own. Sometimes being able to run models really fast allows you to interact with them in different ways. You're able to go back and forward with them as opposed to do this thing and then you know it's going to take ages. So you either do something else or you go to YouTube or you go put the washing on or whatever and come back and you're out of the headspace.

I have fast mode turned on most of the time, except if I feel like this week's going to be particularly tight. I like Astra on low. Thinking on fast mode is really quite quick.

Fast mode, to be specific, is going to cost you two and a half times as much per token. It'll give you the same response. It'll cost you two and a half times as much or two and a half times as much of your limits, of your weekly limits for codecs. So that's something to keep in mind. I definitely talk to people who have accidentally turned on fast mode in April and it's still turned on, and they wonder why they're always hitting their limits.

that burns a lot of stuff. But yeah, I mean, we saw this with Curse's composer model some time ago and that's sort of been superseded and bypassed. But that was a very fast model and it was actually quite fun to use. It was kind of dumb, but it was actually quite fast. So it doesn't matter because you can go back and forth and say, no, you're wrong on that and it'll then do the next thing and you know you can. It's a different way of interacting.

Totally.

It occurs to me that during your generally during your recordings of Applied Wizardry, you will need to have ad breaks during the episode so that the practitioners you're talking to can keep their cash alive in the background. Because otherwise if you talk to them for an hour, all the cash is going to expire and then you're going to. You know.

I would also note though, on caching, which I was talking about before that break, it is very different between OpenAI and anthropic OpenAI cache. The anthropic cache expires at one hour, no more, no less. In general, the OpenAI cache for Codex, it's different for the API but for codecs, on subscription is best effort only. It will often expire within five or 10 or 15 or 20 minutes. It's much harder to predict. The other side of that though, the flip side is that on subscription, they don't make you pay for cash rights. You actually don't need to worry so much about cash with OpenAI models when you're on subscription. If you're on API pricing or token pricing for your organization enterprise account, then they will charge you the full whack for the cash rights. And that starts to get expensive very fast. What a lot of organizations see is when they go from a team account to an enterprise account, they suddenly pay a lot more. A lot more, like six or eight or ten times as much as they were paying before. And there are multiple reasons for that. Some of it is because you're actually paying the cost of the tokens rather than not paying paying the cost of the tokens. But also things like that where suddenly you're paying for cash rights.

Yeah.

My understanding is that in 12 or 24 or 36 months, we'll be less worried about caching if, as expected, RAM prices come down and RAM bandwidths on servers go up further, especially on local. That's going to be another question. But that they. What everyone seems to expect is that caching, it's this kind of temporary challenge, but that it will go away as time goes on, it won't have to be charged as much because you'll be able to cache a lot more and for a longer period. On the other hand, no one seems particularly confident about attention, which is the mechanism which causes the quadratic cost as you go deeper into a context window. No one seems particularly confident at this point that we're going to have a magical breakthrough which allows much longer context windows at a much cheaper price. That's just not something that anyone's seeing

We actually had a 10 million token context window llama. There were some llama models 18 months ago now that were released with a 10 million token context window. And at that point the graph was going in this direction of context windows. We'd started with, you know, GPT2, had 64k or whatever it was, and now we're up to 10 million tokens. It was only going in one direction, it was very clear. And then in the 18 months since then, it's really stagnated. As we said before, the median tokens that anthropic have, obviously they're not public with this, but it seems likely that they're doing sparse attention across those minion tokens, which means that they don't. So the intention mechanism is it has to do matrix multiplications for every token in the context window with every other token in the context window. That's why this stuff gets expensive fast. Doing sparse attention means you don't do every token against every other token. You just kind of guess at the ones that matter. My exactly. As you say, it doesn't look like context is going up. My interpretation of that is that this stuff is magic. We don't quite understand how we can get so much power out of just doing a lot of matrix multiplications and then feeding that back into itself and then doing a lot more matrix multiplications. We don't quite understand how that's possible. So it turns out that you can't magically remove that context, that quadratic relationship, without making the LLM almost useless for all the tasks that we're attempting to achieve with it.

You talked before about skills. You talked about polluting the context window because that is something that happens. It happens like all the skills that you have are going to be loaded, but that's less of a problem than it used to be because often it'll lazy load skills and do a bunch of things. Same with mcps, it's actually less of a problem than it used to be. You can still tell though. You can look at individual sessions and see how much is going in there. For some sessions it might be as much as 5 or 10% of your context window. You might have 50 or 100,000 tokens that are going into your context window before you even start the session. And in a million token context window, that's still material. The system prompt of the thing needs to be taken into account as well. Those are all things that we are broadly thinking about in the backs of our heads from a cost perspective, but also from a quality perspective. Because if you have a bunch of stuff in that context window that is pulling, as you said before, that's kind of pulling the LLM in different directions. That's going to make it harder for you to get quality results. So we often want to tidy those things up. You don't want to have those skills there that you don't use anymore. You don't want to have those MCPS there that you haven't used in six months. You don't want to have that stuff in your agents md that is just probably useless these days. It's always worth going through and tidying that up. The same is true of memories in claude. In the anthropic models, especially in CLAUDE code and CLAUDE desktop, they will do auto memory, which means they'll save things that they think look like they're probably decisions that will be useful in the future. When you go through and look at those and audit them personally and individually, you realize that some of them are extremely useful and a lot are not. Some of them are. It remembered a thing that you said once about stop asking me silly questions and it'll say, okay, user says I should never ask any questions like, yeah, no, that's not exactly what I said and I've forgotten that context anyway. So it's also worth going through those in general. You want to clean up that kind of initial context that goes into your context window. If at all possible, you just want to tidy that up a bit. It's not every day, it's not every week. Maybe every month or every couple of months. You just go through and have a look at what's in there.

Yeah, hit me up on LinkedIn. Dave Slutskin there's not many of me in the world so I'm pretty easy to find. I'm also doing a bit of stuff on Twitter on and off at various times and also have a look at, we have a little open source tool which helps with a bunch of this stuff. It's called antiburn. So if you go to GitHub.com antiburn antiburn. It's free and open source. Download it, use it, give us prs. We love prs. Hopefully it helps you understand what's happening in your sessions and where the money's going, where the limits are going.

Thank you Yaniv. See ya.