Why list prices mislead: OpenRouter Head of Insights on how to choose the right model

A long, long time ago, like six months ago. It used to be pretty simple, right? The pace of this industry and just it feels like we have all been running for the last 18 months or two years. There are more model launches per week this year than last year and last year than the year before. If you were a betting man, what do you think the next couple of months will bring? These models are hyper intelligent in some things and feels like they're not far along at all in others. Using smaller models has cost benefits and often latency benefits as well. You're expanding the portfolio of models that you're using and you need to think carefully getting the right tool for the job. You're listening to Applied Wizardry, the podcast for practitioners who are building real products and businesses using AI. Now, some of you might be feeling nostalgic for just a few months ago when it was so simple. You're either an anthropic stan or an OpenAI Stan and you just ran everything on their models. Fast forward just a few months and we've got a proliferation of models from those providers, from other closed source providers, and then a huge plethora of open source open weight models that have really attractive performance characteristics. It's not so simple anymore to know what to use, what do you do, how do you make sense of it and make sure you do the right model for the right job. Today we have the perfect guest to talk through this. Peter Walker is head of Insights at openrouter where he gets to see and analyze how people are using every model in in every way all over the world. Cannot wait to get into it with him right after the break. One of the amazing things about AI is it allows you to turn your idea into a real business faster than ever before. But you don't want to let yourself get dragged down by paperwork, bureaucracy and government stuff like that. So that's why you need today's sponsor, Spark Launch. For just $499. Their Delaware C Corp Formation package includes Delaware filing and your first year of Register Agent service. Plus keep your company documents and founder equity organized as you grow so you can focus on building the business you build the product. Spark Launch helps you start the company. Visit SparkLaunch IO Wizardry to get another 20% off their already great deals and get started. This episode of Applied Wizardry is brought to you by DAM Secure. DAM Secure is the code security platform for AI. First engineering teams get top tier security PR reviews at a fraction of the cost of other models and get through your PRs 32% faster. Don't just take my word for it, head over to DamSecure AI Wizardry. That's D A M Secure AI Wizardry for your first month, completely free. So Peter, you know, I've been following your work for quite a long time. You were actually on the previous podcast that I hosted, the startup podcast, when you were doing the insights work at Carta, watching people's cap tables and the development of different capital raises and so on, and it was so fascinating. You've got a real gift for sorting through the data and actually providing actionable insights and a better understanding of the world. And so when I saw that you moved to openrouter, it was really exciting for me because I thought, this is perfect. This is exactly what the whole AI industry needs as well. I imagine you thought the same thing. And so now that we have our new podcast as well, Applied Wizardry all about AI, it's just wonderful to have you back to talk about the new stuff. And so I know you recently released a State of the Models deck. It's really interesting and I'd love it if you shared some of the insights. But the real thing here is for our Applied Wizardry listeners is the so what, what should you take away from this that might actually affect the way you build software, the models you choose at the providers you choose, and so much more. Well, I appreciate the invitation again to come on the new podcast. I think one of the misconceptions that I certainly had when I joined only a couple months ago is that the competition in AI is really exclusively focused on the model layer. Different labs coming out with different models at varied levels of intelligence and cost and sort of the competition lives there. And what I've realized after only a little While here at OpenRouter is there is a second, perhaps even more intense competition happening at the inference provider layer. So when you go one layer down and say, where are you actually serving that token that you use from instead of what model you're running it on? The the competition actually expands to another 100, 200 plus players with everything from a couple of folks in a garage with a few GPUs to core weave Cerebras, the whole massive inference layer that we've been talking about for a while now. All of those providers do better and worse at various models and they can have a massive impact on what your actual experience of these models happens to be. You know, if you're on X all the time, you probably read a model launches and then two people directly opposite each other and your feed will Go. This model is amazing, here's why. Or this model sucks, here's why. A lot of the difference between that, in my view is the tokens that they're being served by the providers, not the actual native intelligence of the model itself. That's fascinating. And I have to admit I haven't dug very deeply into that and I imagine a lot of people like you say haven't. So going from the walled garden of Anthropic and OpenAI, although even they have different providers, right? You can get it through AWS bedrock and so on. And you know, if you're interested in data sovereignty, even that's relevant. It definitely is. Yeah. But you're talking about open source models, right? Open weight models, of which there are many, many. And unless you've been living under a rock for the last few months, you would know that these are becoming extremely, extremely capable and are taking up an increasing amount of the global token consumption. And there, like you say, it's a bit the wild west. And I have to admit, I went into OpenRouter and I'm like, okay, I want to use this model. And then there's a list of, yeah, like you said, dozens, hundreds of providers. And I'm like, I don't know, how do you choose a provider for a given model? I think it's one of those things where once you scratch the surface, you realize, as you probably did, that there's a whole world of considerations that I'd rather actually not spend my brain space on because it is, it is another job in and of itself. And so that's obviously part of what the open router thesis is, is that we can do that for you. But when you break down a provider over different levels of quality, the obvious one is price. So a new inference provider may be trying to get into the deep SEQ market, for instance, which is a very, very deep market. There are hundreds of inference providers that can serve you deep seek both inside and outside of China. If you're trying to get into deepseek, which has a ton of token traffic, you might offer a very, very low effective cost rate. So your input and output tokens would be much lower than even per taps getting them from deep SEQ themselves. But that pricing is a function of many, many different things. So one is the list price might be lower, but if you have a terrible cash rate or if there's other behind the scenes impacts to that pricing, you may get a great sticker price and then you get the bill at the end of the month and you go wow. I've been served a lot of more expensive tokens than I had expected. And then that pricing obviously relates very deeply to reliability and customer experience. So can you serve things quickly? What is the latency on this inference provider? Is it reliable? Am I often hitting a 429 because, oops, my provider has run out of capacity and now I'm sitting there hitting send again on this chat message and it keeps throwing an error back at me and I'm annoyed. There's a ton of different little failure rates there. And so bouncing between these providers actually proves to be a really important way to minimize cost and to make sure that there's maximum reliability across both your engineering and other functions across the company. So just to summarise that there is the headline price, then there is the actual price, which is often to do with caching policy, which is not part of the model itself, but to do with how the model is served. Then there is latency and there is reliability. One I didn't hear you mention, but that would definitely be on my mind, is privacy. Yes. What happens to all of the data and all of the prompts that I'm sending to this provider? How do I think about all that when I'm dealing with open weights? Yeah, the question that you're asking is around ZDR0 data retention. And so some open weight providers are ZDR and then others are not. The additional layer with that is whether or not ZDR truly matters if the token that you are receiving is from a specific area of the world. So in particular we're talking here about China versus not China. There are providers who are ZDR whose data centers are based in mainland China, who some American companies for instance, will not work with even though the policy on its face seems to be good enough. You can make that decision for yourself. ZDR also has many sort of different layers to it. I would hazard a guess that this is true for almost all especially large scale businesses. They care more about data retention policies even than they do about cost. You know, it's almost a non starter if you, if you're not offering me ZDR inference, you're just not going to win my business. So it is rather widespread across most inference providers and that means they don't train on the prompts and they don't retain the prompts. And those are two separate things. There are some people who allow for zdr. For instance, we host stealth models often on the platform. Those stealth models, you don't know who it's from. It's like this guessing Game online to figure out who the stealth model is from. And they're often free. And in order to pay for that cost, they are not zdr. I thought Meta is quite interesting because they actually have for their muse, spark, right. They have what they call the contributor mode, right. Where tokens are super, super cheap. But they explicitly say we're going to be training on this stuff. I assume when you're using open router you can configure it to say I only want ZDR providers. Absolutely. I guess, and maybe this is my naivety, but I know I'm thinking of some of these sort of dodgy gray proxies that they sort of sit on top of like a bunch of subscriptions to Claude or whatever and offer the inference for cheaper. And they do various things with that data, including selling it. So I often wonder, you know, there is nearly in my mind a hierarchy of you've got the full ZDR where you get some sort of guarantee that your data isn't going to be used at all. And then there is, your data will be used to train models in some sort of responsible way. And I think of metaspark, musespark, contributor mode probably being that. And then you've got the sort of the black hats who are basically just taking all your data and selling it to the highest bidder. I assume that there's a bunch of that activity happening if you've got hundreds of providers, some of them are just, you know, it's not even like they're using the data responsibly. They're just doing, you know, selling pii, doing goodness knows what with it. Yeah, I don't want to be naive here. There's certainly going to be bad actors in the ecosystem, either misrepresent their policies or act incorrectly with your data. We do a lot of provider analysis and onboarding and review before we onboard a new provider in order to sort of get rid of the worst of that stuff. And that's kind of an ongoing battle, to be honest. I think that this is part of the reason why many businesses have been until recently reluctant to put their toe into the open weight world a little bit because there is this idea that it is just far less secure than the closed frontier world. I'm happy to say that that view is, is certainly changing. We see, we see about 120,000 US organizations. So that's companies from, you know, AI startups to big, big, large organizations that use open router on a daily basis. And they have strict policies, they have a ton of internal security review and they're also finding that there are many open weight waddles that pass or surpass that that threshold and they're happy to use them in production. So I think that's changing and I think it will continue to change. I think another big misconception that's useful for people to understand as they get into this world. It is very rare that a company will say, be a cloud subscription shop and then flip over to open mate models and cancel all their cloud subscriptions and just not touch anthropic anymore. It's almost always a yes. And you're expanding the portfolio of models that you're using. Sometimes you'll need Opus 5.5 for its level of intelligence and other times you're answering emails and you don't need it. So you're going to use a different model for that use case. It's very rare that you say you're just going to cut off a lab entirely and just go all the way to open source for absolutely everything. Yeah. And of course sort of two big categories of use case for inference, like the top level ones are for development use. So using Claude code or whatever to build your product and then using the tokens in product as some AI powered feature. And I don't know what you're seeing on OpenRouter in terms of open source models, what the division is there to the extent that you're able to distinguish them. Because I think for me and a lot of people I talk to the way I think about my own internal workload where I'm still small enough that I can get a subscription. I get lots of cheap tokens on Claude, so it's difficult to pull away from them to an open weight model. But then when I'm talking about my production workloads and I'm thinking about different things, I'm thinking about cost and margin and latency and so on, I'm much more open to exploring that full spectrum of different models and seeing which one fits each use case that I have. Yeah, I think that you nailed it. There's going to be more willingness to experiment in production for a couple of reasons. One, obviously any savings that you win back in terms of cost per token in the production side go directly to your bottom line. So that should be something that every business is consistently optimizing for. But beyond the cost, there's also the idea that we've probably all experienced it like Claude is down quite a lot. There are days in which Claude becomes rather annoying to use because you are Getting a ton of 429 error rates on open router. If you have that set up, the moment that that happens, you fall back immediately and your users don't experience anything to a different model. So that's a lovely sort of fallback use case. It's really interesting. We have data on what kinds of workloads move to different models more quickly and I think this is perhaps understandable on its face. A lot of the coding agentic workflow, execution, the frontier ish workloads, they move models really quickly. A new model comes out, it comes in very high on the benchmarks, People are going to trial it out for big, interesting coding experiments. But if you are, for instance, a business that has a lot of customer support that you need to get through, maybe you don't need that frontier intelligence and you're less likely once you find a model that works to really be incentivized to try the latest and greatest every single time, it's just kind of not worth the headache. A lot of businesses are again subsetting their AI experimentation to agents and frontier coding on the one side and more like routine business operations on the other, and those move at different rates. I think that makes a lot of sense. I did want to ask. One of the things that I've seen, especially with frontier models, both from the big closed labs, but also from a lot of the open labs, is that increasingly models have become reasoning only models. Reasoning is kind of a strong part of how the model works. And what that means is that the latency tends to increase and also the latency tail in particular gets much longer because it maybe normally responds in one second, but some prompts it'll take 10 seconds. And for a lot of use cases, that's actually problematic. And it's an area that I feel I've seen some of the labs abandon it somewhat. So I'm curious from what you've seen, if you've got data on this, how people are responding to latency. And of course another part of that is if you have these latency sensitive workloads, are there particular models or approaches people are taking to doing that? Absolutely. I think it's a fascinating area of exploration. One is to say that you're right to point out most of the frontier models and certainly feels like every kind of new release these days is default a reasoning model and in sometimes exclusively a reasoning model, there's basically no way to turn the reasoning off. When it comes to the actual reasoning, though, that's when I think sub models will be used a lot more Efficiently. So if you take for instance, Claude Opus 5.5 or Claude Haiku 55, Haiku is going to be much faster on a time to first token basis than opus. And so if the model recognizes that this sort of request, or if you specify this kind of request, needs to prioritize latency over, say, reasoning or output quality, I think it's easy to switch from one to the other. The other thing that we're seeing is that there's a whole other set of models that have just recently come on the market, decision models led by this, the model jev, which was released by Type Safe, where effectively what's happening is it's removing the LLM part of the model and saying, no, no, no, this should be done as quickly as possible because all the user is asking us to do is to make a decision, yes or no, which one of these three options, et cetera. And there you don't need to sit and reason and philosophize about it, just like very quickly optimize for what you think is the best answer. I would very soon expect that that sort of little decision architecture will be within the wider models as well as able to be used as a model in itself. So you've already seen this at OpenAI released a decision model as well. So hopefully that speeds up the parts of those requests where you're like, no, what matters is the speed. That is the number one indicator of a good user experience here. Yeah, it used to be a long, long time ago, like six months ago. It used to be pretty simple, right? You either used Claude or you used GPT. And now there are so, so many models and you can't just choose one model. I mean, you can, but it's rarely the case. And you know, you've got data in your deck there as well, right? The number of models used per company or per account is going up. The number of models that reach, what was it, 1 trillion tokens just keeps going up. And it's partly because of competition on the same axes, but it's partly because models are becoming differentiated, I think is the word I'm looking for. Right? They're good at different things. And what we can see with the JEV launch is there's still an appetite. I feel there is a hunger, some sort of latent, implied hunger for a certain model type that when it launches, you can get a very quick foothold. People are like, oh, that's what I needed all along, whether I knew it or not. That's the thing I've been waiting for. It's the, the perfect example of jagged intelligence, which is to say these models are hyper intelligent in some things and feels like they're not far along at all in others. And on top of that, the pace of this industry and just feel, it feels like we have all been running for the last 18 months or two years. The running sort of, I think makes you forget how young this really is. And so the idea that something like jev, which is less of a new model and more of a new architecture type, within intelligence widely there will be more gems, there will be other parts of the stack that get optimized or revolutionized by a new company and that will continue to happen because we have the very early innings. I know it's cliche to say we're early innings, but it is 100% true that this is by no means a mature industry. To your original point though, the way that people use models I think is interesting. So yes, it's very true that the organizations on Open Router probably used between seven and eight models maybe at the beginning of last year. Now they use around 12 models on average a month. So there has been some growth there. But what we thought was going to happen, what I thought was happening was that you would take say 5% of your tokens and we'd use them across many different models. So you have 5% here, 5% there, 5% there, etc. And what's actually happening is what we would call like a hub and spoke sort of design, where you have one model that's kind of your daily driver. It's your main model. It's the one that you go to, specifically your engineers in particular go to when they are doing coding tasks. It is going to take up the lion's share of your tokens and almost certainly the lion's share of your spend as well. And then you have like a host of other models that used, maybe they're used again in your customer service department or your sales organization uses this kind of model to do X, or your creative organization uses this kind of image model and really your engineers don't touch it very often. So you have these other models in the mix. And that specialized sort of intelligence doesn't actually come close to the number of tokens and spend that you're putting on that main model, that daily driver for you. And then the big question becomes how often do you switch that daily driver out and in? And it turns out at least the people in OpenRider do so fairly frequently. But others might take a couple months or Maybe even close to six months to switch from, say, this is my driver to that is my driver. Well, like I said, in terms of personal use, it's fascinating to watch the industry dynamic because of course OpenAI and Anthropic are both trying to lock people in and they're doing a pretty good job. Their first innovation was the subscription, I would say Their second big innovation is the desktop app, which in both cases is getting pretty good. And I switched from CLI to desktop app a couple of months ago and it's hard to go back. There is something nice about a ui. The UX is great and of course there are great open source harnesses as well. You know, there's PI, there's opencode and so on. But when you have to switch not just the model but your workflow, or at least your window into that workflow, it becomes more difficult. And I can tell you, for example, at the time of recording, this is mid October 2026, Gemini for Argon just came out and it's supposed to be really good and I'd love to try it out, but not quite enough to actually switch to anti gravity or whatever it is I have to do to try it out. And so I haven't. And so that's sort of lock in that they're looking for. If I look at my own workflow, when I'm looking for something to use in production for a specific workload, I will actually instruct my daily driver to say, help me find the right model for this. Right. Like here's my open router key or whatever and let's do some evals and figure it out how. Have you seen maybe people both inside OpenRouter, users of Open Router, what are the patterns here of actually evaluation of models? It spans a maturity scale for various companies. So I'll kind of walk you through the path that we see a lot of organizations take. The first is we are subscribers to Claude or ChatGPT. We are a or B, we've chosen a side. Fairly soon you kind of look up and you say, well, okay, maybe we don't want to be tied into just one dominant lab for many, many reasons. So you might actually expand to the other. Or you might say, we're going to dabble in open weight models for the first time. And when you dabble in open weight, what you typically do is you go and find these benchmarks. So you might ask your agent, or you go into yourself at artificial analysis or the many seemingly endless benchmarking sites now available to you, and you Say, okay, I am in the legal space. My use cases are a lot of text classification and all these kinds of things. Find me the best benchmarks for those specific use cases. I'm going to grab the two or three top models that are not cloud models for that and I'm going to trial them out. Okay, great. That's actually not a bad way to think about it, to be honest. But very quickly, what happens after that is someone in your engineering. Org will go, look, we can't be reliant on solely these external benchmarking. We need to evaluate these things for ourselves. How do they work in our own app, and that's the eval process. So again, you can do this yourself and you can write the evals and then back test them and continually do it. Or a lot of customers are now trusting open router to help them do that. With ori, which is our kind of eval tool. The amazing thing about that is one, it is very likely that this is going to be a headache or overhead on your engineering team if they have to continually do it against a portion of your code. So the best kind of evaluation is something where you say every couple weeks, maybe every month, we're going to take the top five projected models on this task and we're going to use real code from our actual application, test it on these models, and if there's a new model that scores better than our current one, we're going to sub it in. That is something that you can do very easily with ori. And I think a lot of organizations are finding it useful when they don't have to spend as much mental cycles to stay on top of what the best is. Because evaluations and benchmarking, they're a very deep rabbit hole. You can spend a lot of time trying to figure them out, trying to back test it against important things, trying to see what's most useful for your organization. I think that a lot of companies are finding it pretty annoying to do because it's like an added engineering task that they hadn't planned for originally. And it's a new type of engineering task when I say that it's not new at all, but it has come into a much broader relevance and prominence. And I think we're planning a whole episode on eval and this isn't it. But I think the key difference for people to understand between testing and eval is that eval is dealing with subjective factors, right? So if you have a bunch of unit testers like you either pass or you don't with evals, it's not as simple as that. And so it's a new craft, it's a new way of thinking that you need to learn. I think it's important to note that the evaluation space is a major reason as to why everything outside of coding has felt slower to adopt AI in the same way. There just isn't the same level of verifiability outside of written code bases. So what a great ad looks like, yes, you can use conversion rates in that sense, but like, what is fantastic writing? How should I think about this workflow? These are all kind of messier evals. And so I think that eval writing is a skill set that every company will need to adopt. And the rate of the maturity of those companies will be vastly different depending on when you start. That's right. So, you know, if you're building a chatbot, which I'm doing, one of the things you want to know is how natural feeling is the conversation. Well, you're not going to write a unit test for that. Right. It's fundamentally subjective and you need to think about what's a cost effective way to test different models and figure out which one is the right one for me. That's also why I think a lot of people don't switch the models that do the talking nearly as often because they get attached to a form of writing, even if the intelligence is lower the style. I mean, the Opus 5 to Opus 5:5 jump for people was like a breath of fresh air because Opus 5 was such a terrible writer. At least 5, 5 you can read it a little bit. And that is, I think that candidly, I don't know why Anthropic and OpenAI don't spend more time on the writing. It is the interface through which most of their models are understood. I think it would be worth so much more effort to make them brilliant writers. Yeah, I agree. And I think the whole Opus five gate kind of proved the point, right, which is we talked about this in an earlier episode actually with one of my other guests, is that it wasn't just annoying and frustrating to use, which it was, but what you actually got was poorer performance than the raw capability of the model. Because if you cannot communicate effectively with the model, you are not going to get the best results from it. So it's not just a sort of nice to have thing. It actually, in terms of the manifested intelligence as opposed to the theoretical intelligence of the model, it makes a huge difference because you are collaborating with humans and anybody who's ever worked in A team knows that the intelligence of the collective, of the team depends very much on effective communication and collaboration. So, yeah, I completely agree with you. Hopefully that Opus 5 moment woke the labs up. It definitely woke Anthropic up to, you can't do such a terrible job as that 100%. But the question is, can they optimise it further and are they invest as much in it as they should be? Yes, yes, yes. Well said. There is nothing more annoying than a model that you know is smart, but you're never going to try again because the writing is such crap. I was a Claude Ride or die until Opus 5 and then I got tempted across to OpenAI and now I'm dual home. So even though Opus 5.5 brought me back to Claude, they opened the door for me to use OpenAI again. And so they definitely lost something from that. And yeah, I know you've got data that shows very clearly that whiplash that Anthropic suffered from this. Yeah, it was a big moment. There's this weird thing that's happening now around the models and retention and users hopping from one to the other, where it's both a mix of real world experience plus the media narrative. And I do sense when I have these conversations that there has never been a bigger gap between people who spend a lot of time on X and people who don't. It's, it's, it's likely bad for your mental health. I'll speak from experience here, but it is just, it is just the case that there is this like very online media narrative swinging back and forth and then there is people who, you know, touch these models on a daily basis but don't really care what's going on on Twitter. It is a, it's night and day, candidly, and I think it's, it is definitely infecting the brains of a lot of people who spend a lot of time there. Could you be specific? Like, what do you see as the difference between the two? Well, one, everything is, you know, the end of the world or so. And you know, speaking of anthropic reported numbers that came out yesterday after they had an investor meeting, you know, it was high, but not high enough and therefore the entire trade is screwed. And get your money into your mattress because there's a coming recession, etc. Etc. I'm sure that the narrative will be different when I log back into X later today because it's always changing, but it definitely contributes to this feeling of both the hamster wheel. We're always running, running, running but not moving. And also this idea that every tiny little bit of new information matters very deeply when you look back and you go, actually, most of that didn't matter at all. I don't know why I was spending so much time thinking about it. Yeah, it's fascinating. At the same time, it is the best source of new information. It's the only place where you get the live wire for sure. That's right, Jev happened on X before it happened anywhere else. Now the question is, does it matter? Because a week later everyone knew about Jev, but nonetheless it's true that it's where it happens. I wouldn't suggest your whole company needs to be on X, but have one or two probably younger people. Younger people with no existing mental health conditions that might be exacerbated by being honest. I'm not sure you can test for that, but yes. Now I wanted to take a moment to talk about cybersecurity and one of today's sponsors, DamSecure really caring about building secure software is one of the key differences between agentic AI software development and mere vibe coding. And DamSecure specialize on providing security PR reviews for you and your AI first engineering team. They do their own research on the latest models and they learn your code base. The result is you get PR reviews that are security focused and faster and cheaper than the general models with far less noise and unnecessary reports. So you can focus on fixing the security issues that really matter and get back to building for your customers. Dam Secure is offering applied Wizardry listeners one month for free. If you go to DamSecure AI Wizardry, that's DamSecure AI slash Wizardry to claim your first month completely free. Building with AI allows you to get your business started so quickly, but to do that right you need a partner that helps you do the paperwork and the bureaucracy at the same speed. With today's sponsor Spark launch, you can create your startup in seconds using their platform or you can even plug in your favorite AI tool and it can talk to it directly. For just $499 you get a Delaware C corp for formed now, Delaware C corporations are a really great choice for startups all over the world. A lot of investors will only invest in Delaware C Corps as well as the registration. That $499 package helps you keep company documents and founder equity organized as you grow. Now for Applied Wizardry listeners you get another 20% off the already fantastic price. So go to SparkLaunch IO Wizardry. That's SparkLaunch IO Wizardry to claim another 20% off to bring us back on track. Peter, there is a surprising amount of material in your state of the models deck about caching and we've already touched on it. It's a topic that I find quite fascinating as well. But alongside providers, I think it's one of the areas that is least understood by practitioners of using LLMs. So what have you seen about caching and what's your takeaway? I guess, and advice around that it is the hidden factor which determines your costs. So it's incredibly important that entrepreneurs and people in companies who are using AI in any sort of agentic fashion understand their cash rates. So first thing, if you take a step back, this didn't matter that much before this year. And the reason was because there weren't really in production that many agents before this year. But it is a sea change in the actual usage trends between agents and humans. And so our data shows really clearly this sort of takeoff moment for agents, which was in February of 2026, where agents used more tokens than humans for the first time. And since then it is astronomic, like agentic usage is going to swamp human usage forever, I think forevermore. Agents tend to do a lot of things differently. So what is an agentic call in an API in our case, where we label these on an API key basis? One, is using a lot more tools, two, the response times are obviously much, much faster, basically instantaneous. Three, it is using a different sort of cache rate, it is calling different tokens. And four, maybe most importantly, there's far more turns in a conversation. There are hundreds and hundreds and hundreds of turns and each of those turns it is rereading the entirety of the message that it's been sent. That is what is in its context window is the sort of short term memory that it relies on to make decisions. And so it's reading everything again and again and again. And each one of those reads, if it's in cache, comes out of 1 50th or even 1 100th of the cost if it was not in cache. So if you go, you know, asking questions back and forth on Claude, if you switch from one CLAUDE model to another, it used to be that that wrecked the chat, it wrecked the cache and you have to re cache everything. And so that one next message is going to cost you 50 times more than the messages before it. So as you can imagine, like keeping the cache warm, keeping as much of this context in kvcache as possible is basically the whole name of the game. And so both the models are optimizing for this. The inference providers are optimizing for this. We at OpenRouter are optimizing for this. Right. We don't want to switch you to another model in the middle of a session and have you lose all the cash and cost you a bunch of money. That would be really bad for us. So it's like the rate at which you can get cash. It used to be, I think back in the beginning of the year, something like 50% on agentic work. It was really bad. And now it's closer to like 85% on median and it'll be over 90% soon and then over 95% and soon it'll just be. Almost all the tokens that are being used are being spent in cash and cost you way less. So this is part of the reason why, even with all these cool and very intelligent and sometimes very expensive models coming out, the blended cost of tokens is falling month over month over month. So is there anything as a builder of products that you should be doing or thinking about when you consider cash dynamics and the importance of it? So first off, if you're not tracking it, if you don't know, especially around your engineering teams, what level of your agentic tokens are cache, you should be tracking that very closely. Two, you should be running routine look backs that say when the cache broke, when it stopped caching, when we had a bad hour or whatever, what happened, so we can fix those things. And third, you make choices around your agentic models based on cash rate as one of the major inputs. So if you simply look at, say a list price, this many per million tokens input, this many dollars per million tokens output, it tells you almost nothing about what you will actually be paying to these models. If you don't know the cash percentage, you will either way overestimate your cost or you won't get this decision right. So you kind of have to understand cache and what these models are doing and it's not hard to read in your own telemetry, like what's the cache rate, how is it failing, etc. If you, for instance, wanted to use deep seq and that's your open weight model, and you wanted to sign up for the Deep SEQ API by itself, that is an interesting way. You get one cache right there and it's being optimized by one company instead of benefiting from the massive competition at the inference provider layer and having 100 companies that are optimizing for cash instead on the same model, in some ways it's outside of Claude and OpenAI signing a direct contract with an open weight model, to me makes less sense because you're just not benefiting from the competition. Yeah. I think like with a lot of this stuff, it's developing quickly. Try to offload as much of the thinking to someone else, like you said, but be aware. I think that the key takeaway here is awareness. In the same way that we talked about models, everyone knows about models, everyone sort of knows about effort. But providers and then caching are the other two big things. If you take away from this episode that you should really have front of mind. Something that I've noticed in my own personal use, Peter, is that the Frontier has moved so fast that I no longer feel that the most expensive model is the best for my needs. Right. It used to be that I'll take as much model as I can afford. Right. Because the Frontier models gave me better performance. But like even in my daily driver use, I see that Sol 6.1 and Opus 5.5 are my sweet spot. Fable and Astra, I can't quite tell the difference anymore. Right. And so I think we now have much more of this sort of Pareto optimal front where the question is, what's the right sweet spot for performance? And one thing you were saying is that the labs are picking up on it more and we now have some really cost effective options. So what are the trends that you're seeing there? You put your finger on something really important, which is I think before the summer, both of the two Frontier labs, and you can even include Google into this, although much to a lesser degree, they were very focused on having each release be far more intelligent than the last. And the cost, not always, but effectively be damned. Right. We're going to put out Astra, we're going to put out Fable, we're pushing the boundaries of this intelligence frontier. And I think, I do think over the summer a lot of people felt the way that you have, which is to say, okay, Opus 5.5 is really damn good. Like I'm not doing things on a day to day basis that require Fable level intelligence all the time. And so I'm going to go at their daily driver model that is a flagship, but not the most expensive, most intelligent. What's the difference, I think is that. And OpenAI actually did this first with GPT Luna. Luna 5.6 came out and it's a very intelligent model at a much lower cost point that kind of expanded OpenAI back into the smaller, cheaper model game where it had been at the forefront. And now it's all the way across the spectrum. I personally messed this up. I didn't think Anthropic was going to match them there. I thought Anthropic were going to spend all their time on the frontier. But their latest Haiku 55 release, while not as cheap as Luna, also establishes a cheaper, though very intelligent model for Anthropic in that way. Now these labs are kind of competing across the price spectrum and across the intelligence spectrum. And again, it just kind of puts the onus on the user to pick which Pareto they care about the most. Do I need the most intelligence for this task? Am I optimizing for cost? And then all the other considerations, Security in region, blah, blah, blah, blah, blah. Using smaller models has cost benefits and often latency benefits as well. Smaller models are usually faster. It's just become a much more complex thing. So it's complicated and it depends on your workload and you need to think carefully about getting the right tool for the job. Also, if you haven't tried a very, very, very fast model, either a nitro version of one of the flagships, or probably more likely an open weight that just runs very quickly, I really suggest that you do so. There's something ineffable about the latency speed. When it's high enough, it's like a little bit more of a dance between you and the model. It's really lovely. It's not always. You don't always need it, but it does matter. Is that saying quantity has a quality all of its own? I think it applies here, right? It's like speed has a quality all of its own and it's. It changes how things are done. And, you know, I'm frustrated. So Grok with a Q, which is an inference hardware company, they've had incredibly fast inference for a long time, but you can only get them on crappy, old, outdated open weight models. And so you can't get very good performance from them. And then of course there's OpenAI ultrafast, but I'm too poor to use that one. I agree with you, I'm in the poor club on that one as well. But let's play a little bit of inside baseball here. And I guess this part of the episode will age a bit faster than the rest. It's mid October 2026. Let's talk about the most exciting models right now. What are the up and comers for the different use cases? What are you seeing in the data? And if you were a betting man, what do you think the next couple of months will bring. I mean, the last couple of months as an indication means that we will look back on Opus 5.5, our current flagship intelligence model, and say, ugh, how stupid, how useless that thing was back in the day. It's coming soon. And though I think that the hidden point around this is that people feel as though this is a very fast moving industry, but the pace of change itself is increasing. There are more model launches per week this year than last year and last year than the year before. It's just every sort of couple of weeks it feels like the pace increases. So there's more and more to try. One, there's the models that are powering the personal assistants. So you've got instinct, you've got dot, you've got bot, you've got Anthropic. Doesn't have one. Maybe they will, maybe they won't. I think that the models there are much more akin to the Gemini Flash or the Muse set of models, which are fantastic. They're not necessarily frontier intelligence, but the speed and latency really deeply matters within the personal assistant space. So I think there's going to be more optimization on that side. I think that's where Muse in particular is going to shine. I don't expect them to rise up our rankings on general models, but within personal assistants, I think they're going to be pretty amazing. There's also a ton of work being done on text to speech and just speech and audio models in general. 11 labs just did a launch on Copenhager for nine different models there. And then here's the big one. As much as there is, there's always this debate how far behind is the best of open weight versus the frontier? And for a while it kind of felt like open weight was one or two months behind and now it feels maybe like five or six months behind. My prediction is that that gap won't actually narrow over the next year or two years, but that most people will view it as narrowing because they simply don't have use cases for the expanded level of intelligence that's coming. I feel like we're not in everything, but we're getting more saturated in day to day conversational work. And like you said, the difference for me today between an opus and a fable, I can barely feel it on the vast majority of the stuff that I try to build. I'm sure that I'm not building as complex stuff as other people, but as that saturation continues up to scale, I think more and more people will view it as it'll be Tougher to tell what's different about the new models. And that I think is a challenge for a lot of these labs because it means that they're only really impacting like a smaller and smaller segment of people with each new release. I think there's a few things to unpack there. I agree that perceived gap between open weight models and Frontier models has grown and I do wonder if it's to do with distillation and the fact that I know that the frontier labs have really tried to make it harder to distill. Do you, do you have thoughts on that? I'm unsurprised that they changed the architecture around that. And yes, I think that interestingly, if you play around with Google of Argon 4, our CEO Alex was doing this internally actually in Slack, was just asking Argonne 4, you know, questions that just standard questions that you ask the other models. Things like, you know, what's one thing that I can do to make my difficult day better? And, you know, random little questions like that. Argon 4 is very clearly not distilled from Claude because its answers are vastly different than all the other answers that these LLMs give. Most of these LLMs give like, oh, try to tackle the hardest thing in your day for two minutes and then maybe you have made progress. And Argon 4 goes on like a long philosophical tristice about who you need to be as a person. So a lot of labs were distilling and it makes everything feel kind of the same. I'd say distillation is going to go down and it's going to be harder. That doesn't mean it's not going to happen, but it will be harder. And anthropic in particular has made that a big, challenging point that will keep the open weights a little bit further behind, but it's not an insurmountable gap. I don't think it's. It changes the pace, it doesn't change the overall setting of the race. The next thing you said, and I think Claydon Christensen used the term performance oversupply for this, which is when something gets so good that the nexus of competition shifts to something other than performance. And most consumer goods are like this. Unless you're a car racing person, you don't choose your car based on the number of horsepower it has, you choose it on something else. And so I think it's starting to happen with models, as you say that, sure, the frontiers might streak ahead and it probably matters a lot for solving all of those unknown problems in maths and physics. Which is just insane to see. But for normal people, it stops mattering. It's already starting to stop mattering. And so then the question is, yeah, where does that leave us? And it means we think about different things, like all the things we talked about. Reliability, data retention, policy, latency, cost, other things. Those are the things. Sovereignty. Right. So I think one of the things with open weights, and I'd be interested in your thoughts on this as well, Peter, is nearly all the good open weight models have come out of China, but now there are starting to be open weight models coming out of the west, let's say in the US and Europe and beyond. And I think that's going to matter to people as well, or to businesses especially. It certainly will. Cost is obviously a major input to most business decisions, but I think in some ways the subscription overhang is already here. You're getting so much value out of your personal consumer subscription to code that you really haven't felt the price differential with the API and you're less sensitive to it. Perhaps in the future that will change. I think that the labs will keep subscriptions around for quite a while. I don't envision that going away, but I could be wrong. When you talk about the capability overhang, I think it comes into two things you've already made. I think one of the key points, which is if there is a capability overhang, then the labs will need to invest more into product rather than just models. The harness that you're using, the way that you experience the answers from these models, like for instance, if Fable5.5, which is rumored to come out soon, is vastly different than Opus 5. Five in the kinds of answers that it gives me, the interactive artifacts that it automatically creates, the way it can explain things to me, or if it's a much better writer, then I might use it, even though I don't quote, unquote need it. And I think that, that I think in some ways this is like a user experience focus for these labs will become more and more important as the intelligence gets a little harder to figure out. Because it's so damn smart that my day to day work maybe doesn't need it when I'm answering emails. Like I do think that there's a lot of work to be done on the products themselves that it lives in. The final thing you mentioned that I wanted to dig into a little bit is, you know, I talked about differentiation before. It's not quite full specialization, but I think for a long time the frontier has been measured primarily on coding. And then I guess coding adjacent activities. And I think that the point you made about for something like Muse Wonder, which was just released earlier this week and some others, is that there are now frontier models that are measuring themselves based on more non technical tasks. Right. Sort of agentic workflows that are long running but are not about writing code. And that's clearly what meta are going for and maybe some others too, because those will be the models powering those personal agents, like you said, like JOT and Muse and Grockbot and all of those. Are you seeing open weight providers move into that space as well? Yes. No open weight provider, at least not here in the States, has sort of a foothold in personal assistance in that same way Instinct might route certain requests through openway providers. I'm not exactly sure about their backend, but most of those personal assistants are owned by one of the labs that is going to use them as sort of the showcase for their model Meta to Muse. In that sense. I do think that the idea that these specialized models will become more focused on non coding tasks, it's not just something that will impact businesses. It's also a change in philosophy within some of these labs and I'm not actually sure that many of them will make it. So let's take Anthropic for instance. If the goal is to produce a model that can create its own successor model at a better, faster rate, coding is effectively the gateway to that. And everything else that you do is a little bit of a distraction. OpenAI had a different thesis and has been far more distractible. They've done a lot more stuff, but they do also seem to have kind of gotten religion at the beginning of this year and back to effectively the same mission. So I think that in some of these models, perhaps you would have expected it from these frontier labs and it just won't happen because they want to spend every waking moment focusing on getting better coding because they view coding as the gateway to all things. There will be other models and I don't. I'm personally of the belief that you will be. We as humans will be using dozens of models in two years. There won't be a consolidation on one of these. And because of that I do think there's room for all of these specialized models. I think the place you see it most is in the creative fields. So image generation, video generation, text to speech, audio. It's interesting how neither of the major labs really even play that much in those places. They've kind of left them. Sora as a great example, was Just shut down by OpenAI. They've said, look, this is a different modality. It's not where we live. We're going to let other labs kind of win there. And so I think that there's more of that in the future. I agree with that. It's. It's a really. It's a really fascinating time. It must be amazing to be right in the middle of it, Peter, and just watching everything unfold in real time. I have a very, very lucky job, to be honest. Well, you're very good at it. So. So, Peter Walker, head of Insights, I believe it is, at OpenRouter. Thanks so much for your time. It's been a really fantastic discussion. You're an incredible follow on LinkedIn or Twitter. Where's. Where are other places where people can catch all of your insights because you're releasing them all the time? Yeah, definitely. LinkedIn and Twitter are sort of the day to day, but if you want to follow along with our weekly newsletter or deeper reports, it's OpenRouter. AI data is where you can get all of that. Fantastic. Well, Peter Walker, thanks again for your time. It was a great chat. Have a good one. You too. Thanks so much. Today's episode of Applied Wizardry was brought to you by DAM Secure, the code security platform for AI first engineering teams. Honestly, I've used some of the big guys and they drive me a bit crazy. You get so many comments, so many vulnerability reports, and most of them are just noise. They're not real vulnerabilities and they hide the stuff that actually matters to you. Not only that, I got the bill and it was so expensive, I couldn't believe it. Now DamSecure is cheaper, faster, it'll do your reviews in under 2 minutes and you'll get 3 times less noise than from the big guys. They do their own model research and so they know what works really well. That is the difference between a specialist like DamSecure and just one of the big guys. So see it for yourself. Go to DamSecure AI Wizardry. That's DamSecure AI Wizardry for your first month, completely free, and see how painless security reviews can be for your product. This episode is sponsored by Spark Launch. When my last company applied to Y Combinator, we were told there were only three incorporation options that Y Combinator would accept. One is a Singapore registration, one is a Canary Islands registration, but the third is the one nearly everybody chooses, and that is a Delaware C Corporation. Spark Launch has a launch package for just $499 that will launch register your company as a Delaware C corporation, give you one year of registered agent service and allows you to keep your company documents and founder equity organized as you grow. This is a great place to start, and if you're thinking of applying to Y Combinator or any number of other accelerators, you may as well get your paperwork right from the beginning. You build the product. SparkLaunch helps you start the company. Visit SparkLaunch IO Wizardry to get 20% off exclusively for Applied Wizardry listeners off their already very low prices. That's SparkLaunch IO Wizardry. And get started today.