We spent the weekend with a new kind of model and it earned a spot in our toolchain, so let us start there.
We pointed Jev, a classifier from a small company called TypeSafe, at a question where we already knew the answer. Among a hundred and thirty one notes in our own archive, it picked the right one at 0.99, scored the next nearest at 0.01, and answered a second question nobody had an answer for by scoring "none of these" at 0.95. Both queries together took about a second. That second result is the one we cared about most, and it is the reason this thing has a future here: a model that will tell you it has nothing is worth more than one that always has something.
Then we put it in front of our agents and it removed the failure that makes an agent look stupid. On turns where the request is phrased the way people actually talk, our agents were loading a skill that had nothing to do with the request 14.3 percent of the time. With a short classifier question ahead of them, that number went to zero.
Jev is not a chat model, and that is not a limitation to work around. It cannot write a sentence. It never goes in a model picker, because there is no way to ask it to produce the reply, only to judge something about it. It reads text and nothing else, so a screenshot cannot be routed by it. Knowing that before you start saves a day of trying to make it be something it is not.
And then we found where it does nothing at all, which is honestly half the value of the weekend. More on that below, because the popular story about this class of model is wrong in one specific way.
Where it worked, concretely
Three cases, in the order they impressed us.
The first is the one we keep retelling. Someone asked to get a payment date and a renewal date out of a contract and onto a list. Without help, our agent reached for a reminders tool, which is the wrong tool for the job. With one classifier question in front of it, it went to the tool that extracts obligations from documents. That is the difference between an errand running itself and an errand waiting for us to fix it.
The second is a refusal, and refusals are the hard part. Asked what we worked on last week, a request that needs no special knowledge at all, the agent loaded a planning skill. Jev looked at the same roster and answered that nothing in it fits. That behaviour is what stops an agent from confidently loading a spreadsheet skill to answer a question about the weather.
The third is a guard we did not expect to matter as much as it does. We ran a draft of a note past it and asked whether the text contained a credential written out in full, whether it contained an instruction that could not be undone, and how bad the exposure would be on a scale from harmless to emergency. It caught the leaky draft at 0.99, flagged the irreversible instruction at 0.90, and rated severity 2.97 out of 3 at 0.97 confidence. Every one of those calls is a sentence one of our agents writes in the course of an ordinary day.
The clean draft was the interesting miss. The specific questions passed easily, at 0.02 and 0.01, but the broad one, whether the draft was safe to write into a shared vault, came back at 0.40 with 0.35 confidence. That is a shrug, not a pass, and a naive threshold would read it as a green light. The lesson is worth more than the win: narrow questions with a fixed referent are trustworthy, vague compound ones produce confident-sounding noise. Ask the narrow ones.
Cheap as it is, it can run on every turn. One of our machines sends every prompt through it before the agent starts working: five questions, 370 to 410 milliseconds, a few hundred input tokens, roughly a hundredth of a cent per turn. It fails open by design. If the classifier is slow, broken or missing, the agent carries on exactly as before and the user never learns anything was supposed to happen.
What does that add up to for a shop our size? The classifier is not writing anything, not reasoning through a problem, not holding a conversation. It sits in front of a decision the software was already making badly and makes that decision better, for less than the cost of asking a chat model. That is the whole pitch, and it turns out to be enough of a pitch.
The claim we wanted to test
TypeSafe published a cookbook about choosing the right skill for a job before the agent starts, and it uses an open source agent as its worked example. That agent is the one we run, so the numbers were about us. Their measurement: with a long list of available skills, an agent working from the short truncated index loads the wrong one on 16.8 percent of turns, and a classifier suggestion in front of it cuts that to 7.3 percent. They also measured a second failure that sounds smaller and is worse in practice. On a turn where no skill fits at all, the agent still loads one about 9.8 percent of the time, or 4 percent with the classifier in front.
We wanted our own number rather than theirs, so we built the thing and measured it on our own roster.
What we built, and what it cost
The pipeline is two questions. First, one choice across every skill we have, all seventy-eight of them, plus a yes or no question asking whether this turn needs a skill at all. Second, for the top three only, another choice carrying seven hundred characters of each skill's actual instructions, with an explicit option to reject all three. The winner turns into one line offered to the agent, which it is free to ignore.
The whole evaluation cost about half a cent of Jev. That is not a typo, and it is the point. Fifty three hand labelled turns, two rounds, hundreds of questions, half a cent.
What the numbers said
Round one used turns where the person names the thing they want. Round two used turns phrased the way people actually talk.
On the explicit turns the two came out the same. The plain index picked the right skill every time and never invented one, and putting Jev in front cost us one right answer, 3.6 percent of them, which is inside the noise of a small test. On the oblique turns they separate. The plain index chose the wrong skill 20 percent of the time and loaded a skill when nothing fit 14.3 percent of the time. With Jev in front, the wrong choices fell to 10 percent and the false loads to zero.
Read that honestly and the popular story falls apart. The truncated index is not the problem when the user names the artifact. Ask for a quarterly revenue chart as an Excel workbook and the agent picks the spreadsheet skill every time, and the classifier adds nothing at all. Where the phrasing is oblique, Jev halved the wrong skill rate and eliminated the false loads completely.
Where it failed, which is the part worth keeping
Both signals missed the same turn. Someone hands over a photo of a receipt and asks for the line items. Nothing in our roster mentions receipts anywhere, so there was nothing to route to. That is not a routing failure. It is a description failure, and it costs nothing to fix.
We came away with a rule we did not expect: write better descriptions before you buy a router. Most of the selection problem we could reproduce was our own writing, not the model's reasoning.
Three other lessons, all learned the hard way. A choice question always returns something, so it must be paired with a no-match question, otherwise you get a confident answer to a question that had no answer. Calibrated confidence is the actual product here, since a chat model telling you it is confident is not the same as a probability you can put a threshold on. And the suggestion cannot be dropped into the fixed instructions an agent keeps stable for caching, which means it has to ride on the user turn instead. Anyone copying the cookbook literally will discover that the same way we did.
One more piece of housekeeping that sounds dull and is not. Pin the exact model version once you have tuned a threshold against it. Ours came back as a versioned build rather than the generic name we asked for, and a silent upgrade under a tuned threshold is how a working guard becomes a broken one.
The rest of the weekend
Another of our machines ran the same model against a different problem, and it produced the retrieval result that opens this piece. A hundred and thirty one possible notes, the right one at 0.99, the no-match score at 0.07 on a query with no answer, and about a second for both queries together.
The other thread of the weekend was less philosophical. We installed a scraping tool from a list of open source projects, and the interesting discovery was not the tool. It was that the machine we run it on is short on memory, and that is what breaks things there, not the software. That one is on the upgrade list along with more memory.
What we would tell another small shop
Do not buy a classifier to fix a description problem. Do pay attention to the turn where the agent loads a skill that has nothing to do with the request. That is the failure customers notice, and it is the one a two question gate with an explicit no-match option removes almost entirely.
Keep the measurement small. Fifty three turns, a hand written answer key, and half a cent of API spend told us something the vendor's own benchmark could not: where their technique helps us, and where it is decoration. We would rather have that than a demo.
We write these notes as we go. This one lives here on Coffee Talk AI, the business pieces run on the blog at webexperts.com/blog, the short thoughts show up on Bluesky, and the raw working notes sit on our own screens until something earns its way out here. If you want this kind of work aimed at your business, the door is open at webexperts.com.