People are building agentic search one of three ways:
1. Actually good retireval. There’s been a lot of progress on serving the kinds of queries agents tend to serve, from places like Hornet, MoxedBread, LightOn. Particularly in late interaction
2. Smarter harnesses with models/judges validating the result. This is now just seen as the generator/ evaluator pattern. Here’s where people try to just use grep or some other naive retrieval system. Let the agent figure it out. But it’ll consume a lot of tokens to get good results as it iterates and loops.
3. A model trained for retrieval. Give it dumb retriever like in (2) but it is fine tuned on the task as in (1).
This article is 3. But we’ve been seeing this all year with SID.ai, Gleans Waldo model etc. if this interests you I’d check those out, particularly SID.
There is so much opportunity for purpose built models like this. Ideally a harness should spin up a subagent to offload to targeted models for specific tasks like this. I know this is not a novel idea. Claude code does some of this by handing off the "explore" agent work to haiku. I just love seeing that specialized LLMs are being developed.
mrinterweb
The big lab models are academically interesting but business wise they seem toast long term. There’s no way for these model companies to compete when the models are becoming a pure commodity and others offering options that are orders of magnitude cheaper.
It’s not that the big labs couldn’t theoretically just also put out 100x cheaper options but their business model requires them to generate huge revenues from higher priced tokens or they’ll implode.
cmiles8
There is a more serious question in here that's not being answered. How effective is the retrieval in finding buried needles in larger and larger haystacks. And there's a correlary question, how effective could you be in finding paired needles in that haystack where you need to hold a needle to unlock finding another needle.
aliljet
This feels like the database equivalent of "use the right data structure." We've spent two years assuming the biggest general-purpose model should do everything. It makes more sense for retrieval, reranking, reasoning, and generation to each have their own optimized model if the routing cost is negligible.
BedVibe_Studios
I have done my own testing and found that smaller models can beat their larger siblings on fact retrieval from documents. I haven’t investigated it in depth with a large enough dataset but my guess is that larger models overthink it while smaller ones just do it. I would like if they compared this with 5.6 Luna instead.
JCharante
Why do we need to train the model to solve for retrieval within the org, so we have to keep training it whenever new dataset is introduced , or am I missing something here ?
linux_devil
Nice, but there's no mention of how Luna or DSFlash perform on the same task? (Being 25x and 50x cheaper respectively.)
Nor of how much faster their custom model performs?
andai
Hmmmm, this is a problem I have been facing recently.
Basic embeddings give decent-ish results (in the top say 20 chunks). Basic agentic retrieval gives slightly better results so long as the agent part of it doesn't go down the wrong track.
I like the idea of what's discussed in the link, however atm we are on Bedrock KBs and so locked in to a very basic implementation of RAG, because Amazon doesn't have the foresight to make things flexible enough - including making it an absolute pita to use their hybrid search. But, I guess they "work" reliably.
One of our core issues centers around a 1300 page document all about the same overall topic but with minor various for specific procedures/situations. Typical embeddings waters this down so that each chunk really just represents the common theme and therefore lacks a lot of contrast.
But now that luna's (and others) price has been cut, perhaps I'll start experimenting with giving it free rein to explore the data a little in the same way that I do a web search.
One thing that definitely helped was providing a separate index of each section where I had another model summarise the primary unique topics in each section to act as a guide for the agent. I think either we should be chucking the entire doc at a model (400k tokens...so not really ideal at this time) or improving RAG accuracy. For the latter I think even with embeddings, meaning of words and semantic connections are not enough at all - attention is KV so it is 2 dimensional and once I started getting into it I've kind of realised that 2 dimensions aren't really enough to represent the logic that exists between tokens (i.e. sections of documents that refer to a sequence of actions dependent on some logic that references "variables" from another section, i.e. "if x, y has happened then refer to z sequence). There's much deeper meaning to human language than I think basic embeddings covers.
I think it's becoming clear to me that in the same way that embeddings encode the web of semantic meaning of a chunk of text, I need something similar to a hybrid of the author's model + reranker + super-embeddings that encodes as much of the entire meaning of a text as possible and not just semantic.
fennecfoxy
One thing that plagues [insert current FAANG] is the large amount of corpus knowledge that is outdated/misleading or just plain wrong. I'm curious how this addresses that if it's deriving the reward function from the corpus itself.
comments (10)
1. Actually good retireval. There’s been a lot of progress on serving the kinds of queries agents tend to serve, from places like Hornet, MoxedBread, LightOn. Particularly in late interaction
2. Smarter harnesses with models/judges validating the result. This is now just seen as the generator/ evaluator pattern. Here’s where people try to just use grep or some other naive retrieval system. Let the agent figure it out. But it’ll consume a lot of tokens to get good results as it iterates and loops.
3. A model trained for retrieval. Give it dumb retriever like in (2) but it is fine tuned on the task as in (1).
This article is 3. But we’ve been seeing this all year with SID.ai, Gleans Waldo model etc. if this interests you I’d check those out, particularly SID.
I wrote about these 3 approaches here https://softwaredoug.com/blog/2026/06/08/three-kinds-of-agen...
softwaredoug
mrinterweb
It’s not that the big labs couldn’t theoretically just also put out 100x cheaper options but their business model requires them to generate huge revenues from higher priced tokens or they’ll implode.
cmiles8
aliljet
BedVibe_Studios
JCharante
linux_devil
Nor of how much faster their custom model performs?
andai
Basic embeddings give decent-ish results (in the top say 20 chunks). Basic agentic retrieval gives slightly better results so long as the agent part of it doesn't go down the wrong track.
I like the idea of what's discussed in the link, however atm we are on Bedrock KBs and so locked in to a very basic implementation of RAG, because Amazon doesn't have the foresight to make things flexible enough - including making it an absolute pita to use their hybrid search. But, I guess they "work" reliably.
One of our core issues centers around a 1300 page document all about the same overall topic but with minor various for specific procedures/situations. Typical embeddings waters this down so that each chunk really just represents the common theme and therefore lacks a lot of contrast.
But now that luna's (and others) price has been cut, perhaps I'll start experimenting with giving it free rein to explore the data a little in the same way that I do a web search.
One thing that definitely helped was providing a separate index of each section where I had another model summarise the primary unique topics in each section to act as a guide for the agent. I think either we should be chucking the entire doc at a model (400k tokens...so not really ideal at this time) or improving RAG accuracy. For the latter I think even with embeddings, meaning of words and semantic connections are not enough at all - attention is KV so it is 2 dimensional and once I started getting into it I've kind of realised that 2 dimensions aren't really enough to represent the logic that exists between tokens (i.e. sections of documents that refer to a sequence of actions dependent on some logic that references "variables" from another section, i.e. "if x, y has happened then refer to z sequence). There's much deeper meaning to human language than I think basic embeddings covers.
I think it's becoming clear to me that in the same way that embeddings encode the web of semantic meaning of a chunk of text, I need something similar to a hybrid of the author's model + reranker + super-embeddings that encodes as much of the entire meaning of a text as possible and not just semantic.
fennecfoxy
richwater