A look back at the ideas that stuck, the ones I am still unsure about, and where I think this is heading.

Buy me coffee via ko-fi!

About a year ago I shifted my main writing home from blogger to Substack. Since then, this newsletter has grown from roughly 3,000 email subscribers to around 5,500. While the affordances of Substack have made it easier for subscribers to follow, the fact that a niche newsletter about academic search and library technology could add 2,500 readers in a year is mostly a signal about the moment we are in. Librarians, researchers, and tool builders are all unsettled by what “AI search” is doing to academic discovery.

I think there are not many places pondering in the open about what this means, trying to work it out rather than deciding they already have figured out that “AI” applies to search (or anything really) is bad.

If this newsletter has a job, that is the job. Thank you for reading, for arguing in the comments, and for forwarding posts to colleagues who hopefully subscribe.

If a year of this has taught me one thing, it is that the freeform chatbot is the wrong mental model for academic search. The fear that dominates most discussion, that these systems fabricate sources and invent findings, is real for a naked language model answering from memory or even one with weak naive RAG implementation.

But wrap a model in an agentic harness that runs deterministic checks in code, and force it to verify every claim against an independent source rather than generate it freely, and most of that risk is manageable (at least for claims that can be checked against independent sources). I was not sure of this twelve months ago, or at least I could not have said it cleanly without experience trying LLMs with agentic harnesses.

This post provides a thematic retrospective of the content on my blog and ends with some upcoming future content.

One question, many posts

There is a narrower question every post here is also trying to answer: when a vendor says “AI-powered search”, what is being claimed, and how should a librarian evaluate it without either hype or unthinking rejection?

That span of 12 months opened with a two part article testing how RAG systems handle retracted papers across Elicit, Scite, SciSpace, Primo Research Assistant, Undermind and others, and it closes with sharper “hot takes” about where librarianship is misreading the whole category. The through-line is the same: take the tools seriously enough to test them properly, and take our own assumptions seriously enough to question them.

Defining the thing before arguing about it

The post I point newcomers to first is What Do We Actually Mean by “AI-Powered Search”?. The phrase hides at least four different things: post-retrieval conveniences like summarisation that do not touch your results at all; semantic retrieval using vector embedding that goes beyond lexical matching; LLMs used in retrieval and ranking; and full synthesis and generation across papers. Your concern about one of those may not apply to another, and most of the unproductive arguments I see come from people defending or attacking different levels while using the same words.

The framework of mine that has resonated the most, though, is a bit older. Why I Think Academic Deep Research — or at Least Deep Search — Will “Win” is where my 2x2 matrix first appeared, and it has since been picked up and cited well beyond this newsletter.

It sorts search systems along two axes. The first is how the system works: a quick, one-shot retrieval, or a deep, iterative process that evaluates many candidates. The second is what it returns: Search, which only retrieves and ranks documents, or RAG, which extracts from them to generate an answer with citations. That gives four quadrants, Quick Search, Quick RAG, Deep Search and Deep RAG, and most of the products people argue about sit cleanly in one of them.

The post argues that the deep quadrants are where the future of academic search is heading and why Deep Research is mostly meant for helping to orientate yourself. Whatever you make of that call, the 2x2 itself has been the durable contribution.

If you are wondering what vector embeddings are, a natural companion piece is Why Embedding Vector Search Is Probably One of the Least Objectionable Use of AI for Search. Embedding-based semantic search dodges many, though not all, of the criticisms aimed at ChatGPT-style generative AI. It does not fabricate text or invent citations the way generation can; it only changes how documents are matched and ranked. Map it onto the four levels above and it sits well below full synthesis on the risk gradient, which is worth remembering before treating every form of "AI search" as equally suspect.

The move to vector embedding is not without direct consequences, and it is the subject of The Case of the Vanishing Hit Count, written up based on a question from my FSCI 2025 workshop. Once you leave Boolean behind for embedding and other AI-driven retrieval, the exact, countable hit set disappears, replaced by the top-k of semantic search or even the evaluated results of deep search, until even the familiar question of how many results you got no longer has a stable answer.

As I tested and used these new, modern “AI search”, I noticed a UX issue which I called initially The Blank Box Problem. We spent two decades complaining that users would not learn Boolean operators and field codes. We have now replaced all of that with a single blinking cursor in an empty box, and users are more confused than ever, caught between not knowing how to speak to the machine (keyword, natural language, or elaborate prompt) and not knowing what the machine can even do with their words. Trading visible constraints for infinite potential turns out to be a usability crisis, not a usability win.

Finally, I attempted to classify the ways LLMs summarise in tools like EBSCO and ProQuest. I distinguish between fixed single-document summaries, query-dependent single-document answers, document-by-document synthesis tables, and full multi-document RAG synthesis. The classification shows that “AI summaries” are not one thing: as tools move from pre-canned article summaries to query-driven synthesis across many papers, usefulness rises, but so do unpredictability, evaluation difficulty, and trust risks.

Deflating the hype, without dismissing the tools

While I am known to be “pro-AI”, particularly for search - I have pushed back on many over-claims and warned about issues this new technology brings.

The AI-Powered Library Search That Refused to Search is probably one of the most read posts on my Substack so far, perhaps because critical posts about AI travel further than careful ones.1.

It documents an odd new problem: content-moderation layers designed for chatbots are creeping into library discovery and quietly refusing to search certain controversial topics at all. How serious this is, I am not yet sure.

The Horseless Carriage of AI Search takes aim at an increasingly common pattern in legacy databases, the claim that they now “do AI”. It argues that bolting an LLM onto the front of a database, such as Primo Research Assistant, to generate Boolean strings is mostly of little benefit. I am not saying Boolean must die, but I am saying that dressing up the old engine is not the breakthrough it is sold as. I have since had to qualify this argument, and I come back to exactly where it breaks near the end.

Deep Research, Shallow Agency takes aim at the word agentic. Most academic deep research tools at the time, such as Asta Paper Finder, Undermind Classic and Scholar Labs, run fixed internal workflows, and they stumble the moment you hand them a literature review task that does not fit what they were built to solve. The agency is largely an illusion.

The follow-up, From Fixed Search Workflows to Agentic Academic Search: Promise, Progress, and the Invisible Menu Problem, looks at what happens as tools start handing control back to users. Undermind’s Projects feature is the worked example: it lets you steer and manage the agent rather than accept a fixed pipeline, which is real progress. But it also exposes what I call the Invisible Menu Problem. When you are managing an agent, you cannot see the full menu of what it can do, so you are left guessing at its capabilities and limits. It echoes the Blank Box Problem one level up, with the opacity shifted from the query to the agent itself.

In Are AI Tools Killing Review Articles? Two Failure Modes Suggest Otherwise, I argued that the most important kinds of review articles are still hard for AI, for two reasons: the corpus construction gap, where tools still struggle with retrieval and exhaustiveness, and the conceptualisation gap, where expert judgement is needed before retrieval even begins.

The uncomfortable mirror for librarians

A few posts are aimed squarely at us librarians.

“We’re Good at Search,” Just Not the Kind the AI Era Demands is a deliberate provocation. When I look at the evaluation rubrics for AI search now circulating, relevancy is often just one subjective category among many, scored on an “I know it when I see it” basis. For a profession that prides itself on search, that is a strange place to be.

AI Academic Search and the Missing Middle of Literature Discovery makes a related point: I argue by judging these tools mainly through undergraduate information literacy and through evidence synthesis, we underrate their strongest current use case, the ordinary narrative literature review, perhaps because our involvement here is less.

My most recent post, Stop Calling Poor Search Rankings Necessary Friction for Learning, takes aim at a comfortable rationalisation. When a system ranks badly, it is tempting to argue that the struggle is good for students, that wading through irrelevant results teaches them something. There is a legitimate version of that idea, where we set out to expose students to adjacent and boundary literature, but that is intentional design. Defending accidental noise as if it were pedagogy is not the same thing, and we should stop confusing the two.

This connects directly to an older post, The Sycophancy Fallacy. Both are the same category error: mistaking a ranking system’s output for a stance. The worry that AI search tools will agree with users the way chatbots do treats retrieval as a conversational partner that can flatter. A retrieval system is not agreeing with you, it is returning what its ranking surfaces. Confusing the two means worrying about the wrong bias and missing the real ones.

The last is a little left field, since digital literacy is not my speciality. Fact-Checking vs the Academic Inquiry Mindset asks whether we are collapsing two different habits of mind into one. Verifying a discrete claim is not the same as open-ended scholarly inquiry, and treating the first as though it were the second risks leaving students short of what research demands.

The plumbing problems nobody is watching

The least glamorous posts may matter most. The Petrol Tank for AI Discovery Might Be Running Dry flags that the comprehensive AI search engines we are starting to lean on, such as Elicit, Consensus and Undermind, depend on access to abstracts and full text that publishers now have a strong commercial incentive to close off.

Why Ghost References Still Haunt Us makes two claims: that Google Scholar's [CITATION] entries and a polluted web can undermine the verification RAG depends on, and that fabricated and ghost references often predate LLMs. I am confident about the first for reasons I give below.

Trying to explain and understand

About a year ago, I felt a sense of unease whenever I heard the term agentic search, (mostly in the context of Undermind), I had an uneasy sense that I did not really fully understand what it meant.

As I spent time thinking about it, trying MCP servers, skills, and finally LLMs with agentic harnesses like Claude Code and Codex, something clicked.

The Agentic Researcher and the MCP server posts were two of my earliest posts documenting my growing sense of understanding of what “agentic” means.

By pairing a general LLM with academic Model Context Protocol servers or other tools, they can match or beat specialised deep research tools. This may point to a real-time, federated model of access to publisher content rather than yet another giant centralised index.

Reviews & evaluation frameworks

I did some hands-on reviews of Consensus, Scholar Labs, ResearchRabbit and Google’s AI Mode as they shipped.

Librarians are always interested in evaluation frameworks, so in AI academic search needs better frameworks for understanding and evaluation I pointed to three librarian projects I found promising. That was followed by Better AI Search Rubrics: Roles, Gates, and Retrieval Tests, an opinionated piece on the common weaknesses in such rubrics and ways to improve them.

Other posts of interest beyond the twelve-month window

I did not suddenly start writing about AI search when I moved here. A few from outside this window that I still consider significant, now cloned onto Substack from my Blogger platform:

• The reproducibility and interpretability of academic Ai Search engines like Primo Research Assistant, Web of Science Research Assistant, Scopus AI and more

• Testing AI Academic Search Engines (1): Defining the tools & Testing AI Academic Search Engines - What to find out and how to test (2)

• Comparative review on Primo Research Assistant, Scopus AI, Web of Science Research Assistant and an explainer for AI search for librarians

• When is a Hallucination Not a Hallucination? The Role of Implicit Knowledge in RAG

• The Rise of Agent-Based Deep Research: Exploring OpenAI’s Deep Research, Gemini Deep Research, Perplexity Deep Research, Ai2 ScholarQA, STORM, and More in 2025

• Why use of new AI enhanced tools that help with literature review should be discouraged for undergraduates

• Primo Research Assistant launches- a first look and some things you should know

What I am unsure about, and what comes next

In keeping with how I try to write, the honest caveat: a lot of this is provisional, and some of it will age badly. The landscape shifts month to month, and several posts above are me thinking out loud rather than concluding.

Two examples.

The first is the ghost references post. I stand by the weakness it identifies in Google Scholar, but I am no longer confident in its conclusion that in the example I investigated the first version of the ghost reference I traced was human-made rather than produced by an LLM. Treat that part as an open question rather than a finding.

Where my “horseless carriage” critique breaks

The second I have to own more directly. I was proud of The Horseless Carriage of AI Search, my critique of legacy vendors taking the easy route to “implement AI” by using an LLM only to generate Boolean.

After using Primo’s MCP server I have had to qualify it.

The critique was aimed at one specific move: using an LLM as a translation layer that feeds Boolean into the same lexical engine over the same document corpus. I still think that is mostly low-value.

What I missed is that database and dataset selection is a different task from normal article search.

There the target is a small, description-only space, and the model earns its keep two ways. It normalises vocabulary, and it draws on pretraining knowledge to go straight to the right resource. That is not dressing up the old engine. It is doing something the old engine could not. The critique holds for article retrieval and does not generalise to query-to-resource matching.

A concrete example helps clarify.

Experienced librarians know that users struggle to find the right database for datasets, because their query terms have to match a very small target space, the database description. A wrong term returns zero hits: search for cost of living datasets/databases rather than CPI and you find nothing.

A capable LLM sidesteps this, because it knows to search for the right terms, CPI, inflation , and so on.

Claude 4.8 is similarly smart, reasoning (after searching) that most database records will not have “cost of living” in their metadata and switching to searching Economic databases.

But what if the database description just doesn’t have the right terms? Here is where LLMs can also help by “cheating” because they already know the answer (from pretraining knowledge).

In this query, I asked for datasets with the right terms and it seems to do as expected, surfacing the obvious databases.

But then I noticed the searches it did

The first search was expected but the second was odd - a known item search of the dataset ExecuComp! In short, a large LLM is capable of knowing the right answer in advance and just search to confirm we had access!

Below shows yet another similar example, where it searched directly for Audit Analytics (a known item search of a dataset in WRDS) when asked about datasets on audit reports.

What comes next is going to double down on understanding agents. At FORCE2026 I presented a small study of the reproducibility of the agentic search tools Undermind and Consensus with the title - Does Agentic Deep Search Converge? Reproducibility Questions for LLM-Driven Literature Discovery, and the results were “interestingly wrong” in ways worth writing up here.

An agentic skill for building and testing high sensitivity Boolean Search Strategies for Pubmed.

The bigger piece is one I have put off for too long, out of impostor syndrome and a sense of how evidence synthesis librarians might react (exacting high standards for obvious reasons): a write-up of a Claude and Codex skill I built for high-sensitivity Boolean searching in PubMed.

• It is designed to heavily do tool use, using PubMed searching and MeSH browsing to check for promising terms, test pilot search strategies etc.

• Instructions also include very specific domain knowledge on how to go about conducting the process based on my (somewhat limited) knowledge of best practices.

• Comprehensive audit logging, so you can see at every step what was done and the decision made

• Hooks that use code to enforce important steps and checks

• Maybe 50+ hours of testing against gold standard sets to test for edge cases.

It is the most concrete demonstration I have of the argument I opened with. Once you wrap the model in a harness that can run deterministic checks in code and make it verify its claims against independent sources rather than generate them, much of what people fear about fabrication stops being inevitable.

The PubMed skill is one small instance of that pattern, and I think it generalises. Descriptive and subject cataloguing is another domain where a similar method could be applied.

While I don’t think my home-brew skill currently exceeds the best experts, it has reached the point where it can reproduce or improve on many of the moves I would make myself, while logging and checking them more consistently (which is why I can’t improve it further easily). I suspect a team consisting of expert evidence synthesis experts and professional coders will be able to push it far further than what I did.

Buy me coffee via ko-fi!

Conclusion

Agentic skills scare me because they look like a second attempt at expert systems, but with a crucial difference. The old systems tried to hard-code expertise. The new ones combine flexible language-model reasoning with tools, logs, tests, and code-enforced constraints. Maybe even that isn’t enough, but this “neurosymbolic AI” approach does make them much more plausible as systems for operationalising professional judgement with all the obvious implications2.

Yes, I could be totally wrong. Even then, I would rather name what could be better and be wrong in public than pretend something is settled. If a take here does not hold up in a year, I would sooner hear about it than quietly forget I wrote it.

That goes especially for the readers who arrived through my more critical posts, and who I suspect sit closer to the resist-AI end of things. I hope some of you stayed once it was clear that I am, on the whole, positive about AI for search, and I would rather argue it out with you than preach to people who already agree.

Thank you, then, for a year of reading and pushing back. The work is better for it. Here is to the next set of posts, and to being usefully wrong together.

This post is part of a “hot takes” series in which I make sharper claims than I usually do. I do not intend to offend, and I am not trying to tar every librarian with the same brush — the patterns I describe and perceive may be a function of my own local context. In my last hot takes post, I argued that quality AI search that gives better rankings might not be fit for purpose for a librarian teaching undergraduate information literacy because “friction is pedagogically useful and one could even argue better search results removes friction!”. Here I examine this argument more carefully.

Buy me coffee via ko-fi!

Months ago I read a review of an AI search tool that spent most of its length complaining about how poorly the tool ranked results, then made a striking turn at the end: poor relevancy ranking was supposedly a virtue, because learning needs friction.

My first reaction was that this confused good friction (the cognitive effort required to master a topic’s boundaries) with bad friction (the time wasted fighting unintutive operators, lexical noise and uninformative junk)1. But that is too easy a dismissal. Versions of the argument keep resurfacing, and underneath the crude version is a harder question that does not collapse so quickly.

Before diving in, some quick framing notes.

First, I am talking about ordinary narrative reviews and information-literacy contexts; systematic reviews have different requirements and are set aside here.

Second, if you think no AI search system currently gives materially better results than a traditional BM25 discovery layer, bracket that for now. Assume a genuinely better-ranking system and ask what follows.

Lastly, I am holding the interface constant. Much of the current anxiety about AI search is really anxiety about RAG-style answer engines and/or deep research tools: systems where the user types a question, gets back a synthesised response, and may never seriously engage with the underlying documents.

Ethan Mollick has recently documented real instances of this kind of cognitive surrender. That is a legitimate concern that deserves discussion, but in this essay I focus just on better retrieval and ranking. If better ranking alone is supposed to damage learning, we should be able to say exactly how.

The weak arguments

Before getting to the harder question, three things need to be cleared away.

The first is a reflex that often appears when new tools arrive - making it easier to do things that once was hard and required expertise. Many experts, may be tempted by the idea that struggle is inherently virtuous, and that making something easier is a bad idea. Perhaps some experts might even have an unconscious vested interest in defending their hard-won expertise, grasping at the friction argument as a justification2.

The second is a concern often raised alongside the friction argument: that LLM- and embedding-based systems encode biases that could be amplified at scale if every novice uses the same AI-ranked tool to map a field. This is real, but it is a different question. Leaving aside that there are a huge variety of AI search engines using very different techniques, traditional lexical systems are not neutral either, and people worried about the dominance of Google Scholar amplifying bias. The homogenisation worry is better understood as an argument for epistemic diversity, interface pluralism, and transparency about retrieval choices, rather than for inferior ranking3.

The third is the crude claim that poor ranking is itself mostly or even always educationally valuable. It is not, and one test disposes of it. If poor ranking were valuable as such, we ought in principle to be willing to worsen it deliberately: degrade the scoring function, shuffle the top results, inject noise. At the extreme, results should be totally random! Nobody wants that.

But no reasonable person believes that? What about a much weaker claim, such that you believe the friction from poor relevancy ranking is “sometimes valuable”? This is what the rest of the post takes up. My belief is, the educational value of poor relevancy, if any, does not lie in worse ranking itself. Rather, it is an effect that weak ranking sometimes produces by accident4.

The question worth asking

What survives, once those are set aside, is a harder question. Suppose we had a search system that ranked documents perfectly by topical relevance, by every standard IR measure: precision, recall, nDCG. Would it also be the best system for someone learning a topic5? If no, then in what way, and why?

I have always thought the answer was clearly yes, but the more I thought about it, the more I felt this question is a interesting one; the rest of this post is an attempt to answer it. I have discussed this with librarians who hold the view that a very good, or even perfect, retrieval and ranking system would be pedagogically detrimental, and this is my best attempt to steelman their argument.

The short version: great topical ranking and pedagogical value can and sometimes do diverge in particular ways, but the reasons matter. They do not automatically justify poor ranking or vindicate the friction case in a straightforward way. Instead, they argue for designing systems and pedagogy around the things straight-forward good ranking alone cannot deliver6.

The different types of irrelevance

Let me suggest that not all irrelevant results are equal.

Some irrelevance in a result list is informative. An “irrelevant” result that appears because it belongs to an adjacent subfield, uses overlapping vocabulary, or reflects a nearby conceptual frame may genuinely help a learner understand where a topic begins and ends.

Other irrelevance is uninformative. A hit that appears because of a homonym, a stopword quirk, or a lexical accident teaches the novice nothing about the structure of the field. It teaches, at most, that search language can be messy7.

Poor ranking does not discriminate between these. It produces a messy mixture of both, especially for novices, who are least equipped to tell the difference. So the answer to the harder question takes a particular shape: when topical ranking and pedagogical value come apart, it is because better ranking can remove informative irrelevance along with the uninformative kind. If that is right, the response is not worse ranking. It is intentional design.

Two careful versions of the objection are worth examining through this lens.

Version one: seeing search terms in context

The first argument is that students benefit from seeing the terms from their assignment prompt used in actual scholarly contexts, and from gradually learning the vocabulary of a field through retrieval itself.

That is plausible, and it is also not undermined by better ranking. If anything, accurate ranking increases the chance that students encounter their search terms in genuinely relevant documents rather than tangential ones. The vocabulary they see on a precise hit is more likely to belong to the field than the vocabulary they would have encountered on a near-miss caused by a lexical accident.

This version survives only if novices learn vocabulary better from a noisier signal than a cleaner one. Pedagogically, I don’t see why this is the case.

You might object: if modern search engines use semantic matching with dense embeddings, how would they perform and show the keyword-style matching in results page seen with conventional keyword engines (e.g. Google Scholar)?

Interestingly, some newer retrieval models can do something akin to, if not better than, old-style keyword highlighting. Late-interaction and multi-vector systems such as ColBERT, or learnt sparse embedding methods like SPLADE, can expose which terms or passages contributed most to the relevance score, including semantic neighbours in addition to exact lexical matches. In principle, that could support vocabulary growth better than traditional keyword-only ranking, because it can surface terminology matches the student would not have thought to search for.

Below shows how a ColBERT type search system, highlights terms (both based on exact lexical matches and semantical matches) in the document that contributes the most to the relevancy score.

Version two: boundary learning for novices

The stronger version is about something more specific than vocabulary or effort. It is about the novice’s need to learn a topic’s boundaries through exposure to meaningful near-misses and adjacent literatures.

A novice on a new topic cannot simply search broader, because they do not yet know what broader would mean. They do not know which adjacent areas matter, which terms are near-synonyms, which distinctions are important. For that learner, boundary formation has to happen partly through retrieval itself. Some of what matters may be found not in the core results but in the edge cases.

This is a fair point, and it is where the informative/uninformative distinction earns its keep.

If the case for “useful friction” rests on accidental exposure to meaningful adjacency, then the relevant question is whether poor ranking is a good mechanism for producing it. It is not. Poor ranking by definition generates adjacency mixed indiscriminately with junk. There is no reason to prefer accidental adjacency mixed with lexical noise over a system that retrieves the core accurately and exposes adjacent material deliberately.

A retrieval system that can reliably retrieve works that are not just in your topic but adjacent topics opens a wealth of possibilities to help the user learn!

An example of what can be done is Undermind’s adjacent papers feature: a dedicated function that displays papers citing the seminal works of the topic you are researching, despite being judged as falling outside the core topic itself8.

Consider the idea of serendipity. When libraries moved physical stacks to remote storage, users complained about losing serendipitous browsing, and they were right that something real was being lost. But the adjacency they valued was not accidental. It came from classification: LC and Dewey were designed to place related works near each other. The well-handled response was not to defend physical stacks at any cost but to build virtual shelf browsing, browse-by-call-number, and related-item interfaces that replicated the function digitally. The function was real and worth preserving; the mechanism changed. The same logic applies here. Adjacency exposure for boundary learning is real and worth preserving. The mechanism should be intentional design (adjacent papers, topic maps, related-citation views), not the accidental side-effects of worse ranking.

This kind of result helps a novice see the edges of a topic and how neighbouring literatures branch from a common core. Topic maps, related-citation views, contrastive recommendations, and “nearby but different” clusters can do similar work. Don’t look now, but all this needs better retrieval and ranking not worse!

The table above shows “adjacent works” identified by Undermind that cite the foundational papers of your topic (In this case - does making papers open access lead to higher citations), but Undermind has judged to be not on your topic. You can see that while it isn’t perfect, it does show you “Adjacent” papers that focus on impact on Open Science, impact of open access on societal impact, influence of funding on open access citation advantage etc. In short, it does some of the “boundary work”.

While I know some users don’t understand why this feature exists in Undermind as a email alert because they are hyper-focused on their topic, I find many of the papers surfaced this way interesting.

As our retrieval and ranking systems get better at surfacing adjacency clusters, exploring and clustering boundaries, we should go beyond simple Chat-style replies with top-five-card summaries or even conventional list style rankings of results9. What we need are innovative, dedicated features that assist the searcher in learning about the field they are exploring.

Buy me coffee via ko-fi!

Conclusion

The crude claim collapses on inspection. Poor relevancy ranking that is accidental is not a pedagogical good in itself.

Topical ranking and pedagogical value can perhaps come apart, particularly for novices who need exposure to topic boundaries, adjacent vocabularies, and the structure of a field. Retrieval is not just a delivery mechanism for known items. It can be part of how a learner forms the map.

But the conclusion that follows is not a defence of worse ranking. It is that we should stop treating poor ranking as a substitute for thoughtful design. If boundary learning matters, expose adjacent concepts deliberately. If vocabulary formation matters, make semantic neighbours visible. If we care about epistemic diversity, promote multiple tools and multiple ways of traversing a literature.

Poor ranking is a bad proxy for all of that. The honest response is to design for productive friction explicitly, not to hope mediocre retrieval generates it by accident or use pedagogy reasons as a poor excuse for poor relevancy!

This post is part of a “hot takes” series in which I make sharper claims than I usually do. I do not intend to offend, and I am not trying to tar every librarian with the same brush — the patterns I describe and perceive may be a function of my own local context. But I am sure some of what I write will resonate. Besides, I think librarianship advances by naming what could be better, not by pretending everything is fine.

TLDR: AI search tools may look unimpressive when judged mainly through the two most visible library search frameworks: undergraduate information literacy and evidence synthesis. But their strongest current use case may be a third one: ordinary narrative literature reviews also often called Scholarly Discovery, where librarianship’s voice is less visible than it should be.

Buy me coffee via ko-fi!

I keep running into librarians, in conversation and online, who say they have tried AI search tools and come away unimpressed. They do not really see what the fuss is about.

Some of this reaction is fair. The category “AI search” is broad, and a fair amount of what gets sold under that label is underwhelming.

At its weakest it runs thin LLM wrappers that translate a natural-language query into a Boolean string and then run it over the same lexical search engine underneath. I have previously called this the horseless carriage of AI search: using new technology to reproduce the shape of the old system rather than rethinking the system itself.

Without radically rethinking retrieval and ranking (e.g. Semantic search via dense embeddings, iterative agentic search etc), the ceiling on improvement will be low.

Several library vendors now ship tools of this kind — Scopus AI, Web of Science Research Assistant, Primo Research Assistant, EBSCOhost Natural Language Search, and others. They use a powerful new technology to reproduce an old artefact, without addressing the ranking-stage problems that make academic search frustrating in the first place1. If a librarian’s exposure to “AI search” is mostly tools of this sort, writing off the category is understandable.

But the reaction I want to focus on is different. It is the dismissal that extends even to the better AI search tools: Undermind, Elicit, Asta Paper Finder, Consensus, and others that are not simply trying to make Boolean search friendlier but are attempting more radical changes to retrieval and ranking.

I will keep talking about “better AI search tools” throughout this essay and you might be wondering what I mean, since there are many types of Academic AI search with different “AI” techniques and architectures used to accomplish different functions.

Clearly, systems ONLY based on horseless carriage AI search techniques are unlikely to do much better but can we say anything more positive here?

While there are no guarantees, I would suggest that academic AI search tools that fall into the “Deep Search” and “Deep Research” type of search tools, should more often than not, give clearly superior retrieval and ranking for “difficult queries” compared to traditional Boolean retrieval but basic TF-IDF/BM25 ranking. But if you need one tool to point at that I consider a “better AI seach tool” - I would consider Undermind.ai as an example2.

In my own testing particularly for difficult queries, the difference is often visible in the first screen of results: the better AI search tools place relevant work high in the ranking, while conventional databases often require more query reformulation and screening before comparable papers appear.

For example, I recently ran a study on reproducibility, running the same query five times across multiple AI search engines. It was not even meant to compare AI search engines relevancy but even with a broad query that had 200+ possible relevant results, only Undermind maintained high precision down to rank 50.

Other respected AI search tools could barely manage this for the top 10 to 20, showing the variance in performance. For “hard queries” the difference was even more stark between AI search tools and conventional databases.

The diagram above is just illustrative

I would not be as confident in this if it were only my own assessment, but feedback from PhD students and faculty has been overwhelmingly positive with continued heavy sustained use of Undermind.ai3. Comments like “if you do not renew the subscription, I will pay for it myself” are common.

To be clear, we are not talking about Cochrane-grade recall here, and these tools do not exhaust the literature. Their strength is not formal exhaustiveness, but top-N relevance: for focused topics, they often put more useful papers into the first 10, 20, or 50 results than conventional database searches.

But this post isn’t just about Undermind.ai, there are quite a few in my view (pretty much any type of Deep Search or Deep Research) that are almost as good.

So the question is: given that the better AI search tools often deliver much stronger early precision and better practical discovery for focused scholarly questions, especially when judged by the relevance of the top 10 to 50 results., and many researchers plainly value this, why do some librarians still not see it?

Leaving aside the usual anti-AI reasons and focusing on the effectiveness angle, my best guess is that librarians who teach or support search often evaluate AI search tools through one of two existing professional lenses: undergraduate information literacy and evidence synthesis. Neither lens is well-suited to what the better AI search tools currently do best. The use case where these tools genuinely shine, the ordinary narrative review, sits in a third space that the profession has generally paid less attention to.

Lens 1: Information literacy for undergraduates

The first lens is the information literacy(IL) lens, often focused on undergraduate learning.

Information literacy specialists will rightly remind me that IL for undergraduates is not the reductive “teach students to find five peer-reviewed papers for an assignment.” At its best, IL teaching helps students understand how information systems work, how authority is constructed, how disciplinary knowledge is produced, how databases and search engines shape what becomes visible, and how inquiry develops over time. Searching is not just a mechanical act of retrieval (Searching as Strategic Exploration!). It is part of learning how knowledge is organised, contested, and evaluated.

That said, the stereotype exists for a reason. A large amount of undergraduate IL instruction is still tied to assignment support: helping students find a handful of credible scholarly sources for essays. The librarian may want to teach richer concepts, but the immediate student need is often narrow.

This is exactly the use case where higher precision and recall do not matter much. A student writing a 2,000-word essay does not need exhaustive recall. They need enough credible sources of a common topic (of which there are many) to support a basic argument, and almost any decent academic database can supply that. JSTOR, ProQuest, EBSCOhost, Scopus, Google Scholar, or the library discovery layer will usually produce something usable. Better ranking helps, but rarely transforms the outcome.

In short, if the dominant teaching use case is helping undergraduates find “some good enough sources,” then the value of a tool that produces a much stronger top-10 or top-20 ranked list may not be obvious. The existing tools already clear the bar.

The pedagogical argument

There’s a deeper issue.

If the goal is simply to help students obtain relevant sources quickly, then better ranking is an obvious win. But if, as a information literacy librarian your goal is to help students understand the process of searching, e.g how keywords fail, how databases differ, how controlled vocabularies work4, then an AI tool that quietly produces a good ranked list with no effort from the searcher is arguably even worse for this purpose!

From that perspective, poor relevance ranking is not really a bug to be designed away. In a sense it is closer to a feature, because it creates the conditions in which these skills become necessary and therefore teachable. A student whose first search returns exactly what they want does not learn anything transferable about searching. A tool that does the work for the user implicitly devalues the skills the librarian is trying to develop, and so its strengths might register as threats rather than wins.

After all friction is pedagogically useful and one could even argue better search results removes friction!

At this point, we can consider if all friction is useful for learning and whether the reduced friction from better search ranking is “useful friction”. My next post will address this interesting topic.

This orientation may also help explain a pattern in some librarian-led AI search evaluation frameworks often designed by IL librarians. Retrieval effectiveness tends to be folded into a long list of weighted criteria alongside ease of use, transparency, accessibility, currency, and ethics. As I have argued in a recent post on AI search rubrics, retrieval should instead be treated as a gating criterion: if a tool cannot find the relevant literature, the other criteria do not rescue it, because surfacing relevant literature is the thing the tool exists to do.

Lens 2 : Evidence Synthesis

The second lens is evidence synthesis: systematic reviews, scoping reviews, and meta-analyses. From this vantage point, the gold standard is near-exhaustive recall with documented and reproducible search strategies. Most AI search tools, even the best ones, do not deliver this by design5.

Why not just ask an LLM for a Boolean search strategy? Plenty of literature since 2023 shows that prompting LLMs to generate Boolean strings produces poor results. Fine-tuning helps moderately. My current view is that a better way to approach acceptable quality is to give an agentic LLM access to PubMed and the MeSH browser and let it pilot-test the search the way a human searcher does. You can see my early attempt at building a Claude skill along these lines.

Beyond that, LLM-driven search is not deterministic, and you cannot publish a systematic review whose search step is “I asked Undermind.” From this lens, AI search is at best a supplement and at worst a distraction.

There are of course specialised AI search tools designed specifically for evidence synthesis. On one hand are those built on pre-LLM machine learning techniques, such as ASReview and Covidence, which rely mostly on active learning and supervised classifiers for screening prioritisation.

On the other hand, a growing number of LLM-native entrants have appeared, including those highlighted in Cochrane's recent announcement of selected AI tools for its innovative platform study. The boundary is blurring, however. Established players like Covidence, Rayyan, and DistillerSR have been retrofitting LLM features onto what were originally classical ML workflows.

These newer LLM-native tools are very new and largely untested. Within the systematic review community, established tools like Rayyan and Covidence are well known, but the recent LLM-first entrants remain unfamiliar even to most specialists. Elicit is perhaps the most mainstream LLM-native tool to enter this space, and recently claimed support for "PRISMA 2020 guidelines, making it reproducible, traceable, and auditable at every step"6.

But notice what Elicit and similar tools actually do. Even they apply LLMs mostly to screening, deduplication, data extraction, and maybe risk-of-bias assessment, while leaving the actual search step to standard Boolean queries on PubMed. That tells us something important: high-recall search is still a hard problem for modern retrieval techniques7.

This is not irrational. For evidence synthesis, high recall is non-negotiable. If your professional frame is systematic review support, then a tool with excellent precision but uncertain recall will naturally look insufficient.

So if these two lenses are all you have, AI search tools look unimpressive twice over. They are unnecessary, or even harmful, for the basic undergraduate case. They are not good enough for the systematic review case.

A view from outside the profession

A recent video by John Frechette, the CEO of Moara is worth watching here, partly because it reaches conclusions similar to mine but from an outsider’s vantage point. He makes two observations, both of which I think are largely correct, and the second of which I expect more librarians to push back on.

You might suspect bias as he runs a commercial AI research platform. But Moara is not mainly a search engine. It is closer to a next generation reference manager or an AI-infused evidence synthesis workspace, somewhat like Covidence with more aggressive LLM use, helping researchers collect, screen, and synthesise sources from anywhere: Google Scholar, Undermind, Claude, conventional databases, PDFs, reference managers. In that sense, Moara is source-agnostic, and John has little incentive to favour one search tool over any (he tends to praise Undermind and using Claude search outright in his demos).

First, he criticises the standard framing from some libguides of literature search as “pick a database, run your search, you are done.” He is right that this is misleading. Top-10 and top-20 overlap across major academic search engines is much smaller than this framing implies, and evidence synthesis methodology has long acknowledged that no single source has comprehensive coverage of any non-trivial broad topic. Though in fairness, many libguides are aimed at undergraduates who are not doing the level of formal review John has in mind.

Second, and more controversially, he notes that many university-recommended discipline-specific databases produced worse results than Google Scholar or Undermind, and that even IDEAS, despite his topic being in economics, returned irrelevant results past the first ten hits. Some librarians will take offence at this. I will say it plainly: I think he is right.

For many difficult, exploratory, or poorly lexicalised scholarly queries, the top-ranked results in conventional academic databases can be noticeably weaker than those from Google Scholar or newer deep-search tools like Undermind, Asta Paper Finder, or Consensus. This is not a knock on the metadata quality of those discipline-specific databases, which is often excellent. But ranking is a separate problem from coverage, and ranking is increasingly where the bottleneck sits. Discipline-specific databases optimised for Boolean retrieval over rich metadata were not built around powerful semantic or agentic/iterative pipelines modern AI search uses, and it shows.

To his credit, John does not argue you should ditch every database for one single AI search, no matter how good. He explicitly says one source is never enough. The point is not “AI search wins, traditional databases lose.” The point is that ranking quality varies, and a tool that reliably surfaces relevant work in its top 10/20/50 is doing something traditional databases often fail at.

The missing middle: narrative reviews

What both the information literacy and evidence synthesis lenses can miss is probably the most common literature search scenario in academia: the ordinary scholarly literature review, often known as scholarly discovery.

Most academics writing the literature review section of a paper, a thesis, a grant application, or a book chapter are not doing systematic reviews.

Don’t get me wrong, they still need to demonstrate command of the relevant literature, identify the key debates and findings, and situate their own contribution. They want practical coverage and recall: enough confidence that they have not missed the main debates, methods, authors, and findings, combined with high precision because their time is finite and reading irrelevant papers is expensive.

This is exactly the niche where the better AI search tools deliver the most value. Undermind, Asta Paper Finder, and Consensus (either alone or more likely in combination with other tools) in my testing tend to produce a tightly relevant result set with strong precision in the top ranks, at the cost of not exhausting the long tail. For a narrative review, that tradeoff is excellent. For a systematic review, it is insufficient on its own. For a freshman essay, it is overkill.

When I look at which researchers most enthusiastically adopt these tools, it is overwhelmingly people doing narrative reviews. That is not a coincidence.

The missing or less visible third camp

I am not saying no librarian cares about this use case (I exist!). The line between supporting “undergraduate find 5 peer review articles” and supporting researchers on capstone or thesis work is thin, and plenty of academic librarians do both. There is a long history of domain-informed, researcher-facing discovery support: the subject specialist (or Reference librarian or Digital Scholarship librarian) helping a researcher navigate a field, identify important work, trace debates, follow citations, recognise key authors, and avoid obvious blind spots will definitely benefit from a strong AI search tool along these lines.

But my thesis is that in the librarian profession today, search support has become most publicly legible and focused into two areas8. On one side is undergraduate IL. On the other is evidence synthesis, especially in health sciences. Between them sits the ordinary working researcher: the faculty member, PhD student, postdoc, policy researcher, or grant writer who is not doing a systematic review, but still needs better literature discovery than a basic database search provides.

If librarians do not make themselves visible in that space, researchers will not wait (they already aren’t). They will use Undermind, Elicit, Consensus, Asta Paper Finder9, Google Scholar, Claude, ChatGPT, Semantic Scholar, Research Rabbit, Connected Papers, citation alerts, PDFs from colleagues, and whatever else works. They will build their own messy discovery workflows, with or without us.

A renewed third camp would not treat AI search as magic. People in this camp would also not dismiss it because it fails systematic review standards or complicates undergraduate pedagogy. They would evaluate these tools the way subject specialists have always evaluated search tools: by asking whether they can surface important, relevant, and useful literature for real research questions. That camp would be willing to say:

• An AI search tool can be imperfect and still useful.

• A tool can be unsuitable for systematic reviews and still valuable for researchers.

• A tool can lack full transparency and still deserve attention if its retrieval performance is strong.

• A tool should not be recommended just because it is easy to use, ethical, accessible, or institutionally licensed if it has sub-par retrieval ranking

• A tool should first be judged by whether it finds relevant literature for the task at hand. Only after that should we ask the other questions.

This is not an argument for ignoring privacy, ethics, accessibility, transparency, or sustainability. Those matter. But they should not be used to avoid the retrieval question. If a search tool does not search well, it has failed at its primary task.

The deeper point is not that librarianship has never occupied this middle territory. It has. The point is that this role needs to become more visible again in the age of AI search. The third camp does not need to be invented from scratch. It needs to be recovered, updated, and have a larger voice at the table.

Buy me coffee via ko-fi!

Conclusion

To summarise the use cases:

• Freshman undergraduate work: mid-to-high precision, recall does not need to be high. Almost any database with the right coverage will do.

• Evidence synthesis: high recall is non-negotiable, and you accept whatever precision you can get. Multiple sources, documented strategies, traditional databases plus maybe AI tools as supplements.

• Narrative reviews: moderate recall for the time spent, high early precision, and strong top-ranked relevance. This is where the better AI search tools sit, and it is the use case where librarian voices are thinnest.

The smarter evidence synthesis librarians have already figured out that there is nothing preventing them from using Undermind as one source among many in a multi-source strategy, and I have seen several publicly endorse this approach. That is exactly right. But the deeper move the profession still needs to make is to recognise that the narrative review may be a use case underserved by the loudest professional voices….

Buy me coffee via ko-fi!

Whenever I give workshops or talks on AI search tools, someone eventually asks whether I have or recommend an evaluation framework or matrix for AI search tools.

I have been hesitant to give a firm answer. I know a thing or two about AI search, but I have much less experience, and frankly less interest, in building formal evaluation frameworks. Still, evaluation frameworks are a serious thread in information literacy, collection assessment, and procurement. The demand for them is real.

My worry is that many evaluation matrices try to do too much at once. They aim to be universal. They give every criterion a score. They weight too many things equally. They also rely heavily on qualitative impressions, including for the thing that matters most when evaluating search tools: whether the tool can retrieve and rank useful material.

To put my argument more clearly: AI search evaluation frameworks should not produce a single universal score at least not for everybody. They should specify the user role and task, identify non-negotiable gates, and test the most important claims empirically. For AI search tools, retrieval quality should be a core task-performance gate. If a tool cannot reliably retrieve and rank useful material for realistic queries, its interface, citation export, and administrative features are secondary.

That does not mean retrieval is the only non-negotiable gate. For institutional procurement, there may be other hard gates: privacy, accessibility, licence terms, data retention, security, local compliance requirements, and so on. A tool that fails those may be unacceptable regardless of how good its retrieval is. My point is narrower. Once we are evaluating something as a search tool, retrieval should not be treated as just one ordinary category among many.

In my last post, I mentioned two librarian-led projects that implemented AI search evaluation frameworks, both coincidentally vibe-coded using Claude and both drawing at least partly on my work. One was by Wang Huajin and the team at Carnegie Mellon, and the other by Alfred Wallace at the University of North Dakota.

Both are good. More than good, actually. They show a relatively sophisticated understanding of how AI search tools differ. For example, they distinguish between tools that merely use an LLM to generate Boolean queries, tools that add reranking, tools that combine lexical and semantic retrieval, and tools that orchestrate multi-step agentic searches. That is already a step above treating “AI search” as a single homogeneous category.

Still, no framework is perfect. In this post, I want to suggest three improvements that could make AI search evaluation frameworks, including other existing ones like the REACT (Relevancy, Ease of Use, Assessing DEIA, Currency, Transparency & Accuracy) framework1, more useful for libraries:

1. Be explicit about whose needs the framework serves.

2. Treat retrieval capability as a gating criterion, not just one category among many.

3. Replace some qualitative judgements with lightweight empirical tests.

None of these suggestions are novel in isolation. But I think putting them together would make these frameworks more practical and more honest.

Share

1. Evaluation matrices should be clear about who they are for

One of the hardest issues in designing an evaluation matrix is deciding whose needs matter most.

An undergraduate does not need the same things from an AI search tool as a faculty researcher. An instructor does not necessarily care about the same things as an evidence synthesis librarian. A graduate student doing exploratory research has different priorities from a librarian supporting a systematic review.

For example, reproducibility and interpretability are crucial for evidence synthesis. If you are supporting a systematic review, you need to know what was searched, how it was searched, and whether the same search can be repeated. But for an undergraduate writing an essay on a fairly well-trodden topic, reproducibility may matter much less.

Similarly, an undergraduate may value ease of use, clear explanations, and help finding a few credible sources. A researcher working in an emerging area, where terminology has not yet stabilised, may care much more about retrieval depth and the ability to surface relevant papers that keyword search would miss.

Many evaluation frameworks seem to assume that there can be one universal matrix for AI search tools that apply for every task or user. They assign points to many criteria and then produce an overall score. I think this is usually a mistake.

Without a clearly stated viewpoint, the best case is that the framework becomes too diffuse to be useful. The worst case is that the designer unintentionally overly encodes their own priorities into the scoring system.

For example, a librarian might give substantial weight to features such as adminstrative features like COUNTER statistics support, authentication options, vendor support speed because those are areas they know and need the most. Those matter for procurement and management. But they do not matter much to the actual user trying to find relevant literature.

This is why I like Alfred Wallace’s Evaluating AI Tools for Research matrix. It allows the evaluator to choose a role, such as faculty researcher, graduate student, undergraduate, instructor, or student. The selected role then changes which criteria are treated as priorities or key questions.

I played around and vibe coded my own AI search evaluation framework based on AI-Powered Tool Assessment Framework and AI Rubric - Evaluating AI Tools for Research.

It is not very good currently and needs much more work, but I can show you a screencap below of roughly what I mean.

Figure: My draft role-based framework, vibe-coded based on AI-Powered Tool Assessment Framework and AI Rubric: Evaluating AI Tools for Research. This shows how different roles have different weightings for the same criteria.

For simplicity, I only define three roles: undergraduate, researcher, and librarian. Each criterion falls into one of three levels depending on the role: critical, key, or important. The defaults can be changed.

In my current system, answers to most criteria fall into four bands, with values from 0 to 3. These are weighted more heavily if the criterion is key rather than important. Critical criteria are handled differently as “gates”, which I will discuss below.

Figure: For this criterion, it is defined as a key question and weighted more heavily, so this tool scores 2 x 10 = 20 points on this criteria

We can quibble about the exact roles and weights. The principle is what matters. A framework should make clear whose interests it is optimising for.

A simplified version of what each role might prioritize might look something like this:

This matters because it helps libraries justify decisions more clearly for which group of users the tools are for. Instead of saying, “For all users, Tool A scored 82 and Tool B scored 76,” we can say something more meaningful: “Tool A is better for undergraduate discovery (scoring 82), while Tool B is more suitable for advanced research or evidence synthesis (scoring 92) taking into account what each cares about”

2. Think Non-negotiable Gates not just composite scores

Many evaluation frameworks are extremely comprehensive. They try to account for everything: retrieval, usability, accessibility, privacy, sustainability, source coverage, citation handling, export options, transparency, administration features, and more.

They assign weights to each criteria (typically evenly without much thought), expect evaluators to score each area and then sum up the scores to get a final score.

But this can result in you missing the big picture. If your mandatory requirement is accessibility for legal reasons, or if you are an evidence synthesis researcher and cannot use tools that have results are not reproducible, you should use these criteria as “gates” to test first.

By a “gate”, I mean a criterion that a tool must pass before we continue evaluating it. If the tool fails that criterion, we stop. It does not matter how well it performs on secondary features.

Your “gates” in the context of AI search, could be environmental issues, copyright issues or impact on learning and of course performance issues. So for example, if your non-negotiable is that the AI search tool should not impact learning by generating direct answers, you might opt for the “gate” of tools that only give listing of results not answers aka Deep Search or Quick search tools.

See What Do We Actually Mean by “AI-Powered Search”?

One way to fix this is simply to set a extremely high weight to this mandatory criteria. But I think a cleaner approach is to set a minimum threshold that the gated criteria must reach, or else it fails no matter what the overall score is.

Retrieval should be a core gate, not just another category

Besides abidding by legal requirements, for AI search tools, in my view retrieval capability should be one of those gates and this is often overlooked.

A search tool with weak retrieval should not be rescued by good citation formatting, a nice interface, or convenient administrative features. Those things are useful, but only after the tool has passed the basic test of finding and ranking relevant material.

In many evaluation matrices, retrieval strength, even when combined with source coverage, ends up accounting for perhaps 20 to 25 per cent of the overall score. This often happens because frameworks have four or five broad categories and weight them equally.

That badly underweights retrieval. Finding relevant material is the central purpose of a search tool. If the retrieval is poor, the rest is secondary.

Again, I am not arguing that retrieval overrides every other concern. I am arguing that retrieval should be treated as at least ONE core task-performance gate for search tools.

For example:

If a tool is rated “poor” on retrieval capability, it fails the evaluation, regardless of its performance on other criteria.

This approach forces evaluators to decide what is non-negotiable. It also prevents a tool from getting an acceptable overall score by compensating for bad retrieval with peripheral strengths.

The screenshot below shows my example, where retrieval capability, measured through a simple precision test, is set as a critical criterion. The tool needs to be rated 2 or above, corresponding to 50 to 80 per cent precision. In this example, it is rated only 1, corresponding to 31 to 50 per cent precision, so it fails the gating criterion.

Figure: Example of a gate-based evaluation model. Retrieval capability is set as a critical criterion and requires a score of 2 or higher. The tool automatically fails here because it is rated only 1

Tools that fail the minimum retrieval threshold are rejected before secondary criteria such as usability, citation handling, export, or administration features are considered.

Figure: The summary screen highlights that, regardless of the overall score, the tool failed the critical criterion.

Paying attention to retrieval is particularly important now because “AI search” is not a mature or stable category. Many vendors are under pressure to show that they are “doing AI”. Some products are genuinely rethinking retrieval. Others are merely bolting an LLM onto a conventional search system, using it to generate Boolean queries and then calling the result AI search.

That may or may not improve performance. Sometimes it may add little. Sometimes it may make things worse.

I have personally tested an “AI search” tool that was worse than standard lexical keyword search. It was not subtly worse. It was obviously broken. Librarians and researchers testing it could tell almost immediately that the retrieval quality was poor.

If I had treated retrieval as a gate, I could have saved everyone time before asking them to test it.

This is why a gate-based model matters. It prevents the evaluation from being diluted by a long list of attractive but secondary features and can save time.

3. Some criteria should be made harder and more testable

Another thing I notice about many librarian evaluation matrices is that they rely heavily on qualitative judgement.

Sometimes this is unavoidable. Some criteria really are soft. Usability, for example, is difficult to reduce to a single objective number unless you run proper usability testing, which takes time and expertise.

But some criteria can be made more empirical, especially the critical ones.

Take reproducibility. A simple test is to run the same query several times and record whether the same results appear in the same order. You might run the query five or ten times, possibly across different sessions or accounts to reduce caching effects, and see how stable the ranking is.

Take interpretability. If an AI search system converts a natural language query into a Boolean string, you can copy the generated Boolean query into the conventional search interface and check whether the results match. If they do not, then either the system is not doing what it claims, or the Boolean explanation does not fully represent the actual search process.

Take claim-source faithfulness. If a tool generates an answer and cites sources, evaluators can check how often the cited sources actually support the claims made.

Michael Gusenbauer’s work has influenced many of my thoughts here. His papers offer comparatively objective ways to assess academic search systems for evidence synthesis, including tests of search functionality, retrieval qualities, database size, and subject coverage. SearchSmart builds on this work by estimating coverage of academic databases using methods such as query hit counts, and the Basket of Keywords approach, rather than relying only on vendor descriptions or self-reported coverage.

But the area where I think empirical testing matters most is retrieval capability. If retrieval is a core task-performance gate, then it should not be assessed only by impression.

This is where things get more complicated.

Alfred Wallace’s framework asks evaluators to assess the “AI search architecture spectrum”, distinguishing between LLM-generated Boolean, Boolean plus reranking, hybrid search, and agentic or deep search.

Figure: Alfred Wallace’s framework asks evaluators to assess the AI search architecture spectrum.

That is a useful first approximation. In general, I would expect a well-implemented agentic or deep search tool to outperform a tool that simply asks an LLM to generate a Boolean query and then runs it through a conventional retrieval system.

But retrieval is messy. Implementation matters. A poorly implemented “deep search” tool may perform worse than a simpler hybrid system. Marketing labels are not enough.

So we need empirical tests, even if they are rough.

How might librarians test retrieval strength?

Last May, I wrote two posts on testing AI academic search engines(I) and (II). I also drafted a third post focused specifically on retrieval testing, but it became too long and technical. It went from TREC-style ad hoc retrieval evaluation to modern RAG evaluation, and I eventually decided it was probably too much for the audience I had in mind.

The challenge is balance. We should not expect most librarians to run formal information retrieval evaluations. We are not trying to reproduce TREC methodology, nor are we trying to run a Study Within A Review such as the Cochrane Evaluation of (Semi-) Automated Review methods(CESAR).

But we can do better than impressions.

Below are two lightweight but far from perfect tests of retrieval for librarian evaluations of search tools that you can try2.

This will not produce bullet-proof, publishable IR research. But it is often better than simply giving your feelings on whether the results “look good” in general.

Retrieval test 1: recall@K using a known set

One relatively simple test is to use a topic where you already know many relevant papers and feel you likely have most if not all relevant papers.

For example, suppose you have worked extensively on a topic and already have a set of papers you consider relevant. You can treat that set as a rough gold standard. Then you run the same query in each AI search tool and check how many of those known relevant papers appear in the top K results.

This gives you recall@K.

For example, if your gold standard set has 20 relevant papers, and a tool retrieves 8 of them in the top 30 results, then its recall@30 is 8/20, or 40%.

The question, of course, is what value of K to use. There is no universal answer. Common values might be 10, 20, or 50. Personally, I would often use 20 or 50 because that roughly matches the number of results I am likely to scan seriously.

One obvious source of gold standard sets is systematic reviews, meta-analyses, review articles, and survey papers. You can use their included studies or references as a benchmark.

There is a catch. If the review article has already been published, some AI search tools may find the review itself and mine its references. That makes the test less clean. Still, for practical library evaluation, it may be good enough, especially if the aim is not formal research but comparative assessment.

For more sophisticated testing, you could also use NDCG@K, or Normalised Discounted Cumulative Gain. This is rank-sensitive, meaning it gives more credit when highly relevant results appear near the top.

Retrieval test 2: precision@K when no gold standard exists

Not everyone has a gold standard set of papers lying around. In that case, a more realistic test is precision@K.

The process is simple:

1. Choose a realistic query.

2. Run it in each tool.

3. Look at the top K results, such as the top 10.

4. Judge how many are relevant.

5. Calculate the proportion of relevant results.

If 7 of the top 10 results are relevant, precision@10 is 70%.

This is easy to understand and relatively easy to run. It does not tell you whether the tool found all the relevant literature, but it does tell you whether the top results are useful.

For many users, especially undergraduates and those just doing quick exploratory searches, this may be a reasonable test. They are often not trying to find everything. They are trying to find enough good material quickly.

However, precision@K has a limitation: it is not order-sensitive. A tool that places the best result at rank 1 and another that places it at rank 10 may receive the same precision@10 score. If ranking quality matters, you could also consider Average Precision (AP@K)

Again, though, I would not overcomplicate this at the start. A rough precision@10 test across several realistic queries is already useful. In fact, I suspect most librarians or researchers already sort of do that, just that they don’t formally keep track of it.

Figure - Three lightweight ways to test retrieval strength.

Why formal information retrieval is not easy

Studying and evaluating information retrieval results requires a ton of expertise.

I don’t want to make the discussion too technical but here are several issues that complicate the story.

• We need to distinguish corpus coverage from retrieval and ranking. A tool may fail to retrieve a paper because it is not in the corpus, because the query did not match it, or because the ranking pushed it too far down - when Undermind does not find a known gold standard relevant result at say recall@50, you should do a known item search in the source it uses to check if the fails is due to the corpus or the search

• We need to be careful when using known sets from systematic reviews, meta-analyses, review articles, or survey papers. These can be useful, but they are not perfect gold standards.

• Precision testing sounds simple, but formally, you have to ensure there are clear inclusion and exclusion critera before judging the results and to ideally need multiple assessors to avoid becoming just another form of subjective impression3.

• We need to choose metrics that are understandable and appropriate. Precision@K, recall@K, known-item success, NDCG@K, and MAP@K answer slightly different questions.

• We will have difficulty deciding what is a “normal” baseline recall/precision@K etc for an academic search. What could be the baseline? Google Scholar? Your library catalog/discovery service?

These are important, and I will discuss them issues and more in future posts. For this post, the simpler point is enough: if retrieval is central to the purpose of the tool, then retrieval should be tested, not merely described.

Choosing good test queries

Still, I want to briefly discuss a critical issue when testing search engines. Choosing which queries to test is obviously tricky.

But a very common mistake I see is testing only “easy” queries. An “easy” query is not always the same as a broad query, but the two often overlap. If your topic has many obviously and easily found (because terminology is known and stable) relevant papers, most decent systems will return something plausible. That makes it hard to distinguish strong retrieval systems from merely adequate ones and with “AI search” we want to do that.

I occasionally see librarians say the new class of AI search engine isn’t much better than conventional search mostly because they make this mistake or alternatively they can’t tell the difference between AI search engines of obviously different quality as the test results look close.

You can often tell this is happening when multiple retrieval systems achieve high apparent precision but return very different sets of results with low overlap. Low overlap between systems may suggest that there are many relevant papers in the index, so the test is not very discriminating.

A better test set should include some harder queries. For example:

• topics where terminology varies across disciplines;

• emerging topics where vocabulary has not stabilised;

• interdisciplinary questions;

• questions involving very specific conditions for relevance;

• known-item searches for papers that should be found but are not obvious from simple keywords.

Overall, if you know your query has very few relevant articles and those articles cannot be found easily with trivial keywords, the query is likely to be hard.

How many queries should you test? The glib answer is: as many as you can. You could even crowdsource them. But some testing is better than none.

I find even five well-chosen queries can reveal a lot, allowing you to distinguish the average from the best, especially if they include difficult cases.

The unanswered question about cost

Even if a tool passes the retrieval gate and satisfies role-specific criteria, there is one final pragmatic hurdle: the cost-to-performance ratio.

In the information retrieval research community, people often chase marginal gains in metrics such as Mean Average Precision (Average Precision across multiple queries) without caring the cost (in terms of compute or latency). In an institutional library context, the question is different. A 5 per cent improvement in retrieval performance may or may not justify a much higher subscription cost, additional staff training or even privacy risk.

So the question is not simply whether Tool B retrieves slightly better results than Tool A. The question is whether that improvement is worth the cost and trade-offs for the institution and for the users the library is trying to serve.

That is not an easy question. But at least a role-based, gate-based, test-informed framework gives us a better way to ask it.

What this means for AI search evaluation frameworks

Putting all this together, I would make three changes to most AI search evaluation frameworks.

First, I would scope the framework to a specific user role and task. A single universal score is usually misleading. The better question is: useful for whom, and for what purpose?

Second, I would treat retrieval capability as a core task-performance gate for search tools. This does not mean retrieval is the only possible hard gate. Privacy, accessibility, licence terms, data governance, and institutional requirements may also be non-negotiable. But if we are evaluating a search tool, it should first be able to search well.

Third, I would make at least some criteria more empirical. Reproducibility can be tested by rerunning queries. Interpretability can sometimes be tested by comparing generated search strings with actual results. Claim-source faithfulness can be tested by checking whether cited sources support generated claims. Retrieval can be tested, even lightly, using realistic queries and simple measures of usefulness.

Buy me coffee via ko-fi!

The gating point matters most to me. A search tool with weak retrieval is not redeemed by good citation handling, a polished interface, or convenient administration features. Those things matter, but only after the tool has passed the basic test of finding useful material.

I am not planning to publish my own framework. I do not want the maintenance burden, and I do not have a strong personal use case for one.

But if you are building one, my advice is simple: decide who it is for, decide what is non-negotiable, and make the most important criteria as testable as you reasonably can.

In future posts, I will go deeper into retrieval testing itself.

Subscribe now

What would help is work that treats these systems as understandable and evaluable: layered architectures with retrieval components, generation components, interfaces, and trade-offs that can be named and weighed.

As Mike Caulfield of SIFT fame has observed there is great value and explanatory power to see what AI does from the lens of “a fancy search result” rather than just an answer. He notes

… stop thinking of the result as an answer, and start thinking of it as a search result with some synthesis on top. When all the sources are in alignment that can of course present as an answer — but it still is downstream from available sources, it still has made decisions about weighting, and so forth. If you think of it as a dressed up search result instead of an intelligence (albeit one that can synthesize in impressive ways) you’ll be better able to process the weirdness that sometimes results.

Over the last week, three librarians — none of them coordinating with me or with each other — have published work on AI search that lean in this more productive direction. Aster Zhao at HKUST, Wang Huajin at Carnegie Mellon, and Alfred Wallace at the University of North Dakota each built something concrete: an interface, a framework, a rubric (vibe coded with Claude). All three happen to cite my work, which is flattering.

Buy me coffee via ko-fi!

Taken together, these three projects make a point I keep returning to. The conversation around AI search needs to happen at a better level. We need fewer sweeping claims and more attempts to clarify how systems differ, what assumptions they encode, and how they should be assessed for particular tasks.

1. Aster Zhao’s (Hong Kong University of Science and Technology) snapshot of GenAI tools for research

The first example is by Aster Zhao from HKUST Library: Snapshot of GenAI Tools for Research.

What I like about this one is that it does more than list tools (and a very comprehensive list it is with 40+ tools). Besides a comprehensive table view, it includes a strong timeline showing the evolution of AI search and the different “roles” that AI plays in search tools. That visual layer makes a messy landscape much easier to understand.

The work also cites some of my own frameworks and distinctions, including:

• the four-quadrant framework, which divides search into Quick Search, Quick RAG, Deep Search, and Deep RAG

• The distinction between academic deep search tools that are “workflow-bound” vs autonomous , with the former struggling when asked to do tasks they were not designed for.

• The Horseless Carriage of AI Search, where I critique vendors trying to sell and equate “AI search” to just simplistic use of LLMs for Boolean query construction and similar bolt-on approaches, while ignoring other more productive approaches like semantic search, better reranking and agentic search.

Overall, this is a strong and current map of the space, helping people to see that AI search is not one thing. I have been planning to update my own list of such tools here but Aster’s work has reduced my incentive to do so.

2. Wang Huajin’s (Carnegie Mellon University) AI-Powered Tool Assessment Framework

The second example is AI-Powered Tool Assessment Framework by Wang Huajin of CMU.

This project is especially interesting because I often get asked during my talks or workshops for an evaluation framework for AI search tools, and in truth I have not been fully happy with many of the existing ones (more on that later). This framework gets closer to something I could actually endorse.

It asks evaluators to consider questions across four areas: retrieval, generation, output, and usability. That already puts it ahead of a great deal of discussion, because it treats these systems as layered and evaluable rather than as black boxes that either “work” or “hallucinate”.

It appears to draw substantially on my older Testing AI Academic Search Engines series (part I and II) - adapting many of the questions I put in them, though I do not want to overstate my influence here. There is at least one section, on sustainability, that does not come from me, and that is a good thing.

Huajin will also be presenting at FORCE2026 in June in Singapore on “Developing an assessment framework to support critical evaluation of AI-powered academic search engines”, which suggests this is not just a quick web experiment but part of a serious and wider conversation within her institution. I am looking forward to her presentation.

3. Alfred Wallace’s (University of North Dakota) adjustable rubric for AI search tools

One reason I have resisted producing a neat evaluation matrix with fixed weights, a catchy name, and a polished PDF is that evaluation is rarely one-size-fits-all. Different users, tasks, and contexts demand different priorities. A tool useful for helping undergraduates find a few relevant papers will be unsuitable for a search tool meant for higher recall results. A tool that is fast and convenient may be unacceptable if transparency or source traceability matters.

The third project by Alfred Wallace at the University of North Dakota - Evaluating AI Tools for Research handles this better than most. Its structure is fairly traditional, focusing on sources, models, and wrappers, and the categories themselves are not ground-breaking.

The framework incorporates the provenance of the tool (established vendor, startup, or user-built), which is essential for maintaining a macro-level, long-term perspective beyond isolated feature sets.

Finally, it has one important feature: users can adjust weights of criteria when calculating a final score. That is a much more sensible approach than pretending there is a universal rubric that works equally well for everyone.

As it now stands, you select a Scenario based on whether you are a faculty researcher, Graduate Student, Undergraduate, instructor or student which will automatically flag different criteria as either “priority” or “key question” categories that affect the weighting. While you cannot change which criteria1 are considered “priority” or “key question”, you can change the size of the multiplier for criteria in either categories.

The criteria listed also reflect an informed understanding of how AI search systems differ architecturally. Alfred cites my writing (citing specifically the horseless carriage post) to support his “AI search architecture spectrum”, including the distinctions between LLM-only approaches, Boolean generation, Boolean plus reranking, hybrid search, and agentic or deep search.

Again, what matters here is not whether every category or weighting is perfect. It is that the tool reflects a more informed understanding of how AI search systems differ under the hood.

This project ends with a fascinating section on “Visions of the future” including the struggle between a future where users work on explicit platforms (both existing and new ones) vs ones where “literature research lives in the same tool that does data analysis, coding…”.

Conclusion

To be clear, I do not want to overstate my influence on any of these projects. Nor do I necessarily endorse every detail in all three.

In fact, I have many things to say about assessment frameworks and metrics for AI search tools in my next post.

But I do think they are all amazing attempts to engage with AI search by librarians.

That said, three projects, all citing me have an obvious selection effect: the librarians already engaging at this level are precisely the ones likely to read what I write2. So this is not the strongest evidence that the wider conversation has shifted. Still, anecdotally, looking at recorded talks and presentations by academic librarians, there seems to be much more understanding of how AI search works under the hood compared to say in 2023/2024.

If AI search tools are going to become part of academic workflows, we need more work of this kind: work that helps us distinguish between kinds of systems, evaluate them in context, and make their assumptions more visible. Blog posts can start those conversations. Interfaces, frameworks, and rubrics make the ideas usable for others.

Buy me coffee via ko-fi

Bonus

As a bonus example, I would also point to my recent SMU Libraries piece on MCP for researchers. It is doing something slightly different from the three projects above, but in a complementary way.

Rather than offering a framework or rubric, it tries to explain in concrete terms what happens when tools like Scite and Consensus are connected to Claude or ChatGPT via MCP, so that the model can search across sources, compare results, and carry out multi-step academic workflows on the user’s behalf.

In that sense, it is less about evaluating AI search and more about making the underlying infrastructure and possibilities legible to researchers. It is also, I think, one of my clearer pieces on the topic, partly because it was written for an institutional audience rather than as one of my longer blog posts where I am often thinking aloud.

Subscribe now

Introduction

Academic search tools might be moving towards a different architectural model. The first generation of AI-powered literature review tools — Undermind, Consensus Deep Search, AI2’s Asta — relied on predetermined, hand-crafted workflows. You entered a query, the system executed a fixed pipeline, and you received results. The LLM’s role was largely confined to query understanding and output formatting.

That might be changing. As LLMs have become more capable of flexible, multi-step reasoning, vendors are beginning to build systems where the AI can dynamically select and chain tools to accomplish tasks that were never explicitly programmed on their platforms. This may mark a move from academic pre-defined “deep research” workflows to more flexible agentic search flows1 being available on vendor platforms and in this post, I will discuss some of the early examples such as Undermind Projects.

But this shift introduces a problem that is not getting enough attention. When a system can theoretically do anything, how does the user discover what it can actually do?

Unlike a advanced user who sets up his Claude Code as a “research agent”, a user of a vendor platform has nearly no insight to how the agents have been setup by the vendor.

I call this the ‘invisible menu’ problem (a rename of what I previously called the “blank box” problem) and I think it may become one of the new usability challenges for the next generation of AI academic search tools that include agents on their platform.

The problem is compounded by the fact that, unlike traditional library search tools where decades of use have established shared conventions and mental models, agentic search is so new that users have no baseline expectations for what these systems should even be capable of.

Share

In my previous post, I argued that bolting LLMs onto legacy search paradigms, what I called the "horseless carriage" approach, delivers limited benefit, and that most of the potential improvements in search will come from better reranking and agentic search.

In this post, I examine how three tools (Elicit, Undermind, and SciSpace) are attempting to move towards embedding agentic capabilities or flows in their platform. I use Elicit and especially Undermind's new Projects feature as the main worked examples and using what I find using them to illustrate the invisible menu problem in practice. SciSpace is discussed more briefly as a contrast point, since I have not tested it as deeply.

Buy me coffee via ko-fi

Why agentic now?

Ian Mulvany, CTO of BMJ Group, responded to my earlier post on the lack of agency in academic deep research tools with an observation that captures the moment well: “The fixed workflows that the current crop of tools are showing were developed when models were weaker, yet from a product perspective, trying to build something that is simultaneously specific and general, is really hard.”

Incidentally, Ian Mulvany will be the second keynote speaker for FORCE 2026 to be held in Singapore, 3-5 June - speaking on “Surviving the Disruption: Scholarly Communication in the Age of AI”. FORCE2026, will have many tracks on this topic, including but not restricted to impact of AI on literature review (including a presentation by yours truly on “Does Agentic Deep Search Converge? Reproducibility Questions for LLM-Driven Literature Discovery”

He is right. The tools I tested in that earlier post often struggled when asked to do things outside their intended workflows, even when they seemed to have some of the underlying capabilities needed to succeed. The models either were not yet good enough to reason flexibly across available tools, or were more likely constrained from doing so by the system design.

That constraint is loosening as vendors realise the capabilities of these new models. And the response is coming from multiple directions simultaneously.

Vendors of specialised academic search tools face a strategic choice. They can make their products available as components in user-built agentic workflows — typically by providing MCP servers, as Scite, Consensus, and SciSpace have done, or APIs, as Elicit has (currently in beta).

“What If Claude or ChatGPT Could Search Academic Databases For You — and Then Do Something Useful With the Results?” is a simple guide I wrote for our users.

But they also want to maintain the value of their own platforms rather than becoming a commodity connector inside Claude or ChatGPT. Being just one MCP server among many risks ceding control of the user experience and, ultimately, the business relationship. Their response, increasingly, is to try to have it both ways: offer the connector while also making their own platform more agentic. Even publishers are moving in this direction — Wiley AI Gateway, is one of the earliest academic MCP server adopters, though as a publisher rather than a search aggregator, their interests are fundamentally different from tools like Scite, Consensus, or SciSpace and Wiley’s MCP server returns full text chunks that may be relevant to the query on top of metadata of the paper (see my earlier review).

Libraries are not standing still either. UT Libraries has built a working demonstration connecting Claude to their library catalogue (Primo/Alma), Texas Archival Resources Online, their digital collections, and the Harry Ransom Center Digital Collections — it can even identify the right subject librarian and give instructions for accessing archives.

Yale University Libraries is piloting a similar MCP connector to their catalogue, alongside a year-long trial of Consensus.app. If libraries can wire LLMs directly to their collections, the role of intermediary search platforms becomes less certain.

The benchmark test: finding uncited papers

Before examining each tool, let me explain the test I have used in the past and use throughout this post. The task is: given an open access paper X, find papers that could have been cited in X but were not

This is a useful yet simple test of agentic capability for several reasons. First, I don’t believe it is a [predefined workflow that any vendor has pre-built, so the system must reason about how to accomplish it rather than following a script. Second, it requires genuine multi-step reasoning: the system must extract the references of paper X, search for papers on similar topics published before X, and then ensure that the papers it presents are not already in the reference list. Third, the target paper I use — "The state of OA: a large-scale analysis of the prevalence and impact of Open Access articles" — is open access and easily accessible, so even if the model lacks a specialised search tool to retrieve the references directly, it should be able to find them via a web search.

I should note its many limitations. This is a single task type that tests one kind of multi-step reasoning. A tool could fail this test and still perform well on other agentic tasks, or pass it through a fortunate sequence of actions rather than robust reasoning2. It is a useful probe, not a comprehensive evaluation.

Elicit Research Agents

Elicit has long been known for providing tailored workflows with human-in-the-loop features. They have expanded the sources they can search, going beyond their own academic index to include clinical trials, PubMed, and the web. Most recently, they added “Research Agents,” which appear to be agentic in the way generic LLMs are, combining available tools or workflows flexibly to solve tasks rather than following a fixed pipeline.

I tested Elicit with my usual test, and found the following prompt usually works:

Search the web find the references of the paper “The state of OA: a large-scale analysis of the prevalence and impact of Open Access articles.” Find papers that could have been cited but were not.

The critical detail here is “search the web.” Without that instruction, Elicit defaults to its academic search tool, which can find the paper but cannot give the agent access to its references. Screenshot below shows what happens if it uses the Elicit academic search.

If you instead direct it to use its web tool, it behaves like any LLM with web access: it finds the full text of the paper, extracts the references, and proceeds with the task.

This is a first early example of what I will be calling “the invisible menu" problem. The user has no obvious way of knowing that Elicit’s academic search tool cannot extract references from a paper, or that switching to the web tool solves the problem. The system has the capability, but the user must guess the right incantation to unlock it.

Undermind Projects

Undermind recently added a beta "Projects" feature that represents a more substantial move towards agentic architecture. Unlike the current Undermind, which offers essentially one workflow (enter a query, answer clarifying questions, receive a report), the Projects feature provides three agents: a Search Architect, a Report Writer, and a Generalist.

The Search Architect is designed to ask clarifying questions and eventually propose a "deep search," which is Undermind's full search capability rather than a standard LLM web search.

The Report Writer works with papers found from past searches to generate reports in the format and structure you want. Like the Search Architect, it typically asks clarifying questions before proceeding.

The Generalist is less clearly defined, but it has access to every paper across all your searches. In practice, the LLMs are capable of calling each other when needed — you might be interacting with the Search Architect, but once a search completes, it may recommend switching to the Report Writer.

Why this matters

One of the most significant advance here is not any single agent but the ability to run multiple searches and combine the results. One longstanding piece of advice I give to users of the current Undermind is to ensure your query covers a fairly focused topic, expecting perhaps 10-50 relevant papers. A broad query like "how are LLMs used in evidence synthesis" is too unfocused; LLMs are used differently at different stages.

With the Search Architect, you can attack a broad research question (e.g. how are LLMs used in evidence synthesis) by running multiple focused searches to address different sub-questions:

a) How well do LLMs perform as title-abstract screeners in evidence synthesis?

b) How well do LLMs perform when generating Boolean search strategies for evidence synthesis?

c) How well do LLMs perform at data extraction in evidence synthesis?

d) How well do LLMs perform in critical appraisal for evidence synthesis?

All results feed into a unified “All papers” library.

Lastly the Report Writer can then synthesise across them. This part is very similar to the “Chat with experts” feature in the original Undermind, you can select predefined instructions to focus on different aspects for the report.

It will also ask clarifying questions.

Test result

I tested Undermind Projects with my standard query: find papers that could have been cited in “The state of OA” paper but were not.

find me papers that could have been cited in the paper “The state of OA: a large-scale analysis of the prevalence and impact of Open Access articles.” but was not

The system asked clarifying questions, then read the full text of the target paper (identified as “piw18” in its system).

After further clarification, it offered to launch a deep search, which I did.

At this point, I was uncertain whether the request would succeed. Undermind’s deep search is capable of finding papers on a similar topic and filtering to papers published before the target, but it cannot by itself filter out papers already cited.

Once the search finished, the Generalist agent suggested running a report.

I proceeded by clicking on the “Generate Report” button, and started prompting in the Report writer agent.

The report writer did its usual clarifying questions

The critical step came when the agent began “reading full texts of Piw18” — it was checking candidate papers against both the in-text citations and bibliography of the target paper.

For example, Archambault et al. 2016 (Arc16), found by the deep search, was checked against piw18 and excluded because it was already cited.

The system passed the test, but my testing of the new Undermind interface revealed several problems along the way.

Invisible menu problems in Undermind Projects and probably other agentic interfaces.

While testing, I found myself uncertain about basic questions of system capability. Can you give prompts to the Report Writer that refer to specific papers using their keys (e.g. piw18)? Can you reference reports or searches that have already been completed? Can you ask the Report Writer to modify an existing report rather than creating a new one?

The answers to all of these turned out to be yes, which I confirmed by trying and then checking with the developers. For instance, you can ask the Report Writer to use results from a particular search and create a report with "a table of the most relevant papers, each with a column describing possible limitations and why it might not fully address my research goal." You can then ask it to add more sections to the same report. Some of these capabilities are surfaced by the system — it sometimes asks whether you want a new report or to extend an existing one, or even shown as a GUI feature — but not all of them are.

I also raised with the Undermind developers that the new system requires more effort to get the same output as the current one. Under the old system, you enter a query, answer clarifying questions, and receive a comprehensive report. Under the Projects system, you must additionally run the Report Writer, potentially multiple times, each time being asked for clarification.

I requested something like a Claude.md or Claude Skill — a way to specify in advance the sections and structure I wanted, so I could get the full report in one shot. Undermind is still considering this, but the developers revealed a useful workaround: the Generalist agent has access to all the same resources as the Report Writer, but is much less likely to ask clarifying questions. If you want to bypass the Report Writer’s clarification loop, give the same instructions to the Generalist instead.

This is a genuinely useful tip. It is also not the sort of capability most users, even experienced ones, would be likely to discover on their own.

The invisible menu problem

Imagine walking into a restaurant that has no menu. You know the kitchen can cook, but you have no idea what dishes are available, what ingredients they have, or what you should ask for. You might get a brilliant meal if you happen to request the right thing, or you might miss the house speciality entirely because it never occurred to you to ask.

Now make it worse: you do not even know what type of cuisine this restaurant serves. Is it Thai? Italian? Fusion? The sign outside just says “food.” That is the situation users face with agentic search tools today.

Compare this to a user who setups Claude Code by hand to do agentic flows, in our analogy, he would be the owner of the restaurant, and he knows how the resturant was setup, who the cooks are, what their specialities are etc. If he just copies the Claude Code workflow from someone else without understanding - we will be back to the same issue of course.

As I noted in the introduction, I previously called this the “blank box” problem, but “invisible menu” better captures what is going on. The issue is not that the system is blank or empty — it is full of capabilities. The problem is that there is no menu to tell you what they are.

The invisible menu problem arises because an agentic system’s capability space is so broad that users struggle to find out what is possible through normal interaction and is worsened if the user were not the one who setup the agentic system and hence does not understand even in principle what is possible.

The system can do things the user will never discover if there are no menus, no buttons, no tooltips — just a prompt field. The user must guess what to order. This is illustrated by all three examples from this post: knowing to say "search the web" in Elicit, knowing that paper keys work as references in Undermind, knowing that the Generalist agent bypasses the Report Writer's clarification loop. In each case, the capability exists but is invisible to normal use.

Compounding this, users cannot even fall back on conventions from elsewhere. With traditional library search tools, there is a shared mental model. Users know roughly what to expect from a discovery layer or a database: keyword search, date filters, subject headings, citation export. The specifics differ between Primo, EDS, and Summon, but decades of use have established conventions, and a user moving from one system to another can transfer most of their expectations.

With agentic search tools, no such shared mental model currently exists. Every tool has a different set of capabilities, different tools connected underneath, and different invisible constraints. Users cannot form reasonable expectations before they start, which makes the per-tool discovery problem even harder.

Consider how the user does not know what system instructions each agent is already given, what tools or sources it has access to and more etc and is either going to not use the system’s agent capabilities to the fullest or on the flipside give prompts or inputs expecting to be doable but failing. This can be particularly dangerous because such agentic systems often “fail silently” or even if they state plainly they failed, the user may not notice this failure.

Advanced users setting up their own agentic flows with Claude Code or Sciclaw will not face this issue as much, as they know exactly what and how it has been setup.

Undermind features an 'agentic sources' button that surfaces example prompts, mitigating the invisible menu problem by explicitly showing users that they can reference paper keys

Below shows what happens when you click on that.

The screenshot gives you example prompts that give you cues on what you can do. For example, this popup reveals you can give instructions to the Report Writer by referring to the paper key in the Undermind Library.

There is not yet a well-established baseline for what an agentic research tool does that users can carry from one product to another. Users cannot even form reasonable expectations before they start.

This is distinct from the well-known “black box” problem of AI opacity. A black box hides how the system reaches its output. An invisible menu hides what the system can do in the first place. Both are problems, but the invisible menu problem is specific to agentic systems and, I think, currently underappreciated.

It also has direct implications for reproducibility and trust. If the user does not know what the system did, what tools it invoked, or what alternative paths it could have taken, how can they evaluate whether the output is reliable? With a fixed workflow, the steps are predictable and auditable. With an agentic system, the tool may take different paths each time, and the user may not even realise what choices were made on their behalf.

Part of this may prove temporary. As these tools mature, shared expectations across the category will form, and users will at least arrive with a baseline of what to expect. What will not go away on its own is the per-tool discovery problem: even with category conventions in place, each system will still have its own hidden affordances unless vendors deliberately surface them. In the interim, the cognitive burden on users is substantial.

SciSpace Agents

I should be upfront that I have not tested SciSpace Agents as deeply as Elicit or Undermind, so what follows is a positioning analysis rather than a hands-on evaluation.

SciSpace has gone all-in on the agentic model. Their platform is packed with over 150 “apps” and connectors to a wide range of tools and agents that go well beyond literature review3 — searching, writing, data analysis, and more.

They offer a rich agents gallery.

In concept, this amounts to a simplified, less technical version of building your own research agent with Claude Code/ sciclaw : all the tools are connected automatically, requiring minimal setup.

The trade-off is cost — SciSpace charges by credit usage — and the fact that for all its versatility, it is unproven that this breadth actually delivers better results than more focused alternatives. SciSpace occupies a middle ground between the full flexibility of a DIY setup like Claude Code and the guided experience of platforms like Undermind and Elicit, which raises the question of whether that middle ground has a natural audience.

In terms of the “invisible menu” issue, again this is mid-way between a system like Undermind Projects and complete DIY setup like Claude Code, since you get your hands dirty more than the former but less than the latter, giving you immediate understanding of the agentic flows.

Conclusion

The academic search landscape may be shifting from rigid, predetermined workflows towards more flexible agentic architectures. Elicit’s Research Agents, Undermind’s Projects feature, and SciSpace Agents each represent different points on this spectrum, but they all reflect the same underlying recognition: LLMs are now capable enough to reason across tools rather than merely execute a fixed pipeline.

Yet as my testing reveals, “more agentic” does not automatically mean “more usable.” When a system can theoretically do anything, the user bears the cognitive burden of figuring out what it can actually do, which tools it will invoke, and how to phrase requests to get the desired outcome. The Undermind example is telling: even a power user (me!) needed developer tips to use the system efficiently. That does not look like a sustainable model for broader adoption.

As I discussed earlier, vendors are hedging their bets — offering MCP connectors while simultaneously layering agentic capabilities onto their own platforms, and libraries are beginning to wire up LLMs directly to their collections. Where this leaves the middle ground is uncertain.

My suspicion is that a general-purpose agentic platform for research, such as SciSpace Agents, will struggle to find a big audience. Technical early adopters will build their own workflows with Claude Code or similar tools like Sciclaw. Less technical users will gravitate towards familiar interfaces like Claude or ChatGPT with a few MCP connectors attached, or towards focused platforms like Undermind and Elicit that guide them through the process. One key question for vendors is not just whether to become more agentic, but how to do so without making their tools harder to use

What seems clear, at least from these examples, is that some tools are moving past the ‘horseless carriage’ stage I described in my previous post. The conversation has moved beyond simply bolting LLMs onto legacy search paradigms.

If the above is correct, a major challenge now is designing agentic systems that are genuinely discoverable — where users can understand what the system is capable of without needing insider knowledge or developer access. Until that challenge is addressed, much of the promise of agentic academic search is likely to remain concentrated among users willing to tinker.

I have been thinking about what it would take to overcome the invisible menu problem — what kinds of affordances, documentation, and design patterns could help users navigate these systems more effectively. I hope to explore that in a future post.

Buy me coffee via ko-fi

TL;DR: Turning natural language into Boolean is not the future of library AI search. It can help with some failed queries, but the bigger problem in most discovery systems is weak ranking, not the need for better Boolean. More radically, I conclude by arguing Boolean itself may no longer be the right foundation for retrieval in an era of BM25 (loose/fuzzy boolean), hybrid search, and agentic search.

There is a growing trend among library vendors and AI search startups to use LLMs to convert natural language queries into Boolean search strings (dubbed LLM-to-Boolean from now on). Examples that use this method as part or whole of their retrieval pipeline include Scopus AI, Web of Science Research Assistant, Web of Science Smart Search, Primo Research Assistant, Summon Research Assistant, EBSCOhost Natural Language Search, Scite.ai Assistant, and many more. Universities are also building their own versions, such as Stony Brook University’s SEARCH AI and San Diego State University’s OneSearch AI Assistant.

With this many vendors and institutions converging on the same approach, you might reasonably assume this is what “AI-powered search” means — that this is the important innovation, perhaps even the whole story of how AI will transform library search.

It is not. Not even close. Using an LLM to convert natural language to Boolean may be the horseless carriage of AI-powered search1. When the automobile was invented, the first instinct was to build something that looked like a horse-drawn carriage but with a motor bolted on. It took years before designers realised that the new technology demanded a fundamentally different form. LLM-to-Boolean is the same kind of thinking: it takes the most powerful text-understanding technology ever built and uses it to produce the exact same artefact — a Boolean query string — that librarians have been crafting by hand for decades. In the simplest implementations, the LLM is mainly a more elaborate query-construction layer placed in front of an otherwise conventional lexical search engine.

I am not saying this approach has zero value. For a novice user who types a full sentence into a strict Boolean search engine and gets zero results, having an LLM translate that into a workable query is a real improvement. But used alone — without modern reranking (e.g. neural ranking, embedding retrieval methods, or agentic search) — I am skeptical LLM-to-Boolean does much to improve result quality for most users2. It is important for vendors and librarians to realise this is not the main event, rather it is a minor supporting act.

Buy me coffee via ko-fi

Retrieval versus ranking: the distinction that matters

If we are serious about improving search in library discovery systems, we need to stop tinkering with query construction alone and start addressing the real bottleneck: what happens after retrieval.

Information retrieval teaches that search engines do two things: they retrieve documents (deciding which ones make it into the result set) and they rank those documents (deciding the order you see them in). These are distinct stages, and improvements to one do not automatically improve the other.

Boolean only retrieves a candidate set of results. Strictly speaking, every document that matches the Boolean logic is treated as equally relevant. Ranking — typically based on lexical relevance functions like TF-IDF (term frequency - inverse document frequency) or its more sophisticated successor BM25 — is what determines the order you actually see results in. Most library search systems already have both stages, even if the ranking layer is not always well documented.

Most library or academic search systems rely heavily on lexical retrieval and lexical/proprietary relevance ranking that are TF-IDF/BM25-like based on a combination of term frequency, document frequency, document normalization and field weighting with some additional weighting for citation counts.

Scopus for example mentions - “The more often a term occurs in a document, the more likely that it is relevant to the topic of the article” and “Not every word is equally important. A term that occurs in nearly all documents will score less than something unusual. We use calculations based on Term Frequency/Inverse. Scopus uses Document Frequency (TF/IDF) (a concept originally introduced by Karen Spärck Jones, 1972) to assign a weight to any particular word in any collection of documents”

LLM-to-Boolean is entirely focused on the retrieval stage. It changes which documents get retrieved, which matters. But it does nothing to upgrade the ranking stage — the part that determines whether the best results appear on your first page or are buried on page five. In an era when users routinely face thousands of results, ranking is what determines what they actually see.

The problem is that BM25, which underpins ranking in most library databases, is decades-old technology3. It scores relevance by counting how often your search terms appear in a document and how rare those terms are across the corpus. It has no understanding of meaning, context, or user intent.

Subscribe now

Who does LLM-generated Boolean actually help?

There are broadly three types of users to consider.

Novice users have the following failure modes.

Too aggressive with keywords — ANDing too many specific terms, getting zero results. LLMs to boolean might help here with adding synonyms though in my experience this doesn’t help because the user often has just too strict a search.

Too broad query— single or too general term, overwhelmed by thousands of results. LLM-to-Boolean may actually make this worse by adding synonyms that broaden the set further. What this user needs is better ranking not more synonyms.

Natural language as query — users thinking they are working with a natural language search systems and typing a full question, getting zero or bizarre results because the system matches “what,” “is,” “the” literally. LLM-to-Boolean helps here by parsing the question into workable Boolean.

For the increasing number of users who expect natural language search and get zero results because the system cannot parse their sentence, LLM-to-Boolean is a genuine improvement over nothing. But this is a relatively narrow use case: even without this many users quickly adjust and switch to keywords when they notice zero results.

Expert searchers who construct systematic review search strategies. These users gain nothing. They build lengthy, carefully piloted Boolean strategies using controlled vocabulary and proximity operators. As of 2026, LLM-based Boolean generation is still not reliable enough to replace expert searchers, although recent benchmark work — notably the AutoBool project using reinforcement learning — has narrowed the gap on some datasets. LLMs can be useful for brainstorming, but they cannot serve as an automated replacement. Expert searchers are not going to benefit from an LLM doing what they already do, and doing it less reliably4.

Users who type in reasonably simple keywords. This is the largest group and the one that matters most — researchers looking for relevant literature, students doing coursework, information literacy librarians conducting narrative reviews.

This group either types in 3-5 reasonable keywords or construct simpler nested Boolean versions of (A OR B) AND (C OR D) used by systematic review searchers.

Many of these users do not have a retrieval problem: their keywords already retrieve relevant documents. They have a ranking problem. The relevant documents are in the result set but buried beneath less relevant ones. What they need is not a fancier query; they need the search engine to put the best results at the top. LLM-to-Boolean does nothing about this — and, as I will argue, can actually make things worse.

The synonym expansion problem

To understand how LLM-to-Boolean can degrade results for that third group, we need to look at what it actually does to a query.

When an LLM converts a natural language query to Boolean, it almost always adds a large number of synonyms. Strip away the marketing language and what the LLM is doing — beyond dropping stop words and extracting key concepts — is essentially query expansion: transforming a user’s simple keywords into nested Boolean strings with multiple OR’d synonyms per concept5.

Query expansion is not new6. It can improve recall by retrieving more potentially relevant documents. But it also injects noise by retrieving many more marginal or incidental matches. Whether that trade-off is worthwhile depends entirely on whether the synonyms added fit the searcher intent exactly and whether the search system has a strong enough ranking layer to separate signal from noise.

In the next section, I provide three separate arguments on why when LLMs generate Boolean from query inputs, they do not necessarily improve things.

Argument 1 : Nested Boolean is a search paradigm that is no longer useful in today’s search environment

I argued as early as 2014, well before the emergence of LLMs, that nested Boolean searching had become far less effective in many modern search environments and use cases (outside of evidence synthesis). The traditional nested Boolean strategy, decomposing a topic into concepts, enumerating synonyms for each, and combining them in the form (A1 OR A2 OR A3) AND (B1 OR B2 OR B3), was developed for a very different retrieval context: one characterised by limited full-text coverage, relatively small databases, and a significant risk of zero results. Under those conditions, extensive synonym expansion was often both rational and necessary.

Contemporary search environments differ markedly. Full-text indexing is now widespread, collections routinely contain hundreds of millions of records, and stemming or related linguistic normalisation techniques handle many lexical variants automatically. In such contexts, straightforward keyword searches already tend to achieve acceptable recall and the focus is on precision. Adding long lists of OR-connected terms therefore often produces an explosion of incidental matches. A monograph that mentions “cardiac event” once in passing on page 247 may enter the result set alongside a systematic review centrally concerned with the topic7. Even in systems without full-text search, the average query now tends to retrieve far more material than in the past, which strengthens the case for improved ranking rather than more aggressive expansion.

Argument 2 : LLM’s today add poor synonyms that create more noise and over-stress outdated relevance ranking systems

LLM-to-Boolean systems amplify this longstanding problem. Worse still, inspection of Boolean strings generated by current systems, especially those relying on lower-cost bundled models such as GPT-4o mini in products such as Primo Research Assistant, suggests that the generated synonyms are often of uneven quality. They may include loosely related terms, overly broad variants, or, in some cases, plainly inappropriate substitutions. Each additional weak term expands the candidate set while simultaneously increasing the proportion of noise within it.

The central mechanism is straightforward. Query expansion may assist strict Boolean retrieval by increasing the number of documents admitted into the result set, but it can also impair relevance ranking under models such as TF-IDF and BM25 by diluting the discriminative value of the user’s original terms. Once a query is expanded from a small number of strong keywords into multiple concepts populated by weaker synonyms, the ranking function may assign undue weight to documents matching several marginal variants. As a result, documents that align only loosely with the user’s actual intent may be ranked above those that match it more precisely. Because many library discovery systems combine Boolean retrieval with TF-IDF or BM25-style ranking, LLM-generated synonym expansion can improve one stage of the retrieval process, at least in the narrow sense of increasing potential recall, while simultaneously degrading another. In the absence of a sufficiently strong reranking layer, the result is not simply a larger result set, but a larger and less coherent one.

Argument 3 : There are search resistant concepts where it is better NOT to represent in the Boolean Search

There is also a further complication. Sometimes, somewhat counter-intuitively, trying to represent hard-to-search concepts, what Farhad Shokraneh calls search-resistant concepts, is a mistake. The reason is simple: if you do not know the right terms, forcing them into the query can exclude many relevant papers. In some cases, trying to be more precise actually makes the search worse. A good example is PICO-based searching. Outcome terms are often expressed in highly variable, non-standardised ways, so it is often better not to filter for outcomes at all, and instead accept a larger set for later screening. Do LLMs that expand searches recognise this distinction?

While evidence synthesis librarians are more willing to look through 100% of results in a much bigger set of results, most other users will not look through every result and here again the potential of a much stronger ranker is where most of the value “AI search” brings.

PubMed: what good query expansion looks like

Compare LLM-generated synonyms to a system that does query expansion well: PubMed. PubMed’s Automatic Term Mapping (ATM) also expands user queries with additional terms, but the quality difference is vast8.

Among other things, ATM maps user terms to MeSH (Medical Subject Headings) — a carefully curated, hierarchically structured controlled vocabulary maintained by domain experts over decades. When ATM expands "heart attack" to include "myocardial infarction," it draws on a tested thesaurus. When an LLM expands "heart attack" to include "cardiac event," "chest pain," and "cardiovascular incident," it generates plausible-sounding terms with no guarantee they map to the same concept.

PubMed also has architectural advantages. Its users often prioritise recall over precision, making aggressive expansion a more defensible trade-off. It searches citation and abstract records plus MeSH metadata but not full text, making every keyword match less likely to be incidental. It covers a well-defined domain (life sciences), reducing the risk that added terms have multiple unrelated meanings.

Most critically, PubMed has a superior ranking system. It uses a two-stage architecture with LambdaMART reranking on the top 500 results — far more sophisticated than generic BM25. So even when ATM broadens the result set, the reranking step ensures the most relevant documents rise to the top. The expansion and the reranking work together as a system. This is precisely the architecture most library databases search lack.

What does the empirical evidence say?

Among the systems which use LLM to convert input to Boolean, Primo Research Assistant is probably the most well studied.

One of the more rigorous studies is Galbreath et al. (2025), which evaluated Ex Libris’ Primo Research Assistant (PRA), which at the time of study was using GPT-3.5 to convert natural language into Boolean — against Boolean searches crafted by instruction librarians at Washington State University.

The headline finding: no appreciable difference in topical relevance. PRA returned relevant sources 46.3 per cent of the time; librarian-built searches returned 45.6 per cent. I was mildly surprised at first at the results. But this result needs careful reading.

For each query, Primo Research Assistant generates ten Boolean variant strings runs the search and the top five are then sent to the LLM for summary generation.

Because Primo Research Assistant only generates answers using retrieval augmented generation based on the top 5 ranked results, it is critical the relevant results make it to the top 5 which requires a very strong relevance ranking system.

Perhaps this is why PRA does more than pure LLM-to-Boolean, but adds an additional reranking step beyond the usual Primo relevance ranking. It retrieves the top thirty matching records using the usual method and then reranks them using vector embeddings (semantic ranking).

Even though reranking the top 30 still is a relatively modest effort (many state of art systems will rerank the top 100!), and the semantic reranking will still fail if the conventional ranking fails to get relevant results in the top 30, it still helps a lot.

You can actually roughly compare how results in the top 5 would look like without the reranking steps9. I have done some comparisons and while I am not the most impressed with Primo Research Assistant relevancy ranking even with the reranking step , the results would be far worse without it!

The other issue is the sample of queries used. The study used zero-result patron queries — queries where users had typed natural language into Primo and received nothing back. These are precisely the cases where LLM-to-Boolean provides the clearest benefit. The results might look very different for the much larger population of queries that already return results.

More importantly, I have a strong suspicion, the queries tested were mostly “easy” queries with many potential relevant matches. This is most telling with the lack of overlap in results between the items retrieved from librarian searches and PRA searches.

Despite both sets achieving roughly 50% precision10, only 7.21 per cent of PRA's citations matched the expert searches' results. Clearly, there were many relevant results and PRA was not finding better results; it was finding different results of comparable quality.

In short, the queries were not “skill-testing”, harder queries with a much smaller and limited set of relevant results would no doubt stress Primo Research Assistant even more and this is where I see systems with more powerful relevancy ranking capabilities that goes beyond simple lexical ranking like Elicit.com, Undermind.ai, Consensus shine.

Not all LLM-to-Boolean tools are equal

The tools I listed at the outset sit on a spectrum, and the differences matter.

At one end, Web of Science Research Assistant (as tested in early 2025) and EBSCOhost NLS are pure LLM-to-Boolean: they generate a Boolean query and run it against the standard index with no changes to ranking. The homebrew implementations from Stony Brook and San Diego State work similarly11.

In the middle, as we have seen - Primo Research Assistant generates Boolean, retrieves the top 30 results, then reranks them using embeddings. Web of Science Smart Search runs Boolean and semantic search together and lets users toggle between combined, Boolean-only, and semantic-only results. (Do not confuse Web of Science Smart Search with Web of Science Research Assistant — the former is bundled with no RAG; the latter is a paid add-on with RAG.)

At the other end, Scopus AI describes a hybrid architecture in which it may use vector search, keyword search, or both depending on the query, combining and reranking the results.

Once you lay out this spectrum, you can make a prediction: tools that go beyond LLM-to-Boolean alone — those adding hybrid search, dense embedding reranking — should typically outperform those that do not and be favoured by serious researchers. And roughly speaking, this is what I found in my informal comparative testing and in various Katina reviews.

So far, we have being discussing “quick search”, where results are returned with very little delay (e.g. < 1 min). Agentic/deep search, which uses the LLMs to do iterative searching and evaluation gives you much superior results but you will have to wait much longer (e.g. >10 minutes) for the results. I dub this the difference between Deep Search/Research tools vs “Quick Search”.

The LLM-to-Boolean approach does have one genuine strength: interpretability. When the system displays the generated Boolean, you can inspect it, critique it, modify it, and rerun it. That transparency in the retrieval stage is worth acknowledging. But it comes at the cost of reproducibility: because an LLM generates the query, the search strategy can change each time you run it. In my testing, Web of Science Research Assistant and Primo Research Assistant generated different Boolean strings roughly one in five times; Scopus AI changed nearly every other time.

Why our profession's focus on Boolean retrieval is understandable but limiting

Library training and practice have always centred on the retrieval side of search — constructing the right query to get the right documents into the result set. That is what we teach, what we are trained in, and where our expertise lies. Ranking — what happens after retrieval — has historically been the vendor’s domain, largely invisible to us and outside our direct control.

This is not a failing of individual librarians. It is a structural feature of how the profession developed. When library catalogues used strict Boolean with no relevance ranking at all — results sorted by accession number or date — retrieval was genuinely the whole game. Getting documents into the result set was the only thing you could influence. Evidence Synthesis Librarians obviously have the same mindset as they typically intend to screen all the retrieved results. The habits, pedagogy, and professional identity that developed around this reality are deeply embedded and entirely rational given that history.

But the search environment has changed. Users now routinely face thousands of results and scan only the first few pages. In that context, ranking determines what users actually encounter. The professional focus on retrieval, while understandable, has become a blind spot — and it shapes which AI innovations we notice and which we overlook.

LLM-to-Boolean is visible, inspectable, and maps directly onto existing expertise. It fits neatly into teaching practices and workflows. It is unsurprising that it has generated more enthusiasm than semantic reranking, which operates invisibly beneath the interface. But we need to be honest that familiarity is not the same as effectiveness. Our experience with Google and Google Scholar teaches us that users overwhelmingly judge search engines by the quality of what appears on the first page, not by whether they can inspect the query logic. A search you can fully explain but that returns mediocre results will be rejected in favour of one that is harder to explain but returns excellent results.

To be fair, since I started in the profession in 2007, I have seen a lesser focus on teaching Boolean by librarians, but the rise of “AI search” may accidently push this back to the agenda….

Vendors have their own motivation to create the narrative LLM to Boolean = AI search

Existing database vendors, for their part, have their own reasons for staying with lexical methods and they have to do with cost. Boolean retrieval over inverted indexes is fast, well understood, and already built. The infrastructure exists and is paid for. Bolting an LLM onto the front end to generate Boolean queries requires no changes to the underlying index, no new data pipelines, and no reindexing of billions of records. It is, from an engineering and business perspective, the cheapest possible way so say you are doing "AI", especially if you bolt on the cheapest LLM you can find12.

Moving to dense embeddings is a different proposition entirely. It requires generating vector representations for every document in the index13. At the scale of a system like Ex Libris CDI, with over five billion records, that is a significant computational investment — not just a one-off cost but an ongoing one, since embeddings need to be regenerated whenever the model is updated or new documents are ingested. Dense retrieval is also slower than inverted index lookup at query time, particularly without heavy optimisation of approximate nearest neighbour search infrastructure.

Cross-encoder reranking or using LLMs directly for ranking, the most powerful option for second-stage ranking, is more expensive still. A cross-encoder scores each query-document pair individually, which means running the model once for every candidate document. Even reranking just the top 500 results per query, at the scale of millions of daily searches, adds up. Vendors looking at these numbers can easily conclude that the return on investment is uncertain — especially when librarians are not asking for it and seem satisfied with LLM-to-Boolean.

The next level beyond these methods would be employing agentic/deep search, which uses the LLMs to do iterative searching and evaluation - you get much superior results but at the cost of latency14.

This creates a reinforcing cycle. Librarians ask for LLM to Boolean retrieval because that is where their expertise lies. Legacy Vendors invest in LLM-to-Boolean because it is cheap to implement and aligns with what librarians are asking for. Neither side pushes for the harder, more expensive, but ultimately more impactful investment in ranking. The result is that library search systems remain stuck with decades-old ranking technology while vendors market a superficial AI layer on top as though it were transformative.

What should AI search do?

Let me suggest two approaches

1.The more conservative approach to maintain interpretabilty

Keep Boolean retrieval for the first stage. Inverted indexes and Boolean logic remain the right tool for scalable initial retrieval. There is no need to abandon this.

Add modern reranking as a second stage. After Boolean retrieval and BM25 produce a candidate set, apply a “neural” reranker — whether a bi-encoder, cross-encoder, or LLM-based ranker — to re-sort those candidates before presenting results. This is not radical. Semantic Scholar uses LightGBM reranking that emphasises title matches and highly-cited recent papers. PubMed uses LambdaMART on the top 500 results. Both are academic search engines at massive scale, and both concluded that BM25 alone was insufficient. Even these comparatively modest reranking techniques, now almost a generation behind the state of the art, produce noticeably better results than BM25 for ranking.

Some librarians may object that these new usually neural reranking methods are not as interpretable as lexical relevance ranking based on matching query terms. This concern is overblown, for a simple reason: the current system is not that interpretable either15.

The Ex Libris “Search and Ranking in CDI” documentation describes a relevance algorithm built on a “continuously tuned, proprietary” mix of dynamic rank factors, static rank factors, field boosting, personalised ranking by discipline, and various other components. I would wager that very few librarians who use Primo could explain in detail how CDI’s current ranking works. We have already accepted opaque ranking (think Google since 2000) — we just have not acknowledged it16.

In return, with Boolean still as a first stage retriever, you get exact hit counts and don’t run into the problem of hit counts becoming approximate if you use semantic search methods as the first stage retriever.

2. Giving up interpretability for effectiveness

There is a stronger version of the argument I have been making that I want to put on the table: perhaps we should stop treating Boolean as the obvious choice for first-stage retrieval at all.

Throughout this post I have advocated a two-stage architecture that keeps Boolean retrieval as the first stage and adds modern reranking as the second. That is a pragmatic position — it preserves what vendors and librarians are comfortable with while addressing the ranking bottleneck. But pragmatism can obscure a deeper problem. Boolean retrieval is not just outdated at the ranking stage. It is limiting at the retrieval stage too.

The fundamental issue is that Boolean is binary. A document either matches the query logic or it is excluded entirely. There is no middle ground. If a highly relevant paper uses none of your specified terms or synonyms — because the authors used different terminology, because the concept is expressed implicitly, or because the relevant discussion appears in a section your metadata fields do not cover — Boolean will never surface it. It does not matter how good your ranking is downstream. A document that never enters the candidate set cannot be ranked.

Having worked with researchers, I can tell you even in areas they know well, they tend to struggle with overly including terms, leading them to put in keywords that end up excluding even known relevant gold standard papers they give me!

The reason for this is that most researchers and even librarians are unfamiliar with what Farhad Shokraneh calls “search resistant concepts” which he defines as “concepts that when added to a the search, are more likely to miss the relevant records.” In such situations, if you want to ensure high recall, you should not even try to search for the concept!

He gives three reasons for why concepts are hard to search but the lack of standardised terminology used by the field is the clearest reason.

In evidence synthesis scenarios, search resistant concepts typically are outcome concepts (from PICO) which are described in so many different ways, such that it is often better not to even try to search for them. Do LLMs know this?

This is precisely why LLM-to-Boolean tools resort to aggressive synonym expansion in the first place. The rigidity of Boolean matching creates a constant risk of missing relevant documents, so the system compensates by throwing in every plausible variant. But as I have argued, that expansion then degrades BM25 ranking by flooding the result set with noise. The root cause of both problems — missed documents and noisy results — is the same: Boolean’s binary matching model.

BM25, used as a first-stage retriever rather than just a ranker, avoids this entirely. BM25 does not hard-exclude documents. It scores every document against the query and returns the top-k by relevance score. A document that matches some but not all of your terms still appears — it is just ranked lower.

This means you do not need exhaustive synonym lists to compensate for rigid matching. A simple, well-chosen set of keywords will retrieve a broad and relevant candidate set because partial matches are included rather than discarded.

BM25 as first-stage retriever versus Boolean: what does this actually mean?

Using BM25 as a first-stage retriever rather than Boolean can be hard to grasp if you are used to Boolean as the default, so here is a concrete example.

When you type open access citation advantage into a strict Boolean search engine, most systems today applies an implied AND. Every result must contain all four terms. If a document contains “open access” and “citation” but not “advantage,” it is excluded entirely — regardless of how relevant it might be.

A BM25-based retriever works differently. It scores every document based on how well it matches the query terms, weighing term frequency and rarity, but it does not hard-exclude documents that are missing a term. A system using BM25 for first-stage retrieval — as Google does, or Scite.ai — might return a document matching only three of your four terms if its overall relevance score is high enough17. The missing term potentially lowers the score vs documents with all four terms but does not eliminate the document from consideration18.

In the information retrieval word, this is still lexical/keyword based as the retrieve and ranking is still just based on term matching19.

This matters because strict Boolean can accidentally drop relevant documents through no fault of the searcher. In my review of EBSCOhost Natural Language Search (NLS), I showed an example where I asked the system to find papers that used randomised controlled trials to test for an open access citation advantage. NLS generated a Boolean query and ran it against the index. It failed to surface one of the few obviously relevant papers, despite that paper being indexed in the database. The reason was straightforward: this early paper did not use the phrase “open access citation advantage,” and the Boolean that NLS constructed did not expand broadly enough to capture the terminology the paper did use. Under strict Boolean, that single vocabulary mismatch was fatal — the paper was excluded from the result set entirely.

A BM25 retriever would not have automatically excluded that paper. It would have scored it lower for missing some query terms, but the paper would still have entered the candidate set. With a strong reranker as a second stage — one capable of recognising that the paper is conceptually about the same topic even without exact term matches — that paper could be pushed to the top of the results. Semantic search approaches would handle this even more naturally, since dense embeddings capture conceptual similarity rather than relying on shared vocabulary at all.

This is the core limitation of Boolean as a retrieval method. It treats term matching as a gate: you are in or you are out. Every term that is missing from the query or absent from the document is a potential point of failure. BM25 and semantic methods treat term matching as a signal — one input among many into a relevance score — which is far more forgiving and far less dependent on the searcher anticipating every possible way an author might express a concept.

The aggressive query expansion that causes so many problems under Boolean becomes largely unnecessary. Add a strong reranker on top of first stage BM25 is what all modern post-2019 information retrieval papers use as a strong baseline comparison.

But if we are willing to give up strict Boolean as first stage, why not go further with either using Semantic search methods alone or as another component at the first stage. Dense embedding retrieval does not depend on term matching at all. It represents both queries and documents as vectors in a shared semantic space and retrieves documents by conceptual similarity. A paper about “myocardial infarction” can be retrieved by a query about “heart attack” without anyone — human or LLM — needing to specify that synonym. The vocabulary mismatch problem that has driven decades of query expansion work in library science is addressed at the architectural level rather than patched at the query level.

The next level beyond semantic search would be agentic/deep search, which uses the LLM to do iterative searching and evaluation - often using both lexical and semantic search methods

The practical objections are real but not insurmountable. Librarians value Boolean as a first stage retriever because it is predictable, inspectable, and gives them direct control over what enters the result set. Abandoning it at the first stage means accepting that retrieval becomes probabilistic — you can no longer guarantee that a specific document will or will not appear for a given query. Hit counts become approximate or meaningless. The ability to teach students a logical, reproducible search process is diminished.

These are genuine losses. But we should weigh them against what is gained. The vocabulary mismatch problem — the single largest source of recall failure in academic search — is substantially mitigated. The need for aggressive synonym expansion, and all the ranking noise it creates, is eliminated. And the overall architecture becomes simpler: instead of Boolean retrieval followed by BM25 ranking followed by (hopefully) neural reranking, you can move to a hybrid first-stage retrieval using both BM25 and dense embeddings, followed by a single reranking step.

Most of the search industry reached this conclusion two decades ago. Google, Bing, and virtually every major web search engine abandoned strict Boolean for first-stage retrieval by the early 2000s going for BM25. Among academic search systems, Semantic Scholar uses BM25 rather than Boolean as its first-stage retriever. The library world remains one of the last holdouts — not because Boolean is technically superior, but because our professional training, our teaching practices, and our vendor relationships are all built around it.

I am not suggesting this transition will be easy or that Boolean should vanish overnight. A hybrid approach — using both lexical methods like BM25 and semantic methods alongside or instead of strict Boolean — is the most realistic path. Scopus AI and top class tools are already moving in this direction. But we should be clear-eyed about what we are defending when we insist on Boolean as the foundation. We are defending a retrieval method whose core limitation — binary matching — is the root cause of many of the problems this post has described.

Buy me coffee (via ko-fi)

The bottom line

LLM-to-Boolean is not worthless. But it is a minor component being marketed as the whole solution. It solves a narrow problem — helping novices who cannot construct any Boolean at all — while doing nothing for the majority who need better-ranked results. As a first-stage query construction aid paired with modern reranking, it could play a useful supporting role. On its own, with the same BM25 ranking underneath, it changes almost nothing that matters.

The horseless carriage eventually gave way to the automobile — a machine designed from the ground up around the capabilities of the engine, not around the form of what came before. Library search needs the same transition. The real bottleneck is not retrieval — it is ranking. And for complex queries, it is not even ranking — it is the absence of iterative, evaluative search processes that can reason about what they find.

The infrastructure to do better already exists. Semantic Scholar and PubMed have proven that two-stage architectures work at scale. Agentic deep search tools have demonstrated what is possible when LLMs are used as reasoning engines rather than Boolean generators. Some libraries are already taking this path — Harvard Library, for instance, bypassed LLM-to-Boolean entirely for their special collections, building a discovery platform called Collections Explorer that relies on embedding models.

The question is whether library vendors will invest in these approaches, and whether we as a profession will demand they do — or whether we will keep admiring the horseless carriage because we can see how the reins work.

When I last wrote about MCP servers in the context of the launch of Wiley’s AI Gateway and the PubMed MCP server/connector you could add to Claude.ai, I received a few private messages from librarians and researchers who shared how much more they had gone beyond just connecting to pre-built MCP servers.

Most of them were using Claude Code or Claude Code Desktop, which they used to:

a) vibe-code MCP servers that did not exist or were not offered, and run the servers locally

b) combine MCP servers and other tools with Claude Skills for powerful workflows

Since then, I started exploring the use of Claude Code and Claude Code Desktop myself, and while I am still a rank novice in this area, I can see why people are so excited by the possibilities.

But before I walk through the practicalities, I want to make the case for why this matters — because it is not just about convenience. It represents an almost fundamentally different model for how academic search and workflows in general can work.

Buy me coffee (via ko-fi)

In my recent post on how “agentic” current academic deep research tools really are, I found that while specialised deep research tools like Consensus Deep Search and Undermind, Elicit , etc. , perform impressively within their designed parameters, but their agency is ultimately bounded by rigid, pre-programmed workflows. They execute complex, multi-step loops rapidly and cheaply, but the moment a research query steps outside those hardcoded tracks — requiring a novel combination of tools and capabilities such as filters, unconventional citation tree traversal, or data extraction the vendor did not anticipate — these fixed pathways become bottlenecks.

This highlights a fundamental tension: as frontier models become increasingly capable of advanced reasoning, forcing them through restrictive, “on-rails” product interfaces artificially caps their utility. These tools treat information retrieval as a static funnel rather than a dynamic, iterative process, leaving the researcher constrained by the vendor’s imagination.

LLMs connected with MCP servers change this. Instead of a vendor-defined loop, it allows researchers to build custom, transparent, and extensible research environments. The AI is not limited to searching a single closed index.

For example, it can reach into a local Zotero library, query live institutional repositories, chain outputs from one search system into another, and execute code to analyse data on the fly.

The agent moves from being a search tool to something closer to a real research assistant that can have autonomy to “decide” which tools to use and in which order and/or an assistant that the researcher can instruct via Claude Skills.

The rest of this post lays out the evidence that this is not just a theoretical advantage, walks through how to actually set it up, and addresses the current limitations of such a setup and implications for librarians.

Subscribe now

Can generic LLMs actually compete on retrieval?

The obvious objection is: surely purpose-built academic search tools such as Elicit.com, Undermind.ai, Consensus retrieve better results than a general-purpose LLMs? The evidence is currently more mixed than you might expect.

A video study by Moara claimed Claude's web search (with no special academic connectors) beat Undermind, Elicit, SciSpace, Consensus, and Scholar Labs on retrieval quality.

This study has significant methodological problems: it tested only a single query and looked only at the top 10 results, the evaluation relied mostly on LLM-as-judge (ChatGPT, Gemini and Claude may be biased towards output from fellow LLMs despite attempts to anonymize), and the query may have favoured grey literature that specialised tools cannot access.

That said, the human evaluators agreed with the ranking, and it is notable that Undermind — which I generally regard as among the strongest specialised tools — came second (See also earlier study by Moara that ranked Undermind first among all specialised tools).

I have found similar patterns in my own testing, though I stress these are illustrative examples rather than systematic evaluations. When asked whether OpenAlex can be used alone for systematic reviews, Undermind found two relevant items. Claude (without any specialised MCP server) found the same two, plus a third relevant grey literature report that was not in Semantic Scholar’s index.

Most striking was how Consensus Deep Search failed to find any relevant papers on this query, while Claude even with the Consensus MCP server (which is limited to just the top 3 results) did find relevant papers. This is a peculiar case where Consensus’s deep search mode, spending far more time and compute, produced worse results than its own index queried through a basic MCP connection.

A more systematic comparison comes from a librarian who tested 11 queries across multiple tools. Using a scoring rubric, Claude (again without any special MCP servers) ended up on top ahead of Consensus, Elicit, and Primo Research Assistant, though not by a statistically significant margin.

Part of the reason for this is that some queries were best answered by non-journal content — monographs and grey literature — where Claude and Primo (which draws on broader library collections) had a structural advantage over tools that rely primarily on OpenAlex or Semantic Scholar.

None of this constitutes definitive evidence that generic LLMs are superior. The sample sizes are small, the methodologies vary, and important details are often unclear (for instance, it is not always specified whether deep search modes were used for the specialised tools, or whether Claude was running in its more powerful “Research” mode). What the evidence does suggest is that generic LLMs are not clearly worse at finding relevant sources, particularly for queries that benefit from content outside journal & preprint article indexes.

To be clear, I am referring to the ability to find relevant sources. Specialised tools offer other affordances not offered by using generic LLMs. This includes structured output with academic references, useful visualizations and methodological transparency (sometimes). But on raw retrieval, the gap has narrowed considerably.

This is why I now recommend users combine a specialised academic deep research tool (such as Undermind, Consensus, or Elicit in their deep modes) with a generic deep research tool (such as OpenAI or Gemini Deep Research, or Claude with Research mode) for greater coverage.

This is particularly important the more humanities-oriented or less journal-centric your subject area is, as the specialised tools tend to rely on OpenAlex and Semantic Scholar, which have weaker coverage in these areas.

What are MCP servers and why do they matter here?

If generic LLMs are already competitive at retrieval , the obvious next question is: how do you make this reliable rather than hit-or-miss?

If indeed, general web-backed LLM search can match specialized tools on top-N relevance for some query types (especially those requiring grey literature or books), could we do even better with access to specialised academic content?

The problem with relying on general web search for academic work is that even when explicitly instructed to find peer-reviewed sources, LLMs will still sometimes cite blog posts, news articles, and other non-academic content. They draw from the open web and do not always distinguish between a journal article and a well-written blog post on the same topic. Giving these models a direct connection to academic indexes, rather than hoping they stumble on the right content through web search, is what turns an occasionally useful capability into a dependable one.

This is exactly what the Model Context Protocol enables. MCP is an open standard, supported to some extent by all three major AI labs (Anthropic, OpenAI, and Google), that allows LLMs to connect to external tools and data sources. Give a model direct access to PubMed, Scopus, or your institutional repository, and it no longer has to guess whether what it found on the web is actually a scholarly source.

And if you can give it access to paywalled full text via publisher MCP servers, such as Wiley’s Gateway AI server, you could potentially give it an advantage over specialised tools like Undermind that rely only on open corpora like Semantic Scholar.

There is already considerable buzz around setting up MCP servers for library-specific resources — the library catalogue, discovery services, institutional repositories, and more. This CNI talk is an excellent example of what leading libraries like Northeastern are currently doing in this space.

The reasoning is straightforward. Whilst librarians and some power users reach for specialised deep research tools like Elicit, Undermind, Consensus, and Scite Assistant, most students and researchers spend the bulk of their time in ChatGPT, Gemini, or Claude. So why not meet them where they are?

MCP servers come in two varieties: remote servers, which are hosted by someone else and you simply point your LLM interface to them; and local servers, which you run on your own machine.

A necessary warning : connecting your LLM to MCP servers carries risks. The operators of remote MCP servers can see your queries, and there are security risks including prompt injection. Running local servers from unverified GitHub repositories carries its own risks (since it involves running code on your machine!), though it does improve privacy since queries stay on your machine.

Setting up remote MCP servers

Remote MCP servers are the simplest to get started with. In Claude.ai, you can find Consensus, PubMed, and Scholar Gateway listed as connectors that are available that can be added directly. Scite.ai has to be manually added.

Support for MCP servers is more complicated with ChatGPT.com. While ChatGPT recently started supporting MCP servers through its apps system, it does not yet list any academic search connectors in its store at the time of writing. You can work around this, though it requires a paid account and a somewhat hacky process:

1. Go to Settings, then Apps, then Advanced settings, and enable Developer mode.

2. Open ChatGPT Apps settings and click “Create app” next to Advanced settings.

3. Create an app for your remote MCP server using the server URL.

The MCP server URLs that work without any authentication are:

• Consensus: https://mcp.consensus.app/mcp (documentation)

• PubMed: https://pubmed.mcp.claude.com/mcp (hosted by Anthropic as a connector for Claude, this is the same connector I tested in my earlier blog post)

• Scite.ai: https://api.scite.ai/mcp (documentation)

Edit : As of 28 Feb, Scite is now listed in ChatGPT store, so you no longer need to turn on developer mode. Scispace.com has followed suit as well.

Edit : As of 29 April, Scite is now listed in list of Claude supported Connectors, so you can just add directly as well.

Once set up, you can select these tools for use in your conversations or let ChatGPT decide when to invoke them.

Below you can see the tool calling and searching Scite.ai

Edit : As of 25 March , Consensus is now listed in Claude Connectors (with ChatGPT coming soon), so you no longer need to create a custom MCP entry. It also automatically authenticates with your Consensus account , which will then give you access to the top 5 results.

Some limitations to be aware of: the Consensus MCP server returns only the top 3 results out of 20 by default (though you can request for a API key). The Scite MCP server provides access to open access citation data and metadata by default, though they note that institutional subscriptions can unlock additional content. It is worth reading the documentation pages for each server to understand their full capabilities (e.g. search parameters, citation searching capabilities).

Setting up local MCP servers

Remote servers are limited to what is available. Code for running Semantic Scholar or OpenAlex MCP servers exists on GitHub, but nobody hosts these as free remote servers.

And if the MCP server you want does not exist at all, you can attempt to vibe-code one yourself (assuming an API exists for the source you want to connect to) and run it locally.

Warning : Again there are risks in running unverified MCP server code you find on GitHub and the ones you vibe code might have security vulnerabilities that can be exploited with prompt injection.

Many early adopters are using Claude Code , Claude Desktop or Claude Cowork for this, both of which support local MCP servers. In my own Claude Desktop setup, I have configured and added several local MCP servers (with code I found on GitHub) alongside the remote ones:

• A Zotero MCP server that connects Claude to my local reference library

• A Rust Research MCP server that connects to multiple academic search systems including Semantic Scholar, OpenAlex, Crossref etc.

• A custom OpenAlex MCP server that I had Claude Opus 4.6 vibe-code using the OpenAlex API as an experiment.

If you are not comfortable with Python or coding in general, the installation instructions for these servers may look intimidating. But you can use Claude Code itself and ask it to install these packages for you!

The real advantage: combining tools through Skills

The retrieval comparisons above are interesting, but they miss the more fundamental point. The real advantage of the MCP approach is not that it searches better — it is that it enables workflows that no single vendor anticipated.

Here is a simple but concrete example: I asked Claude to search PubMed for papers on a topic, take the top 5 ranked results, and then use the Scite MCP server to retrieve mentioning and contrasting citations for each.

See also in my last blog post with my test of “Find me papers that could/should have been cited by Paper ‘The state of OA: a large-scale analysis of the prevalence and impact of Open Access articles’ but were not.”

Claude successfully chained or “orchestrated” both tools together to produce exactly what I wanted1.

This is a trivial example and you get more fancy tasks where no single existing tool offers the specific workflow you want. With Claude+MCP, I did not need to wait for a vendor to build it.

This composability extends further through Claude Skills, which are essentially predetermined prompts that Claude uses when triggered.

For example, you can define a “literature review” skill that searches multiple MCP servers (OpenAlex, Semantic Scholar, general web and your Zotero library), combines the results and writes a synthesis.

Skills are written in markdown and defined in natural language, so creating and modifying them requires no coding ability. Claude can even guide you through creating a skill by asking what you want it to do.

The remaining steps should be self-explanatory

You could easily modify such a skill to only show results found by search that is not already in your Zotero library, or to create an entirely different search workflow. For a more advanced example, see Alfred Wallace’s skill design, which is inspired by work on guiding agentic retrievers on when to search broadly versus when to explore locally. More advanced usage can involve adding resources and scripts for the skill to use.

The key insight is that everything is defined in natural language, so it is easy to change. The researcher defines the workflow, not the vendor.

Limitations and open questions

I’m only scratching the surface here. I’ve been hearing about the incredible workflows researchers and librarians are building by combining these tools. In essence, the process is now this simple:

• First: If you can access a source via API, you can ‘vibe code’ an MCP server to talk to it.

• Second: You create a skill in natural language to trigger that server and mash it up with anything else.

• Third: There is no third step.

In theory, you can set up a skill or flow to do

• a cochrane type systematic review flow that involving coming up with inclusion criteria, searching and downloading from multiple sources, use the LLM to Screen etc.

• searching for papers to do bibliometric analysis using a tool like pybibx,

• comparison of the citations made to 2 different papers in the same area

• some gap analysis etc.

The only academic related platform that can match the capabilities of this setup is SciSpace Agents that allows you to hook and connect a huge variety of premade tools.

SciSpace Agents as an alternative is far less daunting technically to use but as you can expect, it is costly.

Still there are limitations

The technical barrier

Setting up remote MCP servers is straightforward. Configuring local servers, vibe-coding custom ones, writing effective Skills, and ensuring secure operations requires a level of technical comfort that most researchers and many librarians do not currently have. This is not a minor caveat — it is a significant barrier to adoption.

Privacy and security

I flagged this above but it deserves emphasis. For researchers working on sensitive topics — anything from commercial R&D to clinical research involving patient data — queries sent to remote MCP servers are visible to their operators. This is not a theoretical concern. Libraries routinely fight to protect patron privacy, and recommending MCP server setups without clearly communicating this risk would be irresponsible.

Reproducibility and interpretability

Every user’s MCP environment is different — different servers installed, different Skills configured, different model versions. If I run a literature search through my Claude Desktop setup and you run the same query through yours, we will likely get different results even with the same query.

Ironically, the fixed workflows of specialised tools, which I criticised above for limiting flexibility, do at least provide a more consistent and documentable search process. There is a genuine tension here: the flexibility that makes the MCP approach powerful is the same property that makes it difficult to reproduce. Perhaps future practice would require submitting your entire Claude Code setup as part of a systematic review’s methodology, but we are far from established norms on this.

Cost

Claude Code and Claude Desktop with heavy tool use consume tokens rapidly. For a library or research group considering whether to invest in configuring MCP environments versus subscribing to Undermind or Consensus, the cost comparison matters. Specialised tools offer flat-rate subscriptions with predictable costs; the MCP approach scales with usage in ways that can be difficult to forecast.

However with techniques like clever use of Claude Skills to load only necessary context instead of all MCP servers, Server-side context compaction for managing long conversations that approach context window limits & code execution where LLMs write code to call tools instead of direct tool calls, this problem may be reduced.

Implications for libraries

Despite these limitations, the direction of travel seems clear, and it has significant implications for libraries.

Alfred Wallace, a librarian who is perhaps a leader in exploring what is possible with MCP in research contexts, has been experimenting with building comprehensive research agent setups

He has raised the provocative idea of “AI tool fine-tuning as a service” from libraries. The concept is that librarians, with their understanding of information retrieval principles and research methodology, are well-positioned to configure these environments for researchers — selecting appropriate MCP servers, writing effective Skills, and advising on the limitations of different configurations.

In particular, he asks very deep and insightful questions about the setting up, maintenance and control of MCP servers.

There is also institutional-level activity worth watching. Clarivate's Nexus project, while easy to dismiss as merely a browser extension comparable to LibKey Nomad or Lean Library, appears to be a more ambitious play.

Their vision seems to be for Nexus to serve as an aggregation point for library-related MCP servers, not just Clarivate's own. For example, Springshare's LibGuides was mentioned at CNI as an example of other vendor MCPs that could be integrated.

This is a clever and strategic role that Clarivate wants to play, as an an intermediary/gatekeeping role, though it’s unclear if other vendors would allow such a move. This could also compete with a model where libraries assemble their own bundles of Skills and MCP servers.

The question of full-text access also deserves attention. Alfred suggests a simple approach: configure the LLM's instructions to prepend a LibKey prefix to any DOI it retrieves, so that links eventually resolve through the library's access infrastructure. This works reasonably well, with one notable issue: LibKey does not handle non-Crossref DOIs, and the most common non-Crossref DOIs tend to be arXiv papers (which use DataCite DOIs) that are freely accessible anyway. This can confuse users. You can prompt the LLM to handle these exceptions, but it consumes tokens for marginal benefit.

Better solutions might start to emerge. e.g LibKey itself is reportedly launching an MCP server, and protocols like GetFTR have been positioning themselves as a way for AI agents to access full text through their own MCP servers.

The issue here I suspect isn’t technology but business model and usage tracking. Since the rise of chatgpt, publishers have been worried about what is now called “zero-click search” leading to “zero-click results”, will they really allow agentic LLMs to so easily consume full text?

Where this is heading

The general trajectory of frontier LLMs is towards increasingly capable agentic search behaviour. BrowseComp, the most well-known benchmark for agentic web search, has seen top models (Claude and Gemini) improve from barely over 50% in April 2025 to over 80%2 now, in less than a year.

OpenAI has tried building app ecosystems multiple times — ChatGPT plugins in 2023, CustomGPTs, and now apps — and the first two attempts largely gained little traction. The difference this time is that MCP is not an OpenAI-specific implementation but an open standard supported across all three major AI labs.

For academic search specifically, I think the practical recommendation is clear: use both specialised tools and general LLMs with academic MCP connections. The specialised tools offer structured workflows and purpose-built features for specific research tasks. The general LLMs with MCP offer flexibility, broader content coverage, and the ability to create custom workflows. They complement rather than replace each other.

The deeper question — whether the future of research tooling lies in polished, vendor-defined products or in researcher-configured composable environments — remains open. My sense is that we are heading towards a middle ground: vendor tools that expose MCP interfaces, allowing them to be composed in ways the vendors did not anticipate, combined with institutional configurations (possibly maintained by libraries) that provide sensible defaults while remaining customisable3.

But that is speculation, and the landscape is moving fast enough that confident predictions seem unwise.

Buy me Coffee (via ko-fi)

arXiv recently restricted review article submissions in computer science, requiring journal or conference acceptance before deposit. They noted specifically that the change was driven by an “unmanageable influx” and that LLMs made review/position papers “fast and easy to write,” and that many were “little more than annotated bibliographies.”

Separately, arXiv has also tightened its author endorsement policy

It’s hard not to see the shared pressure behind both: scholarly infrastructure is facing a surge of low-quality, high-volume submissions, and AI has made certain genres—especially “survey-like” writing—cheap to produce.

So: are we witnessing the death of the review article?

Galli et al. (2024), writing in Learned Publishing, argue “no” with a useful distinction. They separate descriptive reviews (primarily summarising and aggregating) from reflexive reviews (interpretation, theorisation, agenda-setting). Their claim is that AI will commoditise descriptive reviews, while reflexive reviews, those requiring “human imagination, creativity and abstraction” will retain value.

That framing is plausible. But I think it underestimates something important: even “mere” descriptive reviews are hard to do well, because they depend on two upstream bottlenecks that today’s AI literature tools are still not 100% perfect in yet.

Buy me Coffee via ko-fi

Failure Mode 1: The corpus construction gap (retrieval and exhaustiveness)

Can these tools actually find ALL the relevant papers with clear criteria?

At SMU Libraries, we’ve been early institutional subscribers to Undermind (since 2024). It’s among the earliest and strongest off-the-shelf “deep research” tools we’ve seen, and it’s heavily used and genuinely appreciated by faculty and PhD students.

But when we test it against published systematic review corpora, using the SR’s included-studies list as a “gold standard” target set. Undermind retrieves only ~30–80% of those target papers after running for around 8 minutes.

Systematic reviews we benchmark against are often narrow, with clear inclusion/exclusion criteria and established terminology. If these are “easy mode,” why does retrieval still fall short?

A big part of the answer I believe is architectural and product-driven. Tools like Undermind, Elicit1, and Consensus tend to be tuned for high precision and “reasonable recall,” not exhaustive coverage. They’re designed to help a researcher get to a useful working set quickly—not to support Cochrane/JBI/Campbell-style comprehensiveness.

This aligns with what others (mostly evidence synthesis practitioners) said in response to my LinkedIn post (which this post is built on). John Frechette articulated this “When it comes to retrieving hundreds or thousands of papers on a topic, current tools aren’t really set up for it.” Howard White agreed: “Finding papers is a weak point.”

And this matters because the descriptive/reflexive distinction quietly assumes access to a representative corpus. If your corpus is incomplete in systematic ways, your synthesis (descriptive or reflexive) inherits those blind spots.

To be clear: for many purposes, “good enough” retrieval is good enough. Tools that can cite (almost all) seminal works, surface major strands, and produce coherent overviews in a short amount of time can absolutely save a ton of time when producing reviews.

But for methods where exhaustive recall is the point, current tools are still unreliable. The obvious fix is “throw more compute at it.” But while more time and more queries help, so do less glamorous issues: corpus coverage (including paywalled content), indexing scope, metadata quality, transparent search strategy control, deduplication, and screening workflows.

Even so, if I had to bet, I’d say this first gap is the more solvable one. For example, as of now METR shows a 50% chance of completing a task of over 6 hours, a tool designed just for systematic reviews might be able to close the gap.

The harder gap is the next issue.

Failure Mode 2: The conceptualisation gap (expert judgement starts before retrieval)

The reflexive review problem isn’t only about clever synthesis after you’ve gathered the papers. It often starts earlier.

You need expert judgement to find the papers in the first place.

Over the 1.5 years of our institutional subscription to Undermind, I make it a point to ask the few disappointed users why they don’t like the tool and to collect failed Undermind queries from them.

The failed queries cluster in a very recognizable pattern: broad, often cross-disciplinary questions that require interpretive leaps—connecting subdomains that don’t share vocabulary, or asking for relationships that haven’t been named explicitly in the literature.

Users report that the tools interpret queries narrowly and conservatively. They stay close to established terminology and mainstream frameworks. They struggle to “bridge” conceptual gaps—especially when the question is exploratory (“what research exists?”) rather than confirmatory (“what does this well-defined literature say?”)2.

This shouldn’t be unexpected. These systems are trained to reflect patterns in existing literature, not to make bold interpretative moves. They’re designed to reproduce consensus: great at aggregating what a field already recognises, weaker at proposing new framings or deliberately hunting for minority subdomains and boundary cases.

Neal Haddaway, another expert in evidence synthesis makes a closely related observation from the evidence-synthesis world: in broad evidence maps—where the question is essentially “what research exists?”, AI tends to miss important minority subdomains, not because it can’t describe them, but because it can’t systematically search across diverse topics without collapsing toward the dominant cluster.

“… my experience has been that AI fails for broad evidence maps where the question is “what research exists”, not because AI can’t describe those concepts, but because it can’t systematically search for diverse topics, and tends towards aggregation, missing important minority subdomains.”

The OpenScholar paper (Asai et al., 2025) is interesting too. Nature headlined that experts preferred OpenScholar’s outputs 51–70% of the time over PhD-written answers. But what did it excel at? “Comprehensive coverage, organisation, breadth of sources”—classic descriptive synthesis.. That’s impressive, and genuinely useful. But it doesn’t magically solve the deeper issue: will the system make the right conceptual jumps, notice what’s missing, and generate non-obvious observations that advance the field?

Even if you patiently iterate—rephrase, expand, steer—today’s tools still tend to reproduce the existing shape of the literature rather than challenge it. And for reflexive, field-shaping work, that’s the whole game.

Important to note, most AI literature review tools are using the equalvant of 20 USD Tier models, there are hints that the 200 USD Tier models such as ChatGPT PRO/Gemini DeepThink models can surpass these limits and come up with truly original scientific insights, but we will see in the future when their costs drop enough to be used for off-the-shelf AI literature review tools.

Conclusion

Review articles—descriptive included—won’t die as long as incentives persist and as librarian, Alex Carroll points out - reviews are cited at higher rates than primary research. More importantly, for graduate students, writing a review chapter has long been a pedagogical exercise: you write to build the conceptual scaffolding that lets you do original research.

The question is not merely:

“Can AI write competent reviews?”

It’s:

“What happens to researcher development if AI does the cognitive work that reviews were historically for?”

A well-written review becomes Chapter 1 of a dissertation because writing it forces the author to wrestle with definitions, boundaries, tensions, and absences—exactly the things that later shape novel contributions. If we offload that struggle too early, we may get more polished text and fewer well-formed thinkers.

So yes: increasingly. AI makes descriptive review prose cheap. But judgement—what to include, how to frame, what’s missing, what matters—remains scarce. And in many disciplines, the only reliable way people learn to produce that judgement is still the slow, slightly painful process of doing the review themselves.

Buy me coffee via ko-fi

An earlier version of this argument appeared as LinkedIn post, and this post benefited a lot from reader comments and pushback.

I’ve had a post titled “Common Misconceptions Librarians Have About Information Retrieval” sitting in my drafts for months. I never published it because some points felt like trivial nitpicking (lexical search doesn’t have to be Boolean; overly narrow definitions of “neural search”), and others were eventually covered elsewhere.

But recently I’ve encountered two related ideas that concern me more, because they’re seductive and contain just enough truth to seem plausible—while obscuring where the genuine problems actually lie.

I’m not naming sources—this isn’t about embarrassing anyone—but trust me, these aren’t strawmen arguments I made up to argue with.

Buy me coffee! (via ko-fi)

Misconception 1: “AI search is dangerous because it only gives you what you want”

This argument typically starts with an accurate premise: LLMs like ChatGPT exhibit sycophantic tendencies as result of techniques like RLHF (Reinforcement Learning from Human Feedback)1. From there, it often goes off the rails. Librarians worry that users will be trapped in filter bubbles, with AI tools reinforcing outlandish beliefs and leading them down conspiracy theory rabbit holes.

There are real concerns here but one needs to be clear what exactly we should worry about

The category error

Sycophancy is a conversational behaviour—agreeing with stated positions, excessive hedging, reluctance to correct the user. It’s meaningfully different from retrieval bias, which is what would actually affect search systems.

When you search for documents, the retrieval system isn’t “agreeing” with you. It’s matching your query to content. These are fundamentally different operations. More on that later

The architectural misunderstanding

Many also seem not to realise that conventional AI search is typically search plus generation (now commonly known as Retrieval Augmented Generation), not just an LLM responding from its pretraining.

When someone uses Consensus or Elicit, they’re querying a corpus. The LLM layer summarises retrieved documents; it’s not confabulating from weights. The retrieval step is functionally similar to what a discovery layer does. The generation step is closer functionally to what a reference librarian does when synthesising sources for a patron.

Criticising RAG-based search for “sycophancy” is like criticising a systematic review for only including studies that matched the search strategy2. It’s a category error.

I should note this is most clearly true for simpler RAG architectures. Modern agentic search systems increasingly blur the boundary between retrieval and generation. Tools like Consensus and Undermind now use LLMs to reformulate user queries before retrieval—expanding terms, inferring intent, generating multiple search strategies. Could the LLM interprets your user intent sycophantically at this stage (also see the next section)? I personally think the risk of this is low, because you are using a specialised tool rather than general LLM like ChatGPT, which is not specialised to “knowing” you.

This is not a foregone conclusion and should be worth studying.

Also it has a simple solution: transparency about query reformulation, not abandoning AI-assisted search altogether.

Even pretraining responses mostly reflect corpus consensus

Here’s what’s underappreciated: even pure LLM responses (without retrieval) default to corpus consensus. This is often the opposite of telling users what they want to hear.

Ask a model about vaccines, climate change, or evolution—you’ll get the scientific mainstream regardless of your priors. The model tends to stick to what it knows was most prevalent in its training data.

Sycophancy only kicks in when you state a position and the model defers to it conversationally. That’s a different failure mode entirely, and not one that typically applies to a search query like “what does the evidence say about X.”

LLMs are better described as consensus machines3.

Trained on vast amounts of data, they’re strongly disposed toward mainstream positions. Far from agreeing with fringe views, they often push back aggressively against claims they “know” to be false—sometimes so rigidly that they’ll deny breaking news events because such information conflicts with their training data4.

This has an underexplored downside. If LLMs default to mainstream positions, they may disadvantage emerging research, minority scholarly viewpoints, or fields where “consensus” is contested or disciplinarily biased. A system that systematically privileges well-established views could subtly suppress heterodox scholarship—not through malice, but through training data distributions.

This isn’t the same as the filter-bubble concern (which imagines AI reinforcing individual user biases), but it’s a form of bias worth investigating empirically.

On genuinely contested issues, yes, LLMs may be inclined to agree with the user. But this is addressable through better prompts and system design.

The irony: traditional search has always “given people what they want”

The concern about AI search “giving people what they want” is particularly ironic given that much of traditional search has done this for decades:

• Google’s personalisation literally optimises for engagement

• Academic Citation-based ranking amplifies already-popular work

• Users self-select sources confirming their priors through the databases they choose and the keywords they use

None of this is new with AI. If anything, semantic search that surfaces conceptually related but differently-framed work might reduce confirmation bias compared to keyword matching, which only returns documents using your exact terminology.

That said, Semantic search also might surface conceptually related documents that reinforce a user’s framing rather than challenge it. Whether semantic retrieval increases or decreases exposure to diverse perspectives is an empirical question, not a foregone conclusion in either direction.

Where legitimate concerns actually lie

This isn’t to say there are no concerns worth examining. But they’re empirical questions, not foregone conclusions:

• Summarisation layers might flatten nuance or selectively emphasise certain findings

• Retrieval models trained on user behaviour could preference “satisfying” results over comprehensive ones

• The synthesis step in RAG systems introduces editorial choices

• Query reformulation by LLMs could introduce subtle biases before retrieval even occurs

• Cognitive offloading issues

These are testable hypotheses. We should be investigating them rigorously—and some researchers are beginning to—not assuming the worst based on a misapplied understanding of sycophancy.

Misconception 2: "Friction is good for learning, so AI search shouldn't give you what you ask for"

I’ve heard the suggestion that AI search engines are problematic because they give people what they want.

Think about how strange this sounds. Forget AI—consider traditional Boolean search. If I search for:

vaccines cause autism

and the search engine returns documents containing those words, it’s “giving me what I asked for.” Is that bad?

Semantic search is simply this on steroids. If precise retrieval isn’t problematic for keyword search, why would it be problematic for semantic search?

I should acknowledge that the analogy between Boolean and semantic search isn’t perfect. Boolean search possesses a kind of mechanical indifference: it matches character strings without understanding. Semantic search, by contrast, interprets meaning. It makes invisible judgements about what is “conceptually related” to your query5.

But bad retrieval is not the solution. Even if semantic search introduces these biases, the underlying claim—that we should tolerate poor retrieval because cognitive friction has pedagogical value—is flawed.

There is certainly value in some friction: working through keyword translation (arguably), understanding knowledge organisation, and evaluating sources (definitely) are productive challenges. But a relevancy algorithm that returns irrelevant results isn’t productive friction. It’s just poor retrieval.

Should we deliberately worsen our library catalogue rankings to create more friction? Obviously not. Not all friction is valuable, and we should be precise about which kinds are.

Why you actually need and want good retrieval

Consider what good retrieval enables:

Surfacing multiple perspectives. If a student wants to understand both sides of a debate, they need precise retrieval of the strongest arguments from each position—not random noise in the relevancy rankings.

Addressing filter bubble concerns properly. If you’re worried about users asking “show me papers saying vaccines cause autism,” the solution is teaching them to ask more open-ended questions and evaluate what they find. Degrading retrieval quality isn’t the answer.A brief aside: tools that silently redirect user queries to “safer” queries aren’t the solution either. Paternalistic redirection is just filter bubbles with different politics. You can’t coherently worry about tools giving users what they want while praising tools that substitute what the LLM thinks users should see.

We’ve already seen this problem manifest in practice. As I documented in “The AI-Powered Library Search That Refused to Search,” content-moderation layers in tools like Primo Research Assistant and Summon Research Assistant were blocking searches on topics like “Tulsa race massacre” and “Gaza War”—returning zero results or error messages for legitimate scholarly queries. These filters, designed for social media chatbots, have no place in academic discovery systems. A search tool that decides which historical atrocities are too sensitive to research is not protecting users; it’s censoring scholarship.

If source quality is the concern, we already know what to do! Curate at the corpus level—don’t let models second-guess user intent. Or be inclusive with sources and invest in teaching source evaluation and critical thinking.

The bottom line

“AI search” is still search. These are RAG systems with retrieval and generation components. The retrieval component should optimise for precision and recall. Generation concerns—hallucination, over-summarisation—are separate issues that don’t excuse poor retrieval.

But I want to be careful not to be too tidy here. The concerns librarians raise, though often imprecisely articulated, sometimes point toward real issues of bias:

• The retrieval/generation boundary is messier in agentic systems than in simple RAG

• Semantic interpretation introduces judgements and possible bias that keyword matching doesn’t

• “Consensus machine” behaviour could systematically disadvantage certain scholarship

• We lack rigorous empirical studies on many of these questions

The appropriate response isn’t to dismiss AI search based on vague misunderstandings. It’s to get precise about what the actual failure modes are, and to investigate them empirically.

We can value deliberate friction (slowing users down, prompting reflection, not auto-generating full papers) while still demanding that search actually works. We can acknowledge genuine concerns about agentic query reformulation while rejecting the conflation of sycophancy with retrieval.

The library world has traditionally been far too forgiving of poor search rankings and user interfaces, we certainly don’t want fuzzy thinking about search to give vendors a opportunity to excuse poor products!

Buy me coffee! (via ko-fi)

I found after after publishing this that Mike Caulfield of SIFT fame, also wrote about roughly the same issue from a different angle, you can read his post - “AI sycophancy” is not always harmful

I’ve focused mainly on understanding retrieval for the past few months, e.g., classifying Academic search into the “4 Quadrants of Search” and distinguishing between truly “agentic” systems and more predefined workflows.

But retrieval is just one part of the equation. LLMs and GenAI are also impacting search through their ability to extract and summarise text1.

Regular readers are familiar with how “naive” Retrieval Augmented Generation (RAG) works: it uses retrieval to find relevant content and then uses the generative capabilities of LLMs to summarise it2. But there’s considerably more nuance to how summarisation is deployed across academic search products.

In this post, I’ll distinguish between the different ways LLMs summarise in academic search and what the implications are for evaluation and trust.

Buy me coffee!

A Taxonomy of LLM Summarisation in Academic Search

After examining various academic search products, here is how I categorise them3:

1. Single Document Summarisation — LLM summarises one document at a time

2. Sequential Single Document Summarisation — LLM summarises multiple documents individually (e.g., synthesis tables)

3. Query-based Multi-Document Summarisation — LLM synthesises across multiple documents to answer a query (e.g., RAG)

I’m classifying by (1) how many documents the model is allowed to use at once, and (2) how much the user/query can change the prompt. Those two knobs largely determine utility, risk, and how testable the feature is.

Utility tends to increase as the model is allowed to use more documents and answer more specific questions. Risk increases when you combine broader scope with more prompt freedom and harder reasoning (especially evidence weighting and contradiction handling).

Subscribe now

1. Single document generation (fixed → query-conditioned → free-form)

Single Document Summarisation is straightforward: you use an LLM to summarise an individual document. However, there are three distinct variants worth distinguishing:

1a. Fixed Button Summarisation — A preset prompt summarises the document

1b. Fixed Query-Dependent Answer4 — A preset prompt incorporates your search query into the document

1c. Dynamic Q&A / Chat with PDF — You provide arbitrary prompts over the individual document

While you might expect these systems to feed entire documents to an LLM, most academic implementations—including EBSCO AI Insights and ProQuest Research Assistant—use RAG internally.

How AI Insights are generated

How Proquest Research Assistant generates key takeaways and concepts

If you see supporting snippets shown alongside generated statements, the system is a signal it is using RAG rather than processing the full document directly.

1a. Fixed Button Summarisation

The simplest form: you click a button, a fixed prompt is sent, and a summary appears. EBSCO’s AI Insights exemplifies this approach.

EBSCO AI Insights explains it this way:

When the AI Insight button is clicked, a prompt is sent to a Large Language Model (LLM) prompting the AI to summarize the full text article into 2-5 relevant insights into the article. We ground the AI response on the full text (with publisher permission) to reduce hallucinations. No AI training is done on the full text article. The user query is not used in AI Insights.

This is probably the lowest-risk summarisation feature because everything is predictable with the least variables: the prompt is fixed, the document is known, and the vendor can test quality systematically before deployment. Ex Libris’s Document Insights in Primo uses a similar approach.

Questions to ask vendors:

• Is full text used, or just the abstract? For ebooks, is it summarising a chapter or the entire book?

• Is the system text-only, or can it process images and figures?

• How do you test the accuracy of these summarisations and do you have benchmarks publicly available?

• How deterministic is the output? Given a fixed prompt over a fixed document, why not pre-generate and cache summaries?

On that last point: pre-generation would save compute and ensure consistency, but I suspect most vendors avoid this for IP reasons. That said, the FAQ for Ebook Central Research Assistant notes it will “generate the same response for each chapter, regardless of who the user is or when they submit the prompt.”

1b. Fixed Query-Dependent Answer

This variant is also “fixed” in that you cannot type arbitrary input—but the preset prompt incorporates your search query.

ProQuest Research Assistant demonstrates this well. When you click on an article, a sidebar automatically generates “Key Takeaways” along with options for Essential Details, Findings or Conclusions, Important Concepts, and more.

Most of these features match 1a, except for one crucial difference: the “Relationship to your search terms” section in the auto-generated “Key Takeaways”. This means even with deterministic generation, results vary depending on the query used to find the paper.

I first encountered query-dependent answers in an older beta of JSTOR’s Research Tool, which automatically generated text explaining

How is <query> related to this text

In my view, this feature deserves wider adoption, particularly when paired with “Semantic/Embedding Search”. It directly addresses the “semantic search always gives you something” problem—warning users that even if the system retrieved top-K documents, they might not actually be relevant to the specific question.

Compared to fixed single document summarisation, being query dependent makes the result a bit more unpredictable, but it is likely it is still much easier to test and tune for compared to the next method.

Questions to ask vendors:

• Same questions as in 1a

• How is query incorporated? What prompts are used?

1c. Dynamic Q&A / Chat with PDF

Unlike the fixed button-based approaches, here users type arbitrary queries into a chatbot-style interface. This “Chat with PDF” pattern has spawned numerous startups, though the value proposition as a standalone product remains unclear given that major platforms are integrating similar functionality directly.

In academic search, this feature appears in most AI-powered systems, particularly those from startups. JSTOR Research Tool (beta) includes this capability.

The risk profile is notably higher than fixed approaches: vendors cannot predict user input, making output quality inherently less predictable and harder to test comprehensively.

Questions to ask vendors:

• Same questions as in 1a

• What types of prompts/queries are expected to work and NOT work?

2. Sequential Single Document Summarisation

This method still summarises document-by-document but processes multiple documents in sequence or batch. Elicit.com pioneered this approach in commercial academic search.

The workflow: Elicit finds relevant papers and auto-generates a “summary” column for each. Users can add custom columns, and the LLM extracts or summarises from each paper individually to populate them.

The auto-generated summary column serves a useful secondary purpose—it acts as a relevance indicator. Because the summary instruction is query-dependent (e.g., “In one line, provide a summary of the source in relation to the research question: is there an open access citation advantage”), irrelevant retrievals become obvious when the summary cannot address the query.

This is essentially ProQuest Research Assistant’s “Relationship to your search terms” applied systematically across all retrieved results.

Variants Within This Category

Sequential summarisation has its own meaningful distinctions:

• Parallel vs. sequential processing — Are documents processed simultaneously or one after another? This affects both speed and potential for cross-contamination of context.

• Batch size considerations — How many documents can be processed together? Elicit and similar tools typically work with the top-K retrieved results.

• Extraction vs. summarisation — Some columns extract specific data points (study design, sample size) while others summarise. Extraction is generally more reliable.

This category holds particular promise for evidence synthesis workflows. For extracting well-defined variables from structured research papers, LLM-assisted extraction can significantly accelerate systematic reviews—though human verification remains essential for anything beyond preliminary screening.

3. Query-based Multi-Document Summarisation

This is the most ambitious and most error-prone category: the LLM takes multiple documents and synthesises them to answer a single query. This is technically known as Query-based Multi-Document Summarisation, with RAG being the most common implementation.

Ex Libris’s Primo Research Assistant exemplifies this approach, as do many other current products.

Variants and Hybrid Approaches

You’re not limited to documents from search results. Query-based multi-document summarisation can operate over:

• Search results only — The standard RAG pattern

• User-provided documents only — e.g., Google NotebookLM

• Hybrid: search results plus uploads — e.g., Elicit Systematic Review, NotebookLM with search enabled

Another variant involves using your initial query to retrieve a fixed set of results, then allowing follow-up questions over that same corpus.

JSTOR Research Tool (beta) retrieves the top 25 results for your query, which then serve as the knowledge base for subsequent questions.

Undermind.ai’s “Ask Expert” feature takes a different approach: after its iterative Deep Research process identifies relevant papers, you can ask follow-up questions.

Notably, for questions like “Predict future trends,” the system can search the general web in addition to the papers already found in the academic corpus (Semantic Scholar)—a sensible design choice since trend prediction often requires non-academic sources.

Why Multi-Document Summarisation Is Hard

This category is difficult for at least three compounding reasons:

1. Retrieval quality — You need the right, relevant documents. Without them, nothing downstream works.

2. Accurate contextual summarisation — Each document must be summarised correctly in relation to the query. Errors here propagate into the synthesis.

3. Evidence assessment and contradiction handling — This is arguably the hardest problem. How do you correctly assess the quality and strength of evidence across documents, particularly when they contradict each other? LLMs tend to be credulous—they treat retrieved content as equally authoritative regardless of study design, sample size, or methodological rigour. This is improving with stronger models, but remains a fundamental limitation.

There has been worry about RAG-enabled LLMs happily citing retracted work. To be fair, humans do this too. The solution requires ensuring LLMs have access to updated indexes with retraction information (or perform real-time checks) and that models understand what retraction status means.

Evaluating Summarisation Quality

A taxonomy is only useful if it helps us assess tools. Here’s the uncomfortable truth: robust evaluation of LLM summarisation remains an unsolved problem, and the difficulty scales with each category.

Five practical checks Librarians can try

Robust evaluation of LLM summarisation is still an active research area5. But librarians don’t need perfect metrics to run useful, repeatable checks. Here are five practical “sanity tests” for each category—lightweight enough to do routinely, but strong enough to surface the common failure modes.

Tip 1: Because these systems are often non-deterministic you should consider running each test 3-5 times each!

Category 1: Single-document summarisation (1a / 1b / 1c)

1. Stability / repeatability test
   Run the same document + same button/prompt 3–5 times. If key claims change materially between runs, treat the feature as inherently non-deterministic and harder to trust.

2. Factual grounding spot-check
   Pick 5–10 concrete statements from the output (numbers, causal claims, named entities, “the study found…”). Verify each directly against the source text.

3. Quote-and-page verification
   For each key claim, ask: can a user find the supporting passage quickly? If the UI provides snippets, check whether they actually support the claim (not just vaguely relate).

4. Negative-control question (especially for 1c)
   Ask something you know is not in the document (“Does this paper report a randomized controlled trial?” when it doesn’t). The best behaviour is “not mentioned / not in the source,” not confident invention.

5. Scope test
   Check what’s being summarised: abstract-only vs full text; for ebooks, chapter vs entire book. If scope is unclear, outputs are hard to interpret and easy to over-trust.

Category 2: Sequential single-document summarisation (synthesis tables / extraction columns)

1. Relevance sensitivity test
   Use a query where only some retrieved papers are truly relevant. See whether the per-paper summaries make irrelevance obvious (e.g., summaries that can’t answer the question should look “off”).

2. Extraction accuracy audit (sample-based)
   For any structured column (sample size, method, population, outcome), randomly audit 10–20 rows against the PDFs. Report an error rate; this becomes your baseline.

3. Consistency across papers
   Check whether the system uses consistent definitions and units across rows (e.g., “n=” always means sample size; outcomes aren’t silently rephrased into different constructs).

4. Missingness honesty
   When information is absent or unclear in a paper, does the tool leave it blank / mark “not reported,” or does it fill with plausible-sounding guesses?

5. Provenance per cell
   For each extracted/summarised cell, can the system show the supporting excerpt? If not, every table becomes a manual scavenger hunt (and a trust sink).

Category 3: Query-based multi-document summarisation (RAG answers / synthesis across papers)

1. Attribution test (claim → source mapping)
   Take 5–10 substantive claims in the answer. For each, verify that (a) a citation is provided, and (b) the cited source actually supports the claim.

2. Coverage test (known-key-paper set)
   Use a question where you know the “must-cite” papers. Does the system retrieve and use them?

3. Contradiction test
   Deliberately use a topic with well-known conflicting results (or seed a small set of papers with disagreement). Check whether the answer acknowledges disagreement and characterises it accurately, rather than smoothing it away6.

4. “Not in the sources” also known as negative rejection test
   Ask a question that cannot be answered from the retrieved set. Good systems will say so, ask to broaden retrieval, or qualify strongly—not hallucinate7.

5. Retraction / correction test
   Seed a retracted paper (or a paper with an expression of concern) alongside later work. Check whether the system flags status appropriately and avoids treating retracted findings as live evidence8.

An alternative idea, I have been toying with is formally adapting Harzing’s 12 guidelines for academic referencing (last updated in 2001 which was designed purely for humans) as a standard to assess RAG output.

Tip 2 : Traditionally, LLMs have a strong bias to answer even when they shouldn’t, as such, if you have time for only one test, it is worth testing how they react to “null answers or results” to check for false positives/hallucations - basically 1.4, 2.1, 2.4, 3.4

Tip 3 : Take note of examples or scenarios where these tools fail. It is highly likely these are good scenarios to reuse.

Conclusion

Not all “summaries” are created equal. Treating AI summarisation as a monolithic capability is a mistake. There is a world of difference between a pre-canned summary of a single article (Fixed Button) and a dynamic synthesis of conflicting claims across twenty papers (Query-based Multi-Document Summarisation).

For librarians and researchers, understanding this taxonomy is crucial for information literacy instruction and tool evaluation. A fixed, single-document summary is relatively safe but limited—essentially a glorified abstract. As we move into dynamic, multi-document RAG, utility increases substantially, but so does complexity and error potential.

Buy me coffee

TLDR: There is a single, blinking cursor that defines our modern interaction with information. It sits in an empty rectangular box, waiting.

For two decades, search interfaces had clear affordances. Boolean operators, date filters, and “Advanced Search” screens signaled exactly what a database could and couldn’t do.

The new era of Agentic AI search has traded these constraints for infinite potential, replacing the dashboard with a blank text box. But this minimalism is deceptive. It has created a massive usability crisis I call the Blank Box Problem, trapping users in a double bind:

• Input Ambiguity: We don’t know how to speak to the machine (Keywords vs. Natural Language vs. Engineered Prompts).

• Capability Ambiguity: We don’t know what the machine can actually do with our words (Search / Filter / Agentic tasks).

Introduction

Traditional academic search interfaces were rich in affordances: Boolean operators signalled combinatorial logic, field codes revealed searchable metadata, dropdown menus exposed available filters, and facets signalled post-filter options. Most importantly, nobody in their right mind would expect it to act like a research assistant and try asking it to perform tasks requiring a complicated series of operations.

Then came ChatGPT, Perplexity, and the new era of agentic AI search. The boxes look the same, but the rules of engagement have shattered. The promise is seductive: we can now tell the AI search engine exactly what we want without pressing buttons or learning syntax.

We now face a usability crisis: we no longer know what to input into the search bar because the “Blank Box” hides the mechanics of the machine. Worse, we are caught in a Double Ambiguity:

• Input Ambiguity: We don’t know how to speak to the machine (Keywords vs. Natural Language vs. Prompts).

• Capability Ambiguity: We don’t know what the machine can actually do with our words (Search vs. Filter vs. Agentic tasks).

As I noted in my last blog post, vague use of terms like “AI-powered search engines” fuels this confusion.

There’s a certain irony here. For decades, librarians lamented that users didn’t understand Boolean operators, truncation symbols, or field codes (or just didn’t care to use facets). Now we’ve built systems that accept natural language—and users are more confused than ever about what they can actually do.

Buy me Coffee!

Part 1: Input Ambiguity

Do I talk like a caveman, a human, or a Sorcerer?

For twenty years, we were trained when using academic databases1 to speak a specific pidgin language: “keyword search”. We stripped away grammar (e.g., “of”), articles (e.g., “the”), and politeness2. We didn’t ask, “Is there an open access citation advantage”. Instead we just typed: open access citation advantage. It was unnatural, but it was deterministic. We knew that if we typed X, the machine would look for X.

Now, the rise of “AI search” or semantic search suggests we might want or need to use other modes, but at this stage of transition it is hard to know when to abandon keyword search.

The current state of AI search forces users to gamble on three different input modes, often with no indication of which triggers the best results:

1. The Caveman (Keyword Search)3: The old reliable. Predictable, typically high precision, the longer the query, but often may fall prey to the “vocabulary mismatch problem”.

2. The Colleague (Natural Language): The promise of “ask me anything.” You talk to the AI like a colleague. But does it understand complex intent, or is it just dropping stop words and extracting keywords from your sentence?

3. The Sorcerer (Prompt Engineering): The magic incantations. To extract value, we are told we must provide context, personas, constraints and even weird incentives that supposedly help the black box of a LLM. e.g. “Act as a senior bibliometrician and find...I will give you $500 if you do well”

Because the interface is opaque, users might be developing folk theories about how to make it work. We add phrases like “take a deep breath” or “think step-by-step” not because we understand the system architecture, but because we hear it works well with GPT-3.5 or other old models - not caring that some of this advice is outdated with reasoning models and more importantly academic AI search is a complicated system of different parts not just a LLM. We have moved from the deterministic logic of Boolean operators to the almost superstitious rituals of Prompt Engineering.

Each mode carries different assumptions about how the system processes input. Keywords assume something like traditional information retrieval—matching terms, perhaps with query expansion. Natural language assumes the system understands semantic intent via embeddings. Prompt engineering assumes the system can follow complex instructions, maintain context, and execute multi-constraint searches.

The core problem is architectural opacity - users have no way to know which mode a given system expects or handles best. A system optimised for keyword matching may perform poorly with verbose natural language prompts. A system designed for conversational queries or one having limited capabilities (see later) may strip out or rewrite carefully crafted constraints. Users may sometimes learn this only through failed searches and inconsistent results4.

This isn’t an abstract concern

Choosing the wrong input mode for a given system can meaningfully degrade results.

Semantic Scholar is a case in point. As I showed in my last blog post, despite the name, its primary retrieval is lexical, not semantic.

Semantic Scholar’s primary retrieval pipeline when you search on the website

If you query it with natural language sentences expecting embedding-based retrieval, you’ll miss most of the relevant literature compared to a keyword search without stop words.

Below shows treating Semantic Scholar like a natural language search, results in only 13 results missing a ton of relevant results as compared to keyword search.

Conversely, using “Caveman” keywords on a system designed for agentic reasoning (like the new Deep Research tools) might not be a total disaster but wastes the machine’s potential to understand nuance and counter-arguments.

If you read all this and are confused (you are not an information retrieval engineer!), you can be sure your users are too.

Part 2: Capability Ambiguity

Even if we figure out how to ask, we hit a second wall: we don’t know what the system can actually do (Capability Ambiguity). When an AI search fails to deliver what users expect, the failure can occur in three distinct ways—and the AI blank box provides no way to determine which.

1. Retrieval Abstraction: Similarity Spectrum vs Buckets

Search relevance engineer Doug Turnbull has argued that the AI search industry fundamentally misframed the problem by building around vector similarity alone. Because users interact with LLMs in natural language, developers assumed the underlying retrieval should also use natural language similarity—encoding queries as embeddings and finding semantically similar passages on a spectrum.

But this is not always the right abstraction. Users don't only want semantic similarity. They often want what Turnbull calls selectors or what you can think of as binary buckets - structured filters on specific binary attributes. When a researcher asks for “peer-reviewed papers on CRISPR ethics published after 2020," they're expressing a query with clear selectors: document type (peer-reviewed papers), topic (CRISPR ethics), date range (post-2020). They expect the system to filter on these attributes or buckets not merely find passages that are "close" in some abstract embedding space.

Some academic AI search tools—particularly earlier versions of Elicit5—relied heavily on vector embeddings for retrieval and ranking with no way to apply pre-filters.

So why not combine both vector embedding retrieval with hard filtering? The issue is combining top-K vector search with attribute filtering (e.g., filtered HNSW) is technically difficult, though newer techniques like ACORN are making it more tractable6.

Today, some AI search tools like Consensus and Elicit offer numerous pre-filter options, while others like Primo Research Assistant and Scopus AI offer far fewer filter options than their “parent” non-AI search indexes.

When a user is faced with a blank chat box with no affordances like pre-filter checkboxes or checklists how is he to know to ask if he can ask in natural language to filter by say Publication year?

What I can say is if you assume the lack of affordances means the system can’t do any filtering - this is not always true.

For example, in Undermind.ai, it can get you papers by publication year by natural language querying but it cannot filter reliably by publication type due to the fact that the underlying index (Semantic Scholar) does not even have this metadata value.

Yet in both cases, Undermind will usually not warn you what you are asking for isn’t supported and try anyway to produce a result without an error message.

2. Filter Recognition in natural language

Even when systems basic search index does support structured constraints—date ranges, document types, citation thresholds—a separate question arises: does the natural language interpretation layer recognise when users are invoking them?

Consider common search constraints expressed naturally:

• “from 2000 to 2010” or “published in the last five years”

• “sorted by most cited” or “most recent first”

• “only peer-reviewed articles” or “excluding conference papers”

• “English language only”

• “open access articles”

The underlying search infrastructure might even support all of these filters. But the natural language layer—the part that parses your query7 and decides what to do with it—may or may not recognise these requests.

Consider a user entering a specific request like:

“Show me peer-reviewed articles from 2010 to 2015 about machine learning, ordered by most cited.”

This seems like a clear instruction. However, depending on which AI Research Assistant the user faces, the result is wildly different:

• The “Deep Parser” (e.g., Web of Science Research Assistant): This tool is built to understand a ton of specific metadata triggers (if a filter or sort option appears in Web of Science you can likely trigger it using natural language!). It recognizes “2010-2015” not just as text, but as a command to apply a Publication Date Filter. It understands “ordered by most cited” as a specific Sort Function. The AI successfully translates this and many other filters from natural language into structured database queries.

• The “Surface Skimmer” (e.g., Primo Research Assistant): This tool might process the same query very differently despite being from the same company. While it can actually “understand” and parse the input to set the publication year ranges but does it know what “peer-reviewed” means as a metadata filter? (Spoiler : it does!). Or will it just look for the phrase “peer-reviewed” in the text? The user has no way of knowing. Also, can it sort by citation counts (Spoiler: it cannot).

More recently, Primo NDE8 introduced "natural language search". Here's what it says

The Natural Language Search feature enables users to formulate queries in normal spoken language, and automatically converts them into the structured format compatible with Primo’s Advanced Search. For example, the user could enter the query, “Find me US history journals in English that are available online,” and the system would create a query with the appropriate criteria for the search.

So far, similar (use of LLM to translate to search strategy) to what we see in Web of Science Research Assistant, Primo Research Assistant, EBSCO Natural Language Search etc.

Not only does the Natural Language Search generate queries from free text, it also identifies certain catch words in the text that can be used to define the scope and automatically select the appropriate filters for the search. For example, if the term “journal” appears in the text, the scope of the search is limited to journals, and if a language is specified, the language filter is automatically turned on.

The transformation of the original query into the structured format is performed using generative AI, via ChatGPT 4.1Mini. The elements of the query, such as resource types, date filters, full text preferences, language, and advanced fields like Title or Subject, are classified by the AI system, which uses this information to generate the basic Boolean query and then expands it using related concepts. Ambiguous inputs are handled intelligently by mapping to multiple fields (e.g., both Title and Subject) to ensure broad yet accurate retrieval.

Ah, so it can recognise filters in your natural language query which is invoked by clicking on “Ask Anything” button.

Here’s the problem. Which filters can you use purely by asking in natural language? You won’t know without tons of trial and error or reading the documentation.

I would wager many people might think anything that is a filter in Primo NDE could be invoked by natural language queries. In fact, while a lot of filters and facets can be invoked this way (e.g. resource type, open access, available online, peer-reviewed, held by library, language, creator), some cannot (e.g. subject, collection, etc.).

For example, while testing, I was initially encouraged by the fact that most of the filters in the Primo NDE instance I was testing could be invoked—e.g. the following works correctly:

Find me open access papers on large language models

Find me peer-reviewed papers on large language models

Find me papers on large language models in Spanish

Then I tried to invoke the Subject filter by trying

Find me papers on large language models in <subject as per Primo NDE filter values>

or the sorting function

Find me papers on large language models and sort it by date-newest9

But this time it refused to invoke the subject filter, nor sort by date-newest. I then tried different variants of the query input, etc., to no avail, and I guess it probably wasn’t supported - but you could never by sure.

Of course, when you look at the documentation, you realise it could never have worked as it wasn’t a supported type to filter on.

Because the interface is identical—an empty box—users cannot distinguish between a tool that respects metadata constraints and one that merely performs semantic fuzzy search and always returns the closest top K results even if it ignores the filter constraints.

3. Workflow Composition Problem

The third point concerns what operations the system can perform and whether it can combine them.

Consider this research task: “Find me papers referenced by Article X, then identify related papers that could have been cited but weren’t.”

You normally wouldn't expect this to work in the old days with conventional search but with talk about agentic search, ai-powered research assistants users may expect this to work.

While some agentic systems can indeed do it, this requires running multiple operations in sequence: retrieve the article, extract its reference list, search for semantically related papers, compare the sets, identify gaps. Still, it's a reasonably simple enough research workflow.

But AI search tools vary dramatically in whether they can execute it. In my testing, “deep research” or agentic search tools like Elicit, Undermind, Scite Assistant, and Consensus failed at this task, despite many having citation searching or parsing capability—probably due to predetermined workflows.

Agentic systems with genuine compositional capability can accomplish this: fetch the article, parse references, run searches, synthesise results—potentially working with a sequence of steps that wasn’t predetermined by the designers.

Systems with fixed workflows like AI2’s Asta (formerly AI2 Paper Finder), Undermind.ai, Elicit, Consensus implement predetermined pipelines: “literature search,” “citation chasing,”, “evaluation of results” or “summarisation” in a fairly fixed sequence. The LLM decides which pre-built flow to invoke, but it cannot compose novel sequences. The operations exist in isolation, not as combinable primitives.

The user staring at the search box has no way to know which category their tool falls into. They may assume the limitation is their own—that they haven’t found the right phrasing—when they’re hitting an architectural ceiling. This is perhaps the cruellest form of capability ambiguity: users engage in elaborate prompt engineering to unlock capabilities that don’t exist.

(For more details, see Deep Research, Shallow Agency: What Academic Deep Research Can and Can’t Do.)

Why does a search fail?

Failures in most AI search are usually silent and indistinguishable from the user’s perspective. A search returning irrelevant results could be failing for any of the following reasons and it is not easy for a user to understand why:

• Whether they used the wrong input mode (e.g. using natural language query with a lexical search system)

• Whether they phrased their query poorly within the correct mode (e.g. Use of wrong keywords in a lexical search system)

• Whether their filters were recognized and applied (e.g. asking in natural language to “filter to review articles” but this isn’t a field the system can filter on or it can do so but the natural language parser isn’t set up to do so).

• Whether the system lacks the workflow operations (e.g. do forward citations of paper X) they need to do the task.

• Whether those operations exist (do forward citations of paper x) in the system but can’t be composed into their desired workflow

More dangerously, parts of the search could silently fail or be ignored. This is certainly true for embedding systems that simply find the closest matches, LLMs that rewrite your input query (e.g., Primo Research Assistant), and Deep Research tools (e.g., Undermind.ai). You will always get 'something,' even when your query is not correctly interpreted!

Case Study: The Danger of the “Silent Rewrite”

The most dangerous failure is the one users cannot see. Consider this query I ran in Primo Research Assistant:

Find articles on climate change in 2023 , review articles only and do forward citation search of top 10 papers

It happily ran with no trace of an error despite not being able to filter by review articles, not to mention it definitely couldn’t do things like forward citation. Below shows the result.

The LLM in Primo Research Assistant rewrote the input to a boolean search without the restrictions and ran it. Below is the rewritten query. Notice what it dropped.

(climate change impacts) OR (global warming) OR (climate change adaptation) OR ((climate change) OR (global warming)) OR ((climate change) AND (environmental policy)) OR (”climate change”) OR ((global warming) OR (climate variability)) OR ((climate change) AND (mitigation)) OR ((greenhouse gases) AND (climate impact)) OR (environmental change) OR (climate change)

There is also another subtle failure. Primo Research Assistant can correctly parse query inputs like

Papers on X from 2023 to 2024,

but when the query input is

Papers on X in 2023

was interpreted in Primo Research Assistant as 2023 to 2025 as shown in the applied filter at the bottom of the screen capture!

You might then think - maybe if you just prompt in the right way, you could get it to filter to only 2023 (maybe be adding “Only”)10.

Here lies another problem, even if the user notices that the query input wasn’t followed, it might lead to “prompt thrashing”—users reformulating the same request in increasingly elaborate ways, hoping to unlock functionality that may not exist or may be failing at something query reformulation doesn’t (or can’t ever) address. They blame their query formulation when the actual limitations could, say, be architectural.

Meanwhile, systems that do support complex operations provide no signal that such capabilities are available. A user who never thinks to ask “compare the citation networks of these two papers” will never discover that the system can do it.

Designing a Way Out

Turnbull argues the solution lies in reconceiving the role of LLMs in search. Their power isn't semantic similarity—it's query understanding. An LLM can take free text ("suede, geometric couch") and produce a structured query with typed fields: style (geometric), material (suede), classification (Living Room / Seating / Sofas). Each field maps to an appropriate retrieval technique—visual embeddings for style, taxonomic matching for materials, hierarchical classification for categories.

This decomposition serves multiple purposes. It enables precision where precision is possible. It makes the system’s interpretation visible and correctable. And crucially, it provides affordances: users can see how their query was parsed and understand what refinements are available. The continuous similarity score of vector search offers no such guidance.

But most AI search interfaces don’t expose this architecture. They present a blank box and hope the system will figure out which parts of the input are filter requests, which are content queries, and which are workflow instructions and fail silently.

Concrete Solutions

Solutions exist and some are already in use in existing products. I will list some of them from most frequently seen to less frequently seen.

Sample prompts that demonstrate expectations. It is common practice today to list sample queries below the search box. But example queries should be chosen with care to address both input mode and capability. Are examples terse keywords or elaborate instructions? Do they demonstrate filter syntax, complex operations, or both? Prompts like “Find highly cited papers on CRISPR ethics from 2020-2024” reveal that dates and citation thresholds are recognised; “Compare the methodology of Smith 2020 and Jones 2021” reveals that document comparison is available.

Hybrid UI: the return of filters. There’s no reason we can’t add pull-down menus and buttons to signal available affordances. But consistency matters—if the interface shows a dropdown for document type, users should be able to request the same filter in natural language.

Type-ahead that reveals constraints. As users type “find papers from...” the system could surface suggestions: “...from 2020 onwards,” “...from Nature,” “...from Harvard University.” This signals which constraint types are recognised while guiding users toward phrasings the system handles well.

Explicit constraint confirmation. Before executing a search, the system could display parsed constraints for confirmation: “Date: 2020-2024 | Type: peer-reviewed | Sort: citation count | Topic: CRISPR ethics.” Users see exactly what was recognised, can correct misinterpretations, and learn the system’s vocabulary through exposure.

LLM query intent detection to block unsupported inputs Pretty much what it says on the tin. The LLM tries to interpret the intent of each query, if it detects that a query is asking for something the tool isn’t designed to do, it stops.

Expose the tool layer. For systems built on protocols like MCP, surface available operations directly: “Search database X” “Retrieve (forward) citations from Paper Y,” “Compare documents,” “Extract methodology sections.” When users see that citation network analysis isn’t among the available tools, they stop blaming their phrasing and adjust expectations—or choose a different system. Systems with fixed workflows should make clear what exactly is happening behind the scenes to help head off misunderstandings.

Template libraries for common workflows. Offer structured starting points for workflows: “Find gap in citations,” “Compare methodology across papers,” “Trace influence of seminal work.” Each template pre-configures and exposes the right combination of tools while remaining editable for workflow transparency. See for example SciSpace Agent Gallery.

Conclusion

These solutions address symptoms, but the underlying tension may be irresolvable. The promise of natural language search is liberation from formal query languages; the reality is that formal languages encoded precise information about both input expectations and system capabilities that natural language cannot easily replace.

We’re in a transitional period where AI search tools vary enormously across many dimensions—different retrieval abstractions, different filter recognition capabilities, different workflow architectures. Yet they present nearly identical blank text boxes that reveal nothing.

This results in a massive “Cost of Discovery.” In the past, if you learned how to use Scopus, you largely knew how to use Web of Science. Now, every AI wrapper has a unique, hidden architecture with varying capabilities.

This places an unreasonable burden on users to discover through experimentation what each system expects and what each system can do. The ambiguities compound: users may not be able to easily systematically test because of so much uncertainty in the different subsystems of the search.

For librarians and information professionals, this creates both challenge and opportunity. The challenge is obvious: how do we teach search skills when the skills required vary by platform in ways that aren’t visible? Traditional database instruction could focus on Boolean logic and controlled vocabularies. AI search instruction must address input mode, retrieval method, filter recognition, and workflow composition—none of which are documented completely and consistently.

The opportunity is that this confusion creates demand for exactly the kind of critical evaluation that information professionals excel at. Mapping the capability landscape of AI search tools—not just feature lists but actual functional boundaries at each level—is work that needs doing. Understanding when embedding retrieval is in play and hence when natural language input is appropriate, when filter triggers are recognised, when workflows can be composed: this technical evaluation distinguishes informed tool selection from marketing-driven adoption or fear-driven resistance.

Until AI search interfaces evolve better affordances, we may need to return to an old practice: demanding that documentation exist for input expectations, filter vocabularies, retrieval mechanisms, and workflow capabilities—and that it be surfaced at the point of need rather than buried in help pages no one visits.

The blank search box feels like an invitation to ask anything. It’s actually a test at multiple levels—of whether you know how to ask, whether you know what you can ask for, and whether you can diagnose what actually doesn’t work when you don’t get what you need.

6

Vector similarity of dense embeddings usually involve finding the top K most similar embeddings to the query embedding. With a large enough collection, it may be too slow to do a brute force search so systems employ approximate nearest neighbour (ANN) indices like HNSW (Hierarchical Navigable Small World). But these indices are optimised only for the distribution of the full dataset. For example, HNSW builds a hierarchical graph where nodes (vectors) are connected to their nearest neighbours across multiple layers, with navigation starting at sparse upper layers and refining at denser lower layers. During search, the algorithm greedily traverses edges towards the query vector. However, when filters exclude many vectors, the graph connectivity breaks—the greedy path may dead-end at filtered-out nodes, or the remaining filtered subset forms disconnected islands in the graph structure.

The index doesn't "know" about filter constraints, so traversal routes optimised for the full dataset fail to efficiently find nearest neighbours within the filtered subset. Post-filtering (retrieve K, then filter) risks returning fewer than K results. Pre-filtering (filter, then search) means either rebuilding indices per filter (impractical) or using the mismatched global index (suboptimal recall). Recent work such as ACORN addresses this by selectively expanding search to second-hop neighbours (neighbours of neighbours) when immediate neighbours are filtered out, maintaining connectivity at the cost of additional distance computations.

Buy me Coffee (via ko-fi)

I've been watching the reactions to Google Scholar Labs with considerable interest. The responses range from enthusiastic embrace to outright rejection. One response particularly intrigued me—someone mentioned they were initially reluctant to try because they'd heard it was "AI-powered" but became more interested when they read my review and realized what that actually meant (they expected it to generate answers to questions when all it did was do better ranking).

Another interesting puzzle was when I noticed some library guides listing Semantic Scholar as “Semantic/Neural Search” when a technical look at their main retrieval method reveals the main search is still largely lexical search.

While one can understand and agree with the listing of Semantic Scholar as “AI powered” due to clear AI features like TLDR, doing the same for Lens.org and OpenAlex is a much harder sell, because they not only just do keyword search but they lack the obvious AI features of Semantic Scholar.

It’s made me realise we might all be talking past each other because we haven’t actually defined what we mean.

“AI-powered search engine” is a handy catch-all term used by vendors, but it actually hides a diverse set of systems and functionality. In this post, I’ll dissect the different ways academic search can be “AI-powered” so you can decide which types actually cross the line for you.

I am going to argue that we often mean at least 4 different things when we call something “AI-powered search”

• Level 1: Post-Retrieval AI Features

• Level 2: Going beyond Lexical Search with Semantic Search

• Level 3: LLMs for Retrieval and/or Relevance Ranking

• Level 4: Synthesis and Generation Across Papers

• Level 0??: Use of AI to extract, cluster, or organise metadata used for retrieval

Levels here may not be the right framing, as the four different categories are largely orthogonal (except maybe Level 3 is a subset of level 2), but they map to most common academic search products (e.g. Level 4 is usually Deep Research) and higher levels generally reflect higher risk and greater amount of pushback from librarians.

The Spectrum of AI in Search

Level 1: Post-Retrieval AI Features

Does “AI-powered search engine” mean “AI” that impacts the search process only? Not necessarily.

There’s a whole category of AI features that don’t affect the search results you get at all. Things like optional summarisation of individual items (e.g., AI Insights on Ebscohost databases), translation tools, or text-to-speech features.

These are post-retrieval conveniences. You search, you get your results list (however that list was generated), and then you have the option to use AI to help you process what you found. The search itself? Unchanged.

In theory, if you don’t like these features, you can ignore them.

Level 2: Going beyond Lexical Search with Semantic Search

Now let’s get into actual search mechanisms. Suppose a search engine doesn’t do traditional lexical/keyword matching but instead uses “semantic search”—where queries are run through an encoder model to convert into embeddings and matched against the embeddings of indexed documents to get a relevance score.

“Semantic search” can refer to (1) meaning-aware retrieval (dense embeddings / neural retrievers), (2) sparse neural retrieval (not dense embeddings, but still “neural” e.g. SPLADE) (3) query expansion / thesauri / ontologies (not embeddings). (1) is by far the most popular today and discussed here.

Semantic Search using dense embeddings (also known as vector/embedding/neural search/retrieval)

Is that “AI”?

A quick terminology note: when I say “lexical” or “keyword” search, I’m referring to methods that match based on the actual words in your query and the documents. This includes both Boolean search (exact matching with AND/OR/NOT operators) and probabilistic methods like BM25 (which scores documents based on term frequency, document length, and how rare each term is across the corpus). These are related but distinct—Boolean gives you a set of matching documents, while BM25 gives you a ranked list. Most traditional academic databases like Scopus use Boolean first to retrieve a set of documents and then do relevance ranking with BM25 (or the related TF-IDF). Neither attempts to understand semantic meaning as they just match strings. That said, it is common to stack a reranker on top of that which is trained using supervised learning (learning to rank model) on labelled click data. We will discuss this in the section on Semantic Scholar.

Let’s take for granted that traditional information retrieval methods like Boolean and BM25 aren’t “AI” otherwise by default every search engine is AI!

So what we’re saying is that anything that goes beyond traditional keyword matching—such as semantic search via embedding search/match—is considered “AI-powered search.” An example is JSTOR beta’s semantic results function.

Note that all that changes is the relevancy algorithm. You still get a list of results (though embedding-based search usually only retrieves top K results rather than all hits like a Boolean search). There is no generated answer.

Embedding search is fantastic at understanding broad conceptual similarity, but it can be ‘fuzzy’—sometimes retrieving papers that feel related but miss the specific keywords or strict criteria you need. That is why it is common to do hybrid search combining both methods

A decade or two ago, more librarians would have heavily objected to non-keyword search methods because the results would be less explainable. (I suspect two decades of using Google’s soft or fuzzy Boolean search and getting relevant results without all matching terms has weakened the resistance! ). These days I see more librarians (except systematic review specialists) accepting such retrieval methods, particularly if they improve recall or precision—even if it means losing both interpretability and often reproducibility.

They’re often even more accepting when told the embeddings aren’t strictly “Large Language Models,” which now seems to be the bogeyman many people have in mind. But how different are they really?

Language models compute the conditional probability of a token given a context window, using these probabilities to sample and generate sequences. Embedding models, however, focus on representation learning, mapping input sequences to fixed-size vectors without an autoregressive generation component.

That said these days the embeddings we use (e.g., BERT, SciBERT, SPECTER) are transformer based encoder models which are very close cousins to the GPT style transformer based decoder models. (I will ignore encoder-decoder models)

How decoder models (LLMs) are trained: GPT-style models are trained on massive text corpora using “next token prediction” (also known as causal language modelling)—given a sequence of words, predict what comes next. This autoregressive training is what enables them to generate fluent text. The model learns statistical patterns across billions of documents, building an implicit representation of language, facts, and reasoning patterns.

How encoder models are trained: Models like BERT, SciBERT, and SPECTER use a different pretraining objective—”masked language modelling” or essentially a cloze test. Random words in a sentence are masked out, and the model learns to predict the missing words based on surrounding context. This bidirectional training (looking at context on both sides) produces rich representations of meaning, but the model isn’t designed to generate text—it outputs embeddings that capture semantic relationships.

Historically, only encoder models were used as embedding models and while many modern embedding models (e.g., SBERT, E5, Gecko) are still based on transformer encoder architectures, recent top-performing embeddings models—like OpenAI's ada and Google's Gemini embeddings, e5 mistral etc—are derived from large decoder-based transformers. While decoder models are not naturally designed to output embeddings - they are meant to predict next token, with proper fine-tuning, decoder models can effectively generate embeddings, often excelling due to being derived from their larger base model size (decoder derived embedding tends to have billions of parameters instead of hundreds of millions for encoder models ), though with higher energy costs

The similarities are substantial. Both use the transformer architecture. Both are trained on large text corpora through self-supervised learning. Both learn distributed representations of language. The encoder/decoder distinction is primarily about the training objective and what the model outputs, not fundamental differences in how they learn from data.

This means that if your concern is intellectual property—where the training data comes from, whether it was used with permission, whether the model has “ingested” copyrighted works—the issues are largely the same for both model types. An encoder model trained on scraped academic papers raises similar provenance questions to a decoder model trained on the same corpus.

However, there is one meaningful difference: encoder models do not reproduce text. They convert text into numerical vectors and compare those vectors. A decoder model, by contrast, can regenerate text that closely resembles its training data—this is the basis of concerns about LLMs reproducing copyrighted passages or memorising private information. Encoder models used for semantic search can only say "these two texts are semantically similar," - they don’t generate passages, so the plagiarism-style risk is lower.

So if your concern is specifically about text reproduction and the potential for plagiarism or copyright-infringing outputs, encoder-based semantic search is genuinely lower risk. If your concern is about the ethics of training on data without consent, the distinction offers less comfort.

There are also practical differences: embedding models tend to be much smaller and less energy-hungry than modern LLMs. And because we’re not generating answers, just doing relevance ranking, we avoid issues like citation faithfulness and cognitive offloading.

Level 3: LLMs for Retrieval and/or Relevance Ranking

Okay, so maybe you’re comfortable with embeddings from encoder models and you are happy it doesn’t generate answers.

What if the search engine uses an LLM to generate a Boolean search strategy and then runs keyword search as normal (e.g., Web of Science Research Assistant)? Or does hybrid search where it first runs LLM-generated Boolean before reranking using embeddings (e.g., Primo Research Assistant)? Or does both LLM-generated Boolean plus embedding search together and reranks (e.g., Scopus AI)?

Or what if the system retrieves using keywords (or hybrid methods) then uses GPT-style LLMs to judge relevancy and generate ranking of the top K results?

When using LLMs for relevancy ranking, you can ask the LLM to:

• Give a score or categorise documents into relevancy tiers (point-wise comparison)

• Compare relevancy of two documents (pair-wise comparison)

• Provide a rank sort of multiple documents (list-wise comparison)

You can also prompt the LLM to give a reason for the score or ranking.

Regardless of method, we’re now using those big decoder models—the ones everyone associates with “generative AI”—but we’re still just using them to rank search results (typically after a earlier retrieval phase to rank on promising candidates). No synthesis, no summarisation (except to explain reasoning for relevance), just ordering.

Technical note: You can also use powerful but slow Cross-Encoder or late-interaction models like ColBERT instead of outright GPT-style decoder models to get results almost as good as directly using a LLM. Functionally, with respect to ranking quality there is little difference but you do avoid the use of decoder LLMs and hence the ip risks.

Google Scholar Labs, AI2 Paper Finder (now Asta) uses this type of approach - it uses an LLM to do query intent parsing, rank results and even generate a rationale explaining why each paper was ranked where it was. You’re still ultimately getting a ranked list, but now there’s AI-generated explanatory text attached to each result.

Does this change things, because there is now actual generation of text to explain relevancy or will your objection be that it uses actual LLMs (decoder models) rather than just encoder models?

Level 4: Synthesis and Generation Across Papers

Now we’re getting into what many find most objectionable. This level encompasses two related approaches:

Quick RAG tools like Elicit, scite assistant, and Primo Research Assistant synthesise information across multiple papers to generate answers with citations, typically using Retrieval Augmented Generation. You ask a question, the system reads through papers, extracts relevant information, and writes you a summary with references. It’s not just ranking anymore—it’s creating new text based on the literature.

Deep research tools like Undermind and Consensus’s deep search mode take this further—basically RAG on steroids. These systems don’t just synthesise a quick answer; they conduct extensive multi-step research processes, following leads, refining searches, and building comprehensive analyses that can run for minutes or even hours.

Both approaches share the same fundamental characteristic: the AI is generating novel text that synthesises across sources, not merely ranking or retrieving existing documents.

Leaving aside the use of LLMs, there are reasons to object to Level 4 tools:

• LLMs can produce ghost references (though this can be mitigated)

• Studies have reported LLMs citing retracted papers (here and here)

• There are doubts whether LLMs can faithfully represent what each article says and properly weight findings when they disagree

Even if these performance issues were mitigated, people also worry about the adverse effects of using and relying on such tools - e.g. cognitive offloading from overuse of these functions.

Different Concerns, Different Red Lines

Here’s the thing—there’s no objectively “right” answer about what counts as unacceptable AI use. But it’s crucial to be clear with yourself about what you’re actually concerned about, because different concerns lead to very different red lines.

If you’re worried about generative AI making students lazy, you might be perfectly fine with AI assistance as long as it’s just ranking results. Let the algorithms find relevant papers, but make students read and synthesise themselves.

If you’re worried about environmental impact or IP issues of huge GPT-style LLMs, you might accept encoder embeddings but draw a hard line at any use of GPT-type models.

If you’re worried about reproducibility or interpretability—and regular readers know this is my particular obsession—you might object to any non-Boolean/lexical methods, and maybe even AI-generated metadata.

If you’re worried about accuracy and hallucination, the earlier levels might be fine but synthesis crosses the line. Ranking can be imperfect without being catastrophically wrong, but generation creates new opportunities for confident-sounding nonsense.

Where Do Common Tools Actually Sit?

Given this framework, it’s worth examining where familiar platforms actually fall—because the answer isn’t always obvious.

The “Semantic Scholar” Confusion

Here’s an irony worth noting: Semantic Scholar—the platform whose name suggests semantic search—doesn’t actually use vector embeddings for its main search retrieval - at least not on the search on its home page as of April 2025.

According to a preprint updated April 2025, the platform’s search operates in two stages: first, Elasticsearch retrieves up to 1,000 candidates using keyword matching (probably BM25); then these are reranked using a LightGBM model that emphasises direct title matches and highly-cited recent papers.

LightGBM is a gradient-boosted decision tree ranker—machine learning, but not a neural embedding model and not generative. It’s a feature-driven reranker sitting between BM25-style retrieval and modern transformer-heavy “AI search.”

More specifically, it’s supervised or semi-supervised learning model trained on labelled data (e.g., relevance judgments or interaction signals) to learn a scoring function that combines feature-based signals—query/field match strength, citations, publication year, etc.—into a final relevance score.

It’s typically used in a learning-to-rank–style setup (often with ranking objectives like LambdaRank), meaning it learns how to order candidate documents for a query rather than generate text.

So where do the embeddings come in? Semantic Scholar developed SPECTER and SPECTER2, sophisticated document embedding models trained on citation graphs. But these power auxiliary features: research feed recommendations, author disambiguation, paper clustering, and finding related papers. The embeddings exist; they’re just not driving the core search experience.

There is one exception: their newer “snippet search” API endpoint, designed for retrieving text passages from the S2ORC corpus, does use a genuine hybrid approach. Passages are embedded using mxbai-embed-large-v1, and results are retrieved using the union of embedding-based and keyword-based matches, ranked by a weighted sum of embedding similarity and BM25 scores (Kinney et al., 2023). But this is a specialised API for full-text passage retrieval—not the main search you get from the Semantic Scholar Search bar that most users encounter.

You might be wondering why I am harping on this. After all, Semantic Scholar has many features that are clearly AI-powered, such as “TLDR”, “Ask this paper”, Citation sentiment (e.g., “Highly Influential Citations”), Research Feeds.

This matters because if you’re advising researchers, the name might reasonably lead you to assume you’re getting embedding-based semantic retrieval. In fact, I have seen quite a few libraries list Semantic Scholar in their libguides as “semantic/neural/vector search”.

You’re not—at least not in the primary interface.

This misunderstanding might then lead you to search using natural language rather than keyword, which typically works better in semantic retrieval systems. Unfortunately, this fails here. The query:

Is there an open access citation advantage

returns only 13 results (big underestimate), while keyword search even with quotes gets a more reasonable 54.

Are OpenAlex or Lens.org AI-Powered?

Listing Semantic Scholar as “AI powered search” at least makes some sense because of many AI features.

On the other hand, I also notice quite a few LibGuides classify OpenAlex and Lens.org as “AI powered/LLM search” or worse semantic/neural search. This is a much less defensible move. Both are pure lexical search engines with no LLM used in retrieval or ranking, and no synthesis features.

New! In the latest Walden update, OpenAlex is planning a Vector search endpoint: find relevant works and other entities based on semantic similarity of free-form text. As of 24 Feb 2026, OpenAlex now has semantic search.

This is probably just a misunderstanding by some librarians as they became aware of Lens.org and OpenAlex only recently (Lens.org actually started going beyond Lens search with Scholarly search in Dec 2017) and mistakenly assumed it was in the same class as newer search, but is there a way to argue they are “AI-powered” search engines?

Perhaps, you might argue that using AI to extract, cluster, or organise metadata makes a search engine “AI-powered.” and because we know OpenAlex uses machine learning for topic assignment and author disambiguation - it is AI-powered.

But the problem with this is that by this standard, even traditional databases like Scopus would qualify—they use clustering algorithms for author disambiguation and citation parsing.

You can try to draw a line to decide which methods are “AI” which are not (the same way we can with retrieval algos) and decide you’re okay with good old-fashioned machine learning or even embeddings if they are used to extract or clean up metadata but are not okay if LLMs are used.

Dimensions has a interesting post on the different types of “AI” (Symbolic AI, Discriminative AI, embeddings, knowledge graphs and even generative AI) employed under the hood for metadata.

When you browse Dimensions to identify research funded by NIH, when publications from an author who moved affiliation are under the same profile, when a cluster gets labeled with a coherent name, this is Artificial Intelligence at work, although not all is Generative AI. So, what are the different flavours of AI and what do they do?

But this whole thing sounds very contrived and I think most would agree use of “AI” for metadata handling should not count as AI-powered search engine.

What I’m Getting At

I’m not trying to tell you where to draw your line. What I am saying is: watch out for knee-jerk reactions (my own included!) that treat “AI” as a monolithic thing we either accept or reject wholesale.

The field is rapidly developing systems that incorporate different types of AI at different layers, in different ways, for different purposes. Some of those uses might align with your values and needs. Others might not. But you can’t make that determination until you actually understand what’s under the hood.

And unfortunately, that’s often the hardest part—because many of these systems are black boxes that don’t clearly explain which AI techniques they’re using, where, or why. Which brings me back to my recurring theme: methodological transparency matters.

We can’t make informed decisions about what AI search tools to use (or refuse to use) if we don’t actually know what they’re doing.

So the next time you see an “AI-powered search engine,” maybe the first question shouldn’t be “Is this AI?” but rather “What kind of AI, “where and how it is used in the pipeline”.

Appendix

Questions to ask or think about for a AI-powered search

A) Where is AI used in the pipeline?

1. Query formulation (query rewriting, Boolean drafting)

2. Retrieval (candidate generation)

3. Ranking (reranking / learning-to-rank / LLM judging)

4. Result enrichment (snippets, TLDRs, explanations)

5. Synthesis (multi-document answer)

B) What kind of AI model is used there?

• classic IR (Boolean/BM25)

• “traditional” ML (e.g., boosted trees / LTR -Learning to Rank)

• embeddings (encoder models; dense/sparse)

• generative LLMs (decoder; possibly multimodal)

Images generated with help of Gemini 3 Pro

Buy me Coffee! (via ko-fi)

As early as late 2022, I understood that Retrieval Augmented Generation (RAG) would be the future. By grounding LLM responses in retrieved content, RAG should reduce or even eliminate certain types of hallucinations—including the dreaded ghost reference: citations to papers that simply do not exist.

To be clear about terms:

• Ghost reference: a citation to a work that does not exist.

• Citation unfaithfulness: a citation to a real work that doesn’t actually support the claim being made.

• “Existing” or “real” reference: A work you can point to, access, or verify exists. Quality is not guaranteed—such a work might be a legitimate but retracted paper (a “zombie citation”) or a product of paper mills, which these days often but not always means a fully AI-generated paper. Nevertheless, it still counts as “existing” in this context. An “existing” paper or reference might also make ghost references.

In the early days of ChatGPT (GPT-3.5 era), ghost references were common because these tools (or at the free versions) didn’t perform web searches to ground their outputs - which was akin to asking a human to cite papers without searching to verify to confirm if they “remembered” the reference correctly. But that’s no longer true for most modern systems which search the web and perform Retrieval Augmented Generation (see image below generated by NanoBanana Pro).

While, this problem of ghost references seems to be no longer a serious issue at my institution, a quick scan of librarian blogs and social media suggest this problem is still a huge issue. Why is this?

I think, the answer is more uncomfortable than “LLMs hallucinate”

The answer lies in a troubling interaction between two vulnerabilities.

The first is architectural: Google Scholar creates [CITATION] records for references it cannot match to actual documents. These stubs—inferred from bibliographies rather than verified sources—allow fabricated references to accumulate citations and apparent legitimacy from humans and possibly LLM alike.

The second is that RAG systems using general web search can be fooled by this and similar pollution. When an LLM searches the web to verify a citation, it may find pages that themselves cite the ghost reference, concluding the paper must be real.

In short, ghost references are not primarily an LLM/GenAI phenomenon. They have always existed, propagated by entirely human mechanisms—typos, careless copying, citing without reading. What LLM/GenAI has done is accelerate and amplify a pre-existing structural vulnerability in scholarly communication infrastructure. The web is already contaminated; RAG with web search inherits that contamination.

How Academic RAG Actually Prevents Ghost References

I noticed early on that academic RAG and Deep Research tools largely solved the ghost reference problem. A 2023 thesis demonstrated that even basic early academic RAG systems like Scopus AI, Elicit, and SciSpace do not fabricate references.

Minor citation errors still occur, but these typically originate from upstream metadata issues—think of how Google Scholar sometimes displays incorrect publication years due to merging preprints with the version of record.

Going deeper into how RAG generate citations

LLMs do not naturally cite the text chunks they retrieve, and even if they did, how do you guard against cases where they don’t follow instructions? However, there are architectural approaches that can virtually guarantee zero fabricated citations. Here is one such method (see image below generated by Nano Banana Pro):

Step 1: Unique ID Assignment. The database assigns a unique identifier to each document chunk. When chunks are retrieved in response to a query, they carry these identifiers along with their text.

For example, given the query

“Which library is the oldest public branch library in Singapore?”

the retrieval step might return:

• <text chunk 1> [UniqueID-XYAD]

• <text chunk 2> [UniqueID-QASD]

• First public branch library in Singapore was in Queenstown [UniqueID-XXXX]

Step 2: ID-Only Citation Generation. The LLM is prompted and fine-tuned to generate text with only the unique ID—not the full citation. For instance, it might produce the following:

Singapore’s oldest public branch library is in Queenstown [UniqueID-XXXX].

Step 3: Post-Hoc Verification. A non-LLM method (typically python script) verifies that each generated unique ID matches a retrieved document and exists in the database. This catches any hallucinated identifiers.

Step 4: Programmatic Citation Replacement. Finally, a deterministic process (such as regular expressions with Python Script) replaces each unique ID with the full, verified citation or metadata. We do not trust the LLM to generate complete citations because it might hallucinate details.

This architecture virtually eliminates ghost references, as it never gets a chance to invent citation metadata. That said it cannot prevent unfaithful statements—claims that misrepresent what the cited work actually says. Various methods exist to detect unfaithful claims (e.g. entailment-based verification), but none are foolproof.

Critically, this approach works because academic RAG systems perform retrieval over a known, curated, and bounded corpus. General-purpose LLMs searching the open web cannot easily adopt this strategy—and therein lies part of the problem.

Ghost References Have Always Existed before the use of LLMs

Someone shared with me the following Bluesky post.

This Bluesky post highlighted a ghost reference that had accumulated 43 citations in Google Scholar. The “paper” appeared as a [CITATION] entry —a type of Google Scholar record we’ll examine in detail shortly.

While works in roughly the same topic by the purported authors exist (for instance, Williamson & Piattoeva published on related topics in 2019 and a 2020 book chapter), this specific reference does not correspond to any real publication.

I suspect most people initially assumed this from a LLM hallucinated reference that had propagated through the system. But my response hinted that this might not necessarily be the case.

Regardless of the initial source of the error, the Google Scholar practice of generating [CITATION] records is quite dangerous, as we shall see, this practice can “pollute” the web.

Examining the citing articles revealed something important: citations to this ghost reference came from papers supposedly published in 2019 and 2021!

This was years before ChatGPT’s widespread adoption in late 2022 and while GPT 2 (2019) and GPT 3 (2020) was a thing, it was unlikely to be the cause of a 2019 or even 2021 citing paper.

This complicates the easy narrative that GenAI created the ghost reference problem. These early citations were likely produced by humans through entirely traditional mechanisms:

• Typographical errors that propagated

• Careless copying of reference lists without verification

• Conflating two similar papers into one that doesn’t exist

• Simply misremembering a paper

That said, one can easily see the offending citing papers are from very poor quality sources, so we can’t even be sure if the publication dates are correct!

Still, I won’t be surprised if at least some of them are really just errors from humans.

This shouldn’t surprise us. The empirical literature on citation practices has long documented these patterns. Simkin and Roychowdhury’s influential studies estimated that only about 20% of citers actually read the original papers they cite—the rest probably copy references from other papers’ bibliographies (Simkin & Roychowdhury, 2003). Related are studies about academic paper urban legends - where claims like most papers are not cited are asserted by misrepresenting papers - all of this was done pre-GPT!

In the particular topic under discussion, it seems we also have evidence that ghost references are common (pre-GPT).

But as you will see later, more recent ghost references might perhaps be LLM/GenAI related.

Why “RAG + web search” doesn’t reliably prevent ghost references

When I asked the free version of ChatGPT (currently GPT-5.2 Instant) to locate the ghost reference from the Bluesky post, it confidently claimed the paper existed and pointed to what appeared to be a source. That source? A webpage from another journal that itself had cited the ghost reference.

This is perhaps part of the reason why GPT still generates ghost references even with web search?

To be fair, the paid version of ChatGPT (GPT-5.2 with thinking) occasionally identified the paper as fake by locating the original Bluesky post from the authors. But results were inconsistent—different prompts yielded different conclusions.

It is unclear to me if LLMs could be taught to try to verify papers the way we librarians and researchers do it (imagine connecting to MCP sources like Crossref, PubMed) and can say things like this reference looks fishy, as the DOI doesn’t match or there seems to be no trace of a journal article (beyond appearance in references) that should be easily found.

The broader lesson: LLMs with general web search can fail to reliably verify references because the web itself contains fake citations.

This creates a feedback loop. A ghost reference (either human or LLM generated) gets cited by a real paper. That paper appears online. An LLM finds the citation and concludes the reference must be real. The ghost becomes increasingly entrenched.

This is the “citogenesis” phenomenon famously illustrated by XKCD.

This occurs when Wikipedia errors propagate into published sources and back again. Reference list errors in influential papers get copied by subsequent authors who don’t verify their sources. A typo becomes a “paper” that dozens of researchers claim to have read.

Similarly, once a ghost reference appears (human or LLM generated), this citogenesis process can be accelerated by both LLMs and careless humans.

The scary bit is this, even if LLMs from here on out do not hallucinate ghost references “on their own” and always take backing from web sources, they may still run find existing ghost references and maybe cite them!

What’s changed is not the existence of ghost references but their potential scale and discoverability. GenAI can produce more content, faster, with confident-sounding citations. And when verification is attempted using LLMs with web search, those tools can be fooled by the pre-existing pollution that humans created over decades.

I want to be careful here: we don’t yet have good empirical data on how much GenAI has increased the rate of ghost reference creation. This is an important research question. What we can say is that the architectural conditions exist for acceleration, even if quantifying the actual increase remains difficult.

Why Google Scholar practice of creating [citation] records is dangerous in this day and age

To understand the structural vulnerability at play, we need to examine how citation indexes are constructed. I always recommend librarians and researchers to understand how citation indexes are created because it explains a lot of issues with citation counts.

Citation indexes follow a four-step pipeline:

1. Collect. Citation indexes covers a defined set of works. Even Google Scholar, as broad as it is, focuses on “scholarly work.” So step one is to gather a defined set of works—specific journals, conference proceedings, book chapters, etc. Most well-known citation indexes focus on journal articles and, to a lesser extent, conference proceedings, book chapters, and occasionally preprints (for broader, more open ones).

2. Extract. The system analyzes each collected work to identify its bibliography, pulling out reference strings in indexed works e.g.

Small, H. (1973). Co-citation in the scientific literature: A new measure of the relationship between two documents. Journal of the American Society for Information Science, 24(4), 265–269. https://doi.org/10.1002/asi.4630240406

3. Match. This is the most technically complex step. The system attempts to link extracted references to works already in its database. This is not straight forward. Cited references exhibit enormous variation – some issues

• Author name variants (initials, transliterations, name changes)

• Journal title abbreviations (inconsistent or non-standard)

• Typographical errors in the original references

• Incomplete citation

Matching algorithms must balance precision and recall—too strict and you create false negatives (cited works lose citations); too loose and you create false positives (incorrect links).

4. Total. The system aggregates links to quantify influence, producing the citation counts researchers use to gauge impact.

In short, citation indexes transform static reference lists into dynamic, countable links, converting individual documents into a networked web of scholarship.

Step 3—matching—is the least understood yet most critical part of this process. What happens when an extracted reference cannot be matched to any existing work in the database?

The Google Scholar [Citation] Vulnerability

When an indexed work contains reference strings that don’t match anything in your index, one option is to do nothing.

But Google Scholar’s answer to unmatched references is instead to create [Citation] records—entries that appear when GS detects a reference in an indexed paper’s bibliography but cannot locate the actual source document.

These are essentially citation stubs: metadata records inferred from parsed reference lists rather than from direct indexing of primary documents.

Key characteristics include:

• The title appears with a “[CITATION]” prefix and typically isn’t hyperlinked to full text.

• Metadata is often incomplete or inaccurate (missing authors, incorrect dates, truncated titles) because it’s reconstructed from how other papers cited the work.

• They lack abstracts since Google Scholar never accessed the original document.

• They still accumulate citation counts—which is actually their primary function.

This mechanism serves legitimate purposes. An unmatched reference might represent a real work that simply isn’t indexed by Google Scholar: perhaps a print-only source, a work behind paywalls GS cannot penetrate, or simply a typo that prevented matching. For instance, referencing “Tay, C.H. (2020)” when the actual author is “Tay, C.L. (2020)” would cause matching to fail.

My own Medium posts have legitimate [CITATION] entries because they’ve been cited by indexed scholarly works despite not being “scholarly” content that GS would directly index.

But the [CITATION] mechanism has always been a structural vulnerability. It allows fabricated references—whether created by human error or GenAI hallucination—to enter the scholarly communication system with an appearance of legitimacy. Once a [CITATION] record exists and accumulates citations, it becomes increasingly difficult to distinguish from a real paper that simply isn’t available online.

I don’t believe Google Scholar creates a [CITATION] record for every unmatched reference—there’s likely some threshold, such as multiple indexed works citing the same reference.

But once a ghost reference clears that threshold and a Google Scholar [citation] record is created, the citogenesis cycle accelerates. Careless researchers (and LLMs with web search) see the [CITATION] record, assume the paper exists, and cite it themselves.

Google Scholar has extremely tight anti-bot features, so LLMs usually cannot search and “see” the [CITATION] record directly. But once that record exists on the web, there are numerous ways for it to “leak out”—through library discovery systems, reference management tools, and academic social networks that surface Google Scholar metadata.

A good example, is a journal or ebook with ghost references on a platform that offers link resolver links. The system will create a library link resolver link that when clicked brings the user to a library catalog record that looks like a real record you can ask for ILL or Document Delivery!

This is a sample record produced in my institution’s catalogue generated by a ghost reference on a platform that has link resolver links

This vulnerability existed long before GenAI. The 2019 citations to our example ghost reference prove that. What GenAI changes is the rate at which new ghost references can be generated and how quickly they can propagate across the web, making citation verification increasingly difficult.

Are Academic RAG Tools Safe from This Problem?

Given the problems with general web search, this is precisely why we turn to academic RAG and Deep Research tools like Elicit, Consensus, and Undermind. They don’t retrieve over the open web but instead query curated sources like Semantic Scholar and OpenAlex.

Similarly, LLMs with MCP connectors to trusted content sources like PubMed or Wiley avoid web pollution by querying authoritative databases directly.

But are these tools immune to the ghost reference problem?

Most AI search startups without their own data sources rely on Semantic Scholar, OpenAlex, or some combination of web scraping and partnerships. While more inclusive than Scopus, we can be reasonably assured that works in these indexes are “real” in some sense. OpenAlex (prior to the Walden update) only indexed works it could match with Crossref records.

Important caveats apply: existence says nothing about quality. A work could exist as a PDF, on a preprint server, even have a DOI, and yet still be a paper that is entirely AI-generated—potentially including references to papers that don’t exist. But at least the indexed work in OpenAlex itself does “exist”.

The critical architectural difference is how these indexes handle unmatched references. Unlike Google Scholar, OpenAlex and Semantic Scholar appear to only display references that can be matched against indexed works (with an OpenAlex ID or equivalent).

Two of the most popular commercial citation indexes—Scopus and Web of Science—actually allow searching of unmatched reference strings via the somewhat obscure “Cited Reference Search” and “Secondary documents” features respectively. Does OpenAlex or Semantic Scholar have something similar?

They do not create [Citation] records for unmatched references.

This is an important safeguard.

Leaving aside LLM web search finding such records, as more tools such as Undermind.ai, Consensus Deep Search, implement citation-chaining in Deep Search functionality, this design choice prevents them from surfacing and amplifying fabricated references that exist only as the equalvant of [citation] stubs.

The Loose Matching Problem

A warning remains: while these indexes won’t create new records for ghost references, their matching algorithms may incorrectly link ghost references to existing real works if the algorithms are too permissive.

Consider a ghost reference generated by LLM or human with slightly garbled metadata. An overly loose matching algorithm might incorrectly link it to a real paper with a similar title or author. The ghost reference doesn’t create a new stub—instead, it incorrectly inflates the citation count of an existing work and creates a false trail that future researchers might follow.

The ghost reference that started this investigation had 43 citations, likely from a combination of

• humans finding the Google Scholar [Citation] record and citing it

• humans copying references from papers containing the ghost reference

• LLMs finding “evidence” of such records on webpages and citing it

• LLMs hallucinating (in the traditional sense) variant forms of the ghost reference and being linked through loose matching algorithms.

• It has been suggested that these human errors might be in the pretraining of LLMs leading them to generate such ghost references, I don’t think this is likely - such errors appears in a tiny fraction of training data—far too infrequent to reliably influence generation.

I haven’t seen empirical work directly testing this hypothesis. It would be interesting to check whether known historical ghost references appear at elevated rates in LLM outputs compared to base rates in the literature.

The Missing Layer: Editorial and Peer Review Responsibility

The infrastructural focus of this analysis is important, but we shouldn’t neglect the human gatekeeping failures that allow ghost references to propagate.

Reference verification should be part of editorial and peer review processes. In practice, it rarely is. Reviewers focus on methodology, argumentation, and contribution—not on whether every cited work actually exists. Copy editors check formatting consistency, not ontological validity.

This is understandable given time constraints, but it means that the scholarly communication system lacks a verification layer at the point where it might be most effective. By the time a ghost reference appears in a published paper, it has already gained legitimacy.

Some journals have begun using automated reference checking tools, but these typically verify formatting and DOI resolution rather than comprehensive existence checks. A reference to a non-existent paper without a DOI would sail through.

As ghost references potentially become more common, publishers and editors may need to invest in more robust verification infrastructure—or accept that the scholarly record will become increasingly polluted.

Conclusion: A Pre-Existing Condition, Now Acute

The persistence of ghost references in 2025 may not just be a story about LLM randomly hallucinating ghost references for no reason.

Academic RAG systems that retrieve from curated databases and use proper citation verification have largely solved this problem at the technical level.

The real story is older and more uncomfortable. Ghost references have always existed, created and propagated by human sloppiness: typos, careless copying, citing without reading, conflating similar papers. The scholarly communication infrastructure—particularly Google Scholar’s [Citation] mechanism—has long had structural vulnerabilities that allow these fabrications to persist and accumulate apparent legitimacy.

What GenAI changes is the scale and the difficulty of detection. LLMs can generate ghost references faster than humans ever could. And when we try to verify references using LLMs with web search, those tools are fooled by the pre-existing pollution that humans created over decades. The contamination feeds on itself.

This has several practical implications:

For researchers: Treat Google Scholar [Citation] records with heightened suspicion—but recognise this was always good practice, not just a GenAI-era precaution. If you cannot access the full text of a work, verify its existence through multiple authoritative sources before citing it. The existence of a Google Scholar entry—even one with many citations, even one that predates ChatGPT—does not guarantee that a work is real.

For librarians: When teaching information literacy, the traditional lesson was to prefer Google Scholar over general web search because it indexes “scholarly” content. This guidance needs to be updated with education about whar [citation] records are and how to handle them.

For tool and citation index developers: The choice not to create placeholder records for unmatched references (or at least not make them easily discoverable)—as OpenAlex and Semantic Scholar have made—is a design decision with significant implications for research integrity. As citation-chaining becomes more common in AI search tools, this architectural choice becomes increasingly important.

Even though you can filter such records out in Google Scholar, they might consider whether [Citation] records need additional friction, warnings, or provenance indicators.

For researchers studying scholarly communication: We need better empirical data on this phenomenon. How has the rate of ghost reference creation changed post-2023? How do different citation indexes handle unmatched references, and what are the downstream effects? How do ghost references propagate through citation networks, and can we detect them algorithmically? The pre-GenAI literature on citation errors and “academic urban legends” provides a foundation, but the landscape may have shifted significantly.

The ghost reference problem is a chronic condition that has become acute. The infection predates GenAI; the technology has simply lowered our immune response while accelerating transmission. The cure lies not in blaming LLMs but in understanding—and shoring up—the structural vulnerabilities in scholarly infrastructure that have allowed ghost references to propagate for far longer than we’d like to admit.

Introduction

When AI academic search tools promise “research agents” and “AI assistants,” do we know exactly are they offering? What types of literature review tasks do these “agents” and “assistants” support? Are they able to reason flexibly, use the tools at their disposal to accomplish tasks like human research assistants?

I ran a simple test—asking tools to find papers that should have been cited by a given article but weren’t—and discovered something revealing: most fail completely.

Why do “agents” or “Research Assistants” fail such a simple task?

In short, we find that while today’s Academic Deep Re/search tools are impressive—they are mostly workflow-bound, not fully autonomous agents. They are closer to systems that execute pre-built scripts with AI decision points—not use human-level reasoning to flexibly achieve the goal.

So yes, these are agents in a technical sense. But calling them “research assistants” or “agentic” systems risks misrepresenting their limitations—especially for novel or off-template tasks not covered by the default workflows.

The Definition Problem

We now have four distinct quadrants of AI search tools:

Quick Search: Conventional single-shot search (lexical, semantic, or hybrid) showing ranked results. Examples: most traditional databases, Scopus, Google Scholar.

Quick RAG: Single-shot retrieval plus short generated answer. Examples: Primo Research Assistant, Scite Assistant (to some degree).

Deep Search: Iterative retrieval taking minutes, with LLM relevance ranking, returning ranked results. Examples: AI2 Paperfinder (now Asta), Scholar Labs.

Deep Research: Deep Search plus report generation. Examples: Undermind, Elicit Reports, Consensus Deep Search.

But where does “agentic search” or agents that search fit? I’ve been implying that Deep (Re)search is synonymous with agentic search. Is this actually true?

While the definition of “agent” has been hotly debated in the AI industry. one technical definition gaining traction is simply to say

An LLM agent runs tools in a loop to achieve a goal.

or as defined by Anthropic - agents are

LLMs autonomously using tools in a loop

By the “LLM using tools in a loop” definition, virtually all Deep Search and Deep Research tools ARE agents—they use LLMs to make decisions, call tools (search APIs, citation networks), evaluate results, and iterate. AI2 Paperfinder, Elicit, Consensus Deep Search, Undermind—all qualify.

Case closed? Not quite.

Autonomy or Agentic Behavior is a spectrum

Anthropic defines agents as LLMs autonomously using tools in a loop but there are different degrees of autonomy or agency that could be given to the LLMs.

Imagine at one end of the spectrum we have

100% Handcrafted systems : These systems follow predetermined workflows. The LLM makes decisions at specific points, but the workflow structure is fixed by human designers. Think of it as a decision tree: sophisticated, yes, but anticipated in advance.

At the other end we have

100% Flexible systems: These systems reason from scratch about how to accomplish a task. Given a novel request, they devise an approach dynamically, combining available tools as needed to try to accomplish the goal.

Indeed one early analysis of Deep Research tools proposed this very distinction.

A clear example of a handcrafted system is Allen Institute for AI’s PaperFinder. They makes this explicit when describing their Paperfinder architecture as a “semi-rigid flow: a predefined structure that is influenced at key points by various LLM decisions.” They acknowledge: “Of course, it would be cool if the LLM component were more dynamic and allowed more autonomy in how it influences the flow.”

In comparison, tools like OpenAI Deep Research sit much closer to the trained / autonomous end of the spectrum than most academic “Deep Search” products.

Where a workflow-based tool like AI2 Paperfinder follows a predefined sequence (query analyser → router → sub-flow A/B/C, etc.), Deep Research behaves more like a general-purpose research agent:

• It starts from a natural language prompt:

“Write a detailed literature review on X, highlighting key debates and gaps.”

• It plans its own work: decomposes the task into subtasks (e.g., “define scope”, “gather sources”, “cluster perspectives”, “draft sections”).

• It chooses tools dynamically: web search, citation lookups, summarisation, comparison, checking contradictions, etc.

• It adjusts the plan on the fly if it hits coverage gaps, conflicting evidence, or poor sources.

• Unlike LLMs that have fixed workflows, the same prompt can be handled very differently (e.g. might use different set of tools) due to the non-deterministic nature of LLMs

These types of systems tend to be trained with reinforcement learning

This spectrum of agency or autonomy captures something crucial. Most current academic Deep Search and Deep Research tools sit firmly on the handcrafted end: sophisticated workflows, yes, but workflows that human designers anticipated in advance. The LLM’s role is to make intelligent choices within the workflow—not to design the workflow itself.

Which means such tools will almost certainly fail when asked to do tasks that don’t fit neatly into those predetermined workflows. Worse, they may still return results—it may not even be obvious that the system has failed until you examine the output carefully.

Of course, one could argue The “LLM + tools in loop” definition is too permissive a defintion for agents or agentic behavior. Perhaps tools like AI2 Paperfinder are “agentic systems/frameworks” to automate certain tasks as opposed to being actual agents with agentic behavior. But there is no right definition!

Does the “agentic” or “research assistant” framing lead to misunderstandings?

Many of these AI literature review or AI powered tools have adopted marketing language, that may suggest advanced autonomous capabilities— by either talking about ”research agents” if not about “research assistants”.

I am not pointing fingers at particular players, pretty much all of them do similar marketing.

Consider these framings:

Consensus

Scholar Agent is your personal research assistant in Consensus, built into Pro and Deep search modes. It goes beyond quick summaries by performing multi-step searches, applying rigorous academic filters, and surfacing deeper insights, trends, and key papers-fast.

Undermind.ai homepage is more restrained on their homepage but you see on their Y Combinator tagline they also invoke the “agent” idea

An AI agent for scientific research

Meanwhile, pretty much every tool even if it avoids the “Agentic” language - positions their tool as a “research assistant” (e.g. Elicit.com, Clarivate products).

This language—”research assistant,” “agent,” “multi-step reasoning”—implies flexibility. It suggests that given any literature review task, these systems can devise an approach, just as a human research assistant would.

But if the underlying architecture is handcrafted workflows, users will inevitably encounter tasks that fall outside those workflows—and be surprised when the system fails.

How autonomous or flexible is your agent?

Any AI search tool can find papers on a topic fitting some criteria. But can they handle this simple task?

Find me papers that could/should have been cited by Paper X but were not.

A human research assistant would immediately devise a plan:

1. Find papers on the same topic as X, filtered by publication year

2. Extract the references from Paper X

3. Report papers found in step 1 that don’t appear in step 2

This requires no specialized knowledge—just the ability to reason about task structure and combine tools appropriately (well within reach of the reasoning capabilities of modern LLMs). I also choose this task because many Academic Deep Research tools like Undermind.ai, Consensus Deep Search, all claim the ability to do citation searching.

It’s precisely the kind of task that reveals whether a system has flexible reasoning or merely executes handcrafted workflows.

We may expect that these LLM “agents” or “research assistants” when given a task to reason from scratch and “decide” what to do with the tools they have. Systems with fixed workflows will probably fail, unless the designers had anticipated such tasks and created a workflow in advance for this (unlikely).

I tested this using an open access paper: “The state of OA: a large-scale analysis of the prevalence and impact of Open Access articles.” My prompt:

find the paper “The state of OA: a large-scale analysis of the prevalence and impact of Open Access articles.” Then, identify papers that are related to this paper that could have been cited but are not cited by it

The paper is open access, so failure wouldn’t result from access restrictions. It has a relatively comprehensive bibliography, so randomly suggesting papers wouldn’t work.

For the system to succeed, it needs to have the capability to access the paper, extract its references, find related papers, and compare and filter.

Most importantly, it needs either a predefined workflow for this specific task (unlikely) or the ability to reason flexibly about combining available tools.

See Appendix for more details on the tests.

The Results

General-purpose LLMs succeeded. Claude Sonnet—even the free version with default web search without specialised MCP connectors to gain access to academic or even paywalled content like Wiley AI Gateway or Pubmed —reasoned through the task structure, identified the paper, extracted references, found related work, and filtered appropriately.

Some suggestions were tangentially related (the obvious ones were already cited), but it clearly understood and executed the task logic. .

GPT models similarly succeeded when I tested them months ago.

Gemini and GPT-5.1 currently struggle, but primarily due to access issues (they claim to be blocked from full-text). Uploading the PDF directly works.

Notably, I used these LLMs in standard mode, not Deep Research mode. Deep Research modes should work as well unless they trigger predetermined workflows rather than flexible reasoning.

In conclusion, almost all top general-purpose LLMs sit on the trained/flexible end of the spectrum: they reason from scratch about each prompt and devise approaches dynamically.

Most Specialized Academic Deep Search/Research Tools Fail

AI2 Paperfinder has a pretty transparent workflow in their blog post, so I asked Gemini 3 to extract the workflow and generate a flow diagram using Nano Banana Pro and this was the result.

[Update April 2026] I updated the image using the details mentioned in a more recent 2026 paper appendix F3 that describes the inner-workings of a “frozen-in-time subset of the PaperFinder subcomponent of the Asta project (”the PaperFinder Product”)"

While the diagram above may not be 100% correct, it gives you a big picture view of what is happening.

This looks very complicated but currently “PaperFinder agent works as a pipeline of manual coded steps which involve LLM decisions” and they found “manual-coding approach to be more efficient (in terms of number of LLM calls, number of tokens, and in terms of the ability to parallelize) and more reliable than a more dynamic process that grants more autonomy to the LLM in several key-points”

Important to note is that AI2 Paperfinder (now Find paper in Asta) tries to direct your input to one of several subflows including “specific paper search, semantic search with potential metadata constraints, pure-metadata queries, and queries that involve an author name.” (In the above flowchart, they correspond to navigational queries, semantic queries and metadata queries)

When given my query it naturally considered it as a simple specific paper lookup and stopped once it found the target article. It had no capability of going further.

Consensus Deep Search also failed—despite using GPT5 to act as four different sub-agents (planning Agent, Search Agent, Reading Agent, Analysis Agent) and returned papers that were already cited.