Let’s take a closer look at NVIDIA’s latest finding, one that challenges a core assumption about how RAG should work.

The Overlooked Problem in RAG

The typical architecture behind Retrieval-Augmented Generation (RAG):

• A retriever (such as BM25, BGE, or ColBERT) fetches relevant documents from a corpus.

• A generator (usually an LLM) re-encodes these retrieved texts and then produces the final answer.

This separation creates a persistent mismatch: the retriever and generator often operate in different representation spaces.

Read more

This post might offer an insightful perspective.

Motivation

Current Visual Language Models (VLM) and OCR tools already do a strong job extracting content from individual PDF pages.

However, they typically output structures limited to individual pages. What downstream tasks like RAG (Retrieval-Augmented Generation), document Q&A, and automated analytics really need is a coherent, document-level structure that spans multiple pages.

Specifically, existing OCR methods struggle in three main areas:

1. Cross-page Fragmentation: Paragraphs and tables often get split at page boundaries, and OCR systems struggle to recognize these separated parts as belonging to the same logical unit.

2. Structural Ambiguity: Recovering heading hierarchies and linking captions, images, tables, and governing sections requires reasoning over multiple OCR elements, often across page boundaries.

3. Cost and Scalability: Relying heavily on manual review or high-end VLMs for post-processing is expensive, making it impractical for large-scale projects, privacy-sensitive situations, or local deployments.

MinerU-Popo: From Broken Pages to Smart Documents

The core idea of MinerU-Popo is straightforward: adding a lightweight “document structure fixer” after OCR. It transforms the fragmented, page-level OCR outputs into coherent, document-level structures.

Read more

For a RAG (Retrieval-Augmented Generation) system, the ultimate goal of PDF parsing is to extract structured content that’s searchable, referenceable, and analyzable.

My interest in PDF parsing within RAG-Anything, a well-known open-source RAG project, led me to write this article.

Key Concepts

Let’s start with a few essential concepts.

A parser can be thought of as a “document disassembler.” It takes inputs like PDFs, images, or Office documents and breaks them down into data chunks suitable for subsequent system processing. In RAG-Anything’s code, a unified interface (raganything/parser.py:68) defines key methods: parse_pdf(), parse_image(), parse_document(), and check_installation().

The content_list is the standardized intermediate format after parsing. Think of it intuitively as a “table of contents” for document content: page 0 might contain text, page 1 an image, page 2 a table, and page 3 an equation. Technically, it’s represented as List[Dict[str, Any]], with each item specifying its type, page_idx, and the actual content. Detailed structures for text, images, tables, and equations can be found in the README.

Multimodal processing refers to handling non-textual content, which can’t simply be inserted into a RAG system as regular text without losing important semantic details. Hence, the parser identifies images, tables, and equations as distinct items. Later, dedicated modal processors utilize visual models or language models to generate descriptions, entities, and relationships from these non-textual components.

PDF Parsing Workflow

Figure 3 outlines the complete PDF parsing workflow used by RAG-Anything.

When processing a PDF, ProcessorMixin.parse_document() first checks if the local file exists. It then generates a cache key (raganything/processor.py:48) based on the file path, modification time, parser name, parse method, and certain parsing parameters. Then get cached parsing result if available and valid (raganything/processor.py:239).

If the cache misses, the PDF file triggers the current parser’s parse_pdf() method. To prevent blocking the main asynchronous workflow, this potentially heavy parsing task runs in a separate thread using asyncio.to_thread()(raganything/processor.py:470).

In the default PDF parsing flow (raganything/parser.py:1078), it creates an output directory named using a hashed file path to prevent filename collisions during batch processing. The _run_mineru_command() function (raganything/parser.py:714) then builds and executes the MinerU CLI command. MinerU (From Big Picture to Details: MinerU 2.5 Redefines Document Parsing — AI Innovations and Insights 77) is designed to handle OCR, table, formula extraction, and multiple backend parsing scenarios, accepting parameters like method, backend, device, start_page, end_page, formula, table, and source.

After running MinerU, the project does not treat the command-line output as the final result. Instead, it reads JSON and Markdown files produced by MinerU (raganything/parser.py:960). The code accommodates variations in field names, such as img_caption versus image_caption. It also converts paths for images, tables, and equations into absolute paths and verifies them to avoid security issues.

Once parsing completes, the system generates a content-based doc_id. It then separates text and multimodal content (raganything/utils.py:89). Textual data moves into LightRAG, while images, tables, and equations are handled by the multimodal processors (raganything/processor.py:607).

Why Three Parsers?

If the goal were just extracting text from simple PDFs, a lightweight library would suffice. But RAG-Anything tackles more challenging scenarios: scanned documents, intricate layouts, tables, equations, images, and subsequent multimodal understanding.

This complexity drives the project’s choice to combine specialized external parsers with an internal standardization layer. External tools manage extraction, while RAG-Anything handles parser selection, caching, field compatibility, path normalization, content_list standardization, and integration with LightRAG (AI Innovations and Trends 03: LightRAG, Docling, DRIFT, and More).

So RAG-Anything doesn’t limit PDF parsing to just one library. Instead, it introduces a unified parser abstraction alongside multiple implementations.

• The default approach is intended for extracting complex PDF structures Using MinerU 2.0 (raganything/parser.py:694). But this strength comes with costs: relying on external CLIs and models means dealing with environment setup, initial model downloads, and subprocess errors.

• Another implementation leverages Docling’s Python API (raganything/parser.py:1455, AI Innovations and Trends 03: LightRAG, Docling, DRIFT, and More). It optimizes efficiency by caching the DocumentConverter through _get_converter() (raganything/parser.py:1627), reducing redundant model loading. DoclingParser supports both Office documents and HTML parsing. However, to clarify: even though DoclingParser itself includes parse_html(), the higher-level method ProcessorMixin.parse_document() currently doesn’t route HTML files directly through this method. Thus, it’s not entirely accurate to say HTML parsing is fully integrated into the top-level flow.

• The third option is PaddleOCRParser (raganything/parser.py:2052) functions more like a dedicated OCR path. PDFs are rendered into images page by page using pypdfium2, followed by OCR processing (raganything/parser.py:2205). The output is compatible with content_list format, but structural semantics like tables, images, or equations aren’t preserved.

This design is about downstream processes depending only on a unified content_list, making parsers interchangeable based on specific needs.

The three implementations also differ operationally: MinerU shells out to a CLI, Docling uses an in-process Python API with converter caching, and PaddleOCR lazily imports optional OCR dependencies.

Thoughts

RAG-Anything’s PDF parsing can be summarized as follows: specialized parsers handle complex documents, the results feed into a standardized intermediate structure called content_list, and from there, textual and multimodal contents flow separately into the RAG pipeline.

But I have a concern. While content_list serves as the project’s stable boundary, it also represents a point where information may be lost. Details like the original PDF layout, bounding boxes, hierarchical structure, and confidence in reading order are not fully preserved.

Finally, if you want to explore other core mechanisms of RAG-Anything or have any thoughts about this post, feel free to leave a comment.

Reference: https://github.com/HKUDS/RAG-Anything.

Today, let’s dive into the approach and findings related to using RAG for code completion.

Motivation

When using RAG for code completion, it’s common practice to first divide the code into chunks before retrieval. However, the question of exactly how to chunk the code hasn’t been systematically studied.

Most existing work has focused primarily on optimizing the retriever or generator, treating chunking merely as a default preprocessing step. But chunking directly influences the smallest unit of code available to the retriever. Poor chunking can degrade the quality of completions, no matter how advanced the retriever or the language model itself may be.

So here’s something worth thinking about: In retrieval-augmented code completion, how important is chunking really? Is structured chunking truly better than a simple sliding window approach? And when balancing chunk size, context length, cost, and effectiveness, what’s the best trade-off?

Retrieval-Augmented Code Completion: Key Technologies

Before jumping into the findings, let’s first unpack the key technologies involved.

Chunking: How to Divide Code for Retrieval

Chunking means breaking source code files into small, retrievable pieces that the retriever can work with later. There are four representative strategies:

• Function Chunking: Extracts each top-level function or method, keeping its signature and body together when possible; oversized functions are split along direct child-statement boundaries.

• Declaration Chunking: Retains class headers, field definitions, method signatures, and function-level declarations while omitting method bodies; in the Python setup, private underscore-prefixed functions are filtered out.

• Sliding Window: Splits files into fixed-size line-aligned windows with configurable line overlap, without using syntax structure. This method is simple, language-agnostic, but risks splitting functions awkwardly.

• cAST: Operates over all Abstract Syntax Tree (AST) node types, recursively splitting large nodes and greedily merging adjacent siblings to produce more uniform chunks.

Read more

That gap between short bursts of intelligence and sustained understanding is exactly where memory becomes impossible to ignore.

Background

LLMs fundamentally operate by responding based on current context. Conversation history, tool invocation, or task execution depends essentially on the information provided in the prompt.

However, real-world agents typically aren’t limited to answering single, standalone questions; they need to handle longer-term tasks involving multi-turn dialogues, personal assistance, software engineering projects, gaming, scientific discovery, and more. For these complex, long-duration tasks, memory becomes a critical module for LLM-based agents. Memory allows agents to accumulate knowledge, reuse past experiences, and make more coherent decisions throughout ongoing tasks.

As illustrated in Figure 1, the simplest method to manage context is “packing large portions of history directly into the prompt,” known as naive long-context prompting. But this approach quickly encounters practical issues: context windows overflow, token consumption increases significantly, latency grows, and the model might struggle to reliably pinpoint crucial information from overly lengthy contexts.

In contrast, memory-augmented prompting stores interaction history in an external memory system, retrieving the relevant bits as needed. When well designed, this can reduce unnecessary tokens and help the model focus more effectively on truly valuable context. Here, “memory” isn’t some mysterious cognitive process like human memory; rather, it’s an external mechanism designed to process, store, and update historical interactions, retrieving useful information when answering new questions.

This distinction clarifies why agent memory and Retrieval-Augmented Generation (RAG) are similar but fundamentally different concepts. RAG primarily retrieves factual evidence from external document or knowledge bases, supplementing domain-specific information and reducing hallucinations. Agent memory, however, specifically stores information relevant to the current user, ongoing conversations, and long-term interactions, such as user preferences, key events, historical decisions, and task constraints. While the two approaches can complement each other, their focuses are distinct.

Over the years, numerous agent memory methods have emerged, including A-MEM, MemoryBank, MemGPT, Mem0, Mem0𝑔, MemoChat, Zep, MemTree, MemoryOS, and MemOS.

However, the literature points out that the problem isn’t a lack of methods but rather the absence of a unified framework for comparison. Many existing methods report overall performance but rarely dissect the contributions of individual modules. Moreover, systematic comparisons focusing on accuracy and efficiency across methods remain insufficient.

Figure 2 starts to make these differences explicit by categorizing representative methods along four dimensions: information extraction, management operations, storage structure, and retrieval mechanism.

Core Issues with Existing Methods

Current approaches generally struggle in three main areas.

• First, directly placing extensive historical interactions into prompts quickly becomes expensive and unreliable. Context windows get overloaded, resulting in high token usage and making it tough for models to reliably extract meaningful information.

• Second, memory systems aren’t as simple as saving raw text snippets. Effective memory must also handle redundant, conflicting, outdated, or structurally related information. Raw dialogue can be valuable, but it needs to be paired with management operations, such as consolidation, updating, filtering, and linking, so the system can preserve useful detail without drowning retrieval in noise.

• Third, existing methods differ significantly in their extraction, storage, management, and retrieval strategies. Without a unified framework, it is hard to compare the trade-offs among tree structures, graph-based memories, summarization strategies, and retrieval mechanisms under the same experimental setting.

This leads to essential questions: For tasks involving long-term dialogues and multi-session interactions, how can a memory mechanism for LLM-based agents be systematically designed, compared, and evaluated? Such a system needs to reliably store, update, and retrieve historical information within limited contexts and token budgets.

Agent Memory: Core Concept

Here’s a simple way to visualize how an agent’s memory system works: think of it as an organized archive room that sorts itself automatically. Every time a user sends a new message, this archive room first decides which information is worth keeping. Next, it considers whether to merge this new information with existing files, update outdated records, or remove duplicates. Then, it files the information into the most appropriate cabinet, such as short-term storage, long-term archives, a vector database, or a graph-based store. Later, when the user asks a question, the system fetches relevant information from these cabinets and hands it to the LLM.

Read more

The hard part is not finding more papers, but understanding how the ideas actually grew into what we see today.

Why AI Research Still Relies on Human Intuition

Existing academic tools like Google Scholar, Semantic Scholar, and OpenAlex are largely centered around papers themselves. They can show which paper cites which, but they rarely capture how one method evolves from another, what bottlenecks it addresses, or which new directions it opens. Understanding these “method evolution relationships” still largely relies on researchers reading papers and mentally mapping the connections.

Now, AI research agents are starting to participate in the scientific process. Unlike humans, they struggle to reliably reconstruct the evolution of methods from vast amounts of unstructured papers. This limitation hinders their ability to grasp research trajectories, spot gaps, and evaluate or generate new ideas.

What if there were a better approach? Imagine creating an explicit, queryable graph that captures methods, the evolutionary links between them, and evidence showing how specific bottlenecks were overcome. Such a graph could serve as the foundation for AI research agents, giving them a structured understanding of the landscape instead of forcing them to piece it together from raw text.

Read more

Long-document understanding is not just about reading more pages, it is about not losing trust along the way.

Why Long Documents Break Multimodal Models

Currently, existing approaches for long-document question answering generally fall into three categories:

• OCR/Parsing-based Methods: These methods first extract text and layout using OCR or PDF parsers and then analyze the extracted information using LLMs. Examples include LongFormer, LayTokenLLM, HiVT5, DocLLM, and DocVLM. The main challenges are sensitivity to OCR errors and the potential loss of important layout and visual details.

• Visual RAG Methods: These methods initially use visual retrieval to find the top-K most relevant pages before passing them to multimodal large language models (MLLMs) for answering. Representative retrievers include ColPali and DSE, while RAG systems include VisRAG (AI Innovations and Trends 02: VisRAG, GraphRAG, RAGLAB, and More), SV-RAG, VDocRAG, and M3DocRAG (MultiModal RAG Unveiled: A Deep Dive into Cutting-Edge Advancements). The primary issue here is determining the optimal number of pages (top-K): too few pages can miss key evidence, while too many can introduce unnecessary noise.

• End-to-End MLLM Methods: These methods directly input multi-page document images into multimodal large language models for processing. Examples of this approach are mPLUG-DocOwl2 (Let AI Instantly Parse Heavy Documents: The Magic of MPLUG-DOCOWL2’s Efficient Compression), Qwen2.5-VL, and InternVL3. However, handling the large number of visual tokens in lengthy documents leads to a severe low Signal-to-Noise Ratio (SNR), where crucial evidence is buried within vast irrelevant content, negatively affecting both efficiency and overall performance.

In addition, beyond architectural limits, most existing datasets only provide final short answers. Training only on short-answer labels can encourage models to learn brittle shortcuts or dataset-specific patterns, rather than robust, interpretable evidence localization and reasoning.

DocSeeker: Three-step ALR Visual Reasoning

DocSeeker is built around a simple idea: breaking down long-document question answering into three clear steps: first analyzing the question, then locating relevant evidence, and finally reasoning based on that evidence. This is designed to make evidence localization and reasoning explicit, giving users page-level traces that can be checked.

Read more

As a standout among coding tools, Claude Code’s approach to code retrieval grabbed my attention.

After reading its leaked source code, I found that Claude Code tackles this with a clear goal: enabling the model to explore codebases like an engineer. It starts by locating relevant files, narrows down content, reads crucial snippets, and occasionally leverages the Language Server Protocol (LSP) to understand definitions and references.

The notable design choice is that Claude Code does not appear to maintain a persistent full-text index or a vector database for code content by default. Instead, it relies heavily on live ripgrep searches, structured tool calls, and precise file reads.

Quick Background: Terms You Need to Know

Glob patterns are rules for matching file paths. For example, src/**/*.ts means “all TypeScript files under src.” In Claude Code, GlobTool narrows the search space by finding candidate paths. Its implementation lives in src/tools/GlobTool/GlobTool.ts.

Grep handles content searches. It answers the question: “Which files contain lines matching this pattern?” Claude Code’s GrepTool supports three output modes: content, files_with_matches, and count. You can find its implementation in src/tools/GrepTool/GrepTool.ts.

ripgrep, usually invoked as rg, is essentially a fast text search tool. It’s great at searching large numbers of files using regular expressions. Claude Code wraps it in src/utils/ripgrep.ts rather than reimplementing file scanning itself.

Read means precise reading. Search tools such as Grep and Glob usually help indicate “where something might be relevant,” but actually understanding the code requires reading exact file range. Claude Code handles that through FileReadTool.

LSP, or Language Server Protocol, provides IDE-like capabilities such as go- to-definition and find-references. It understands language structure better than plain text search. Claude Code’s LSPTool is useful when a language server is available, but it is not the baseline retrieval mechanism.

Indexing refers to a pre-built structure for quick lookups. Claude Code does have an index, but it primarily supports @file path auto-completion rather than full-text content search. The corresponding implementation is in src/native-ts/file-index/index.ts:43.

Where This Fits in the Tool System

Claude Code exposes search as structured tools, not as arbitrary shell text.

At a high level, the execution path looks like this: Model decides it needs code -> emits a tool call -> toolExecution validates the input schema -> permissions are checked -> GrepTool / GlobTool / Read / LSP runs -> results are bounded and returned -> model decides the next step.

The shared Tool abstraction is defined in src/Tool.ts. It gives each tool a schema, validation logic, permission checks, execution behavior, and metadata such as whether the tool is read-only or safe to run concurrently.

This matters because code retrieval is not only about locating text. It also has to be controlled. A dedicated GrepTool can enforce path rules, output limits, permission checks, and UI grouping in a way that raw Bash commands cannot.

Core Flow: How a Single Code Search Works

When the model needs to find a specific implementation, it might first invoke GrepTool to search keywords.

GrepTool begins by validating inputs like pattern, path, glob, type, and output_mode against a schema. Then, it verifies that the target path exists and checks read permissions through filesystem permission logic in src/utils/permissions/filesystem.ts.

After validation, GrepTool translates user inputs into parameters for ripgrep. There are key details here:

• version-control directories are excluded from normal content search;

• glob and file-type filters can narrow the search;

• context lines can be requested when needed;

• very long lines are capped with a max-column setting to avoid flooding the model context;

• deny rules can be converted into ignore patterns so blocked files do not appear in search results.

That wrapper chooses between available ripgrep implementations and handles process-level concerns such as timeouts, cancellation, buffering, partial output, and exit codes. For example, ripgrep’s exit code 1 means “no match found,” which Claude Code treats as an empty result rather than an error.

If the model knows the filename shape but lacks specific content keywords, it might first use GlobTool. Under the hood, GlobTool enumerates matching paths using rg --files --glob, implemented in src/utils/glob.ts.

Once candidates are found, the model calls Read to inspect exact ranges. This keeps the retrieval loop evidence-driven: search results identify possible locations, while file reads provide the source code the model actually reasons over.

Path Completion: Another Easily Confused Search Mechanism

Claude Code also includes something called FileIndex, but it’s not designed for content searching.

Instead, FileIndex supports path autocompletion when using the @file command in the user interface. You can find the related logic inside src/hooks/fileSuggestions.ts and src/native-ts/file-index/index.ts .

The file suggestion path works differently from content search. It gathers project paths, preferring git ls-files when available and falling back to ripgrep -- files. It then builds an in-memory fuzzy index over paths, using structures such as paths, lowerPaths, charBits, and pathLens.

This design is intentional: users expect immediate responses when typing @xxx, making it worthwhile to maintain a lightweight, in-memory index for path completion.

For content search, Claude Code uses live ripgrep calls through GrepTool, not a persistent content index. A reasonable inference is that this helps keep results aligned with the current filesystem and current read-permission rules, though the source does not explicitly state that as the design motivation.

Key Insight: Why Not Use Retrieval-Augmented Generation (RAG)?

The main reason is that the core problem isn’t about finding “semantically similar” snippets. Instead, it’s about precisely, instantly, and securely finding evidence in the current local codebase.

• A vector-index-based RAG involves this workflow: Chunking code → generating embeddings → storing in vector DB → similarity search → feeding results to the LLM.

• Claude Code uses a very different approach: Glob finds files → Grep precisely searches content → Read extracts snippets → optionally, LSP queries symbols.

Here’s why default vector-index-based RAG isn’t ideal:

1. Dynamic codebase: Files can be created, edited, renamed, or deleted during a session. A vector index would need invalidation, rechunking, re-embedding, and deletion logic. From the source code alone, we cannot say Claude Code rejected RAG for this reason, but the current live-search design avoids that entire class of synchronization problems.

2. Exact matching is critical: A model may need to find MACRO.VERSION, checkReadPermissionForTool, an error string, an environment variable, a CLI flag, or a route path. These are not fuzzy semantic questions. They are often exact-match or regex-match questions, where GrepTool is a better first tool than embedding similarity.

3. Simpler permission handling: Claude Code’s search path checks read permissions and can apply deny rules before returning results. With a vector index, restricted content must either never be indexed or must be reliably filtered at query time, including after permission changes. That is possible, but it is more complex and riskier than applying current deny rules to live search.

4. Verifiable results: Vector retrieval is less directly verifiable. A similarity result may be useful, but the reason it ranked highly is not always obvious. Grep gives concrete file paths and matching lines. Read then fetches the actual source. That makes the model’s next step easier to audit.

So the better summary is not “RAG is bad for code.” It is: Claude Code’s default retrieval path optimizes for live, exact, permission- aware evidence. Vector RAG optimizes for semantic recall over preprocessed corpora. Those are different retrieval problems.

What’s Truly Worth Learning Here?

From reading the leaked code, in my view, rather than building one large retrieval system to handle every query, Claude Code splits the code retrieval problem into smaller mechanisms:

file discovery      -> GlobTool
content search      -> GrepTool
source inspection   -> Read
symbol navigation   -> LSPTool
path completion     -> FileIndex

For an AI coding agent, that is a practical design. The model does not need a perfect global understanding of the repository upfront. It needs a reliable way to gather evidence, test hypotheses, and refine its search until it reaches the code that matters.

But I have a concern about this algorithm.

Repeated broad searches may rescan the filesystem. Search results do not have the kind of semantic ranking a vector system could provide. Grep cannot find conceptually related code unless there is a matching token or pattern. LSP can fill some of that gap, but only when available and applicable. This means Claude Code’s approach is strongest when the task can be grounded in names, strings, paths, symbols, and concrete code evidence. It is weaker when the user asks a vague conceptual question and no obvious search terms exist.

There are also emerging approaches that combine Skills with RAG, and this article takes a look at a representative one.

The Missing Layer in RAG: Navigability

Traditional RAG systems simply retrieve a few similar text passages for the model.

But when dealing with enterprise knowledge bases, the challenge is different. Questions often require the model to understand the structure of the knowledge base, know where to look, backtrack when necessary, and combine evidence across topics.

As a result, the focus is shifting to making knowledge bases not just searchable, but truly navigable.”

Here’s what that really means:

1. Standard RAG is too passive: The model only sees the top-k results and has no sense of what other topics exist in the knowledge base, whether critical documents are missing, or if the current evidence is complete.

2. Complex enterprise questions span multiple topics: A single Wix support question, for example, might touch on account types, payments, legal entities, and verification policies all at once. Simple similarity-based retrieval can return superficially related articles that don’t actually address the key issues.

3. Agentic RAG can search iteratively but lacks a map: Agents can continue issuing searches, but without a corpus map they are still guessing which query terms or search directions will be productive.

4. Existing hierarchical RAGs still hide the hierarchy: Systems like RAPTOR (Advanced RAG 12: Enhancing Global Understanding) or GraphRAG (AI Exploration Journey: Graph RAG) build trees or graphs, but queries still rely heavily on vector search. The model cannot directly “see and browse” the structure.

CORPUS2SKILL: The Agentic Shift from Searching to Navigating

CORPUS2SKILL centers on a simple idea: first, organize the document repository offline into a navigable skill forest, then let an LLM explore it like folders when looking for answers, rather than relying on direct vector retrieval.

Read more

Today, let’s explore a framework for evolving agent skills specifically designed for the multi-user OpenClaw ecosystem.

The Limitations of Existing Skill Methods

Every day, LLM agents interact extensively with users, continuously accumulating experiences. Yet, these experiences rarely contribute meaningfully to enhancing the overall system.

To clarify further:

• Agent skills today are largely static. Typically, skills are manually installed and maintained. Once deployed, they seldom change. Even if an agent discovers better methods through actual usage, these improvements rarely make it beyond the current interaction and rarely enter the skill repository.

• Different users repeatedly encounter similar issues. Users often face similar workflows, call similar tools, and even make similar mistakes like incorrect parameter formats, improper tool invocation sequences, or missing validation steps. However, the system doesn’t integrate these recurrent success patterns or failure modes across users and over time. As a result, many users end up rediscovering similar fixes independently.

• Existing solutions fall short. Memory-based methods typically just store historical records for easier retrieval, but they struggle to abstract these into genuinely reusable or improvable skills. Conversely, skill-based methods do structure experience into skills, but these repositories are usually static and do not evolve naturally through ongoing use.

If individual users’ fragmented experiences from multiple interactions could be transformed into skills shared and continuously enhanced across the entire multi-user agent system, the impact would be significant. Validated improvements discovered in one context can be propagated through the shared skill repository, so future agents that use the relevant skills can benefit from collective experience.

SkillClaw's Architecture: A Closed-Loop Pipeline for Collective Evolution

As shown in Figure 2, SkillClaw gathers interaction traces from daily multi-user agent usage, evolves them into improved skills, and then synchronizes these enhanced skills back to all agents, forming an automated feedback loop.

Read more

Where Autoregressive OCR Starts to Break Down

Most existing OCR and visual language models (VLMs) rely heavily on autoregressive decoding, meaning they generate text tokens sequentially, one by one, from left to right.

While this approach works well for standard text generation tasks, it’s far from ideal for document OCR. Here’s why:

1. Speed Issues: Documents, especially lengthy ones filled with tables, formulas, and complex layouts, require generating many tokens. Decoding each token sequentially leads to significant latency, slowing the entire recognition process.

2. Error Propagation: Autoregressive methods are highly sensitive to early mistakes. A single recognition error can distort the context for subsequent tokens, causing a cascade of inaccuracies that build upon one another.

3. Over-Reliance on Language Priors: In Semantic Shuffle benchmark, AR models often lean heavily on linguistic cues and semantic coherence. This means they may “guess” rather than clearly perceive the actual text. When the semantic structure is disrupted or ambiguous, AR performance typically drops dramatically.

4. OCR as Inverse Rendering: Fundamentally, document OCR is better thought of as “inverse rendering.” The goal is to reconstruct structured information (like text, layouts, tables, and equations) from a two-dimensional image. The correct interpretation primarily depends on visual evidence and spatial arrangements. Forcing a strict left-to-right serialization is merely an "implementation artifact" for representation convenience, rather than a fundamental property of how documents are actually structured.

5. A Strong Fit for Diffusion: Unlike open-ended text generation (like chatting with ChatGPT), OCR is a near-deterministic task with limited semantic ambiguity. This makes OCR a strong candidate for masked diffusion, where masked tokens can be predicted in parallel conditioned on the image and partially observed sequence, producing a tunable speed–accuracy trade-off.

Given these considerations, document OCR systems would greatly benefit from decoding strategies that are parallelized, globally consistent, and strongly grounded in visual features. Rather than forcing OCR into the sequential patterns of autoregressive language generation, it’s more natural to employ methods designed specifically to exploit visual structure.

MinerU-Diffusion: From Left-to-Right OCR to Parallel Visual Decoding

MinerU-Diffusion uses diffusion-based decoding instead of the traditional autoregressive method, enabling the model to simultaneously confirm or correct multiple tokens through visual context. This approach boosts processing speed, reduces error propagation, and decreases reliance on linguistic context for guessing content.

The method can be understood through four practical components.

1. Unified Output Format

Text, layout annotations, table symbols, and formula indicators are all represented as a unified sequence of tokens.

For document parsing, the model outputs a structured sequence rather than only plain text; task-specific prompts can still produce plain text, LaTeX, or table markup.

2. Diffusion-Based Decoding Replacing Autoregression

During training, tokens are randomly masked, prompting the model to predict these masked elements based on the surrounding context and visual evidence from the document image.

At inference, the model progressively reconstructs masked positions, using already decoded context and visual features rather than generating strictly left to right. Over multiple iterative rounds, uncertain tokens are progressively revealed and corrected rather than sequentially generating each token from left to right.

3. Block-wise Diffusion

Diffusing across an entire document sequence can be slow and unstable, so sequences are divided into smaller blocks:

• Within blocks: Diffusion is parallelized, and context is considered bidirectionally.

• Between blocks: A coarse, front-to-back dependency helps preserve sequence coherence and reduce long-range drift.

• System Efficiency: The causal (front-to-back) structure across blocks naturally enables efficient KV-caching during inference, reducing memory and computation costs compared to full-attention diffusion models.

This design maintains fast parallel decoding while mitigating position drift and error accumulation common in lengthy documents.

4. Confidence-Driven Dynamic Decoding + Two-Stage Training

During inference, tokens with high confidence are confirmed first, while low-confidence tokens undergo further iterative correction. Confidence thresholds balance decoding speed and accuracy.

After multimodal initialization, training happens in two stages: initial broad-scale training provides general capabilities, followed by an uncertainty-driven refinement. The model automatically mines challenging examples (like complex tables or ambiguous boundaries) by measuring its own inference consistency, focusing its learning on the hardest cases to enhance robustness.

In short, MinerU-Diffusion treats document OCR as the inverse problem of reconstructing structured text from images, leveraging block-wise diffusion to parallelly refine tokens, and employing confidence-driven scheduling and challenging-case training to boost decoding speed, stability, and reliability.

Evaluation

Document Parsing Evaluation

MinerU-Diffusion’s capability in full-page document parsing is evaluated using OmniDocBench v1.5, measuring its performance through various metrics such as text edit distance, formula correctness (CDM), table extraction quality (TEDS), and reading order.

The results showed that MinerU-Diffusion achieved an overall score of 88.94 without using ground-truth layouts. When provided with ground-truth layouts, the score improved significantly to 93.37, coming very close to the performance of strong autoregressive OCR systems. This mainly shows that once layout errors are removed, its recognition quality is highly competitive.

Efficiency Evaluation

Efficiency was tested by adjusting the confidence thresholds, which determine how many tokens the model finalizes in a single decoding step. Lower thresholds led to faster decoding speeds, while higher thresholds improved stability.

MinerU-Diffusion achieved up to a 3.2× decoding speedup, maintaining a clear advantage in speed even at high accuracy levels.

Thoughts

At its core, MinerU-Diffusion transforms OCR decoding from sequential token-by-token generation into a visually-driven, block-wise diffusion process: tokens are refined in parallel within each block, while blocks retain a coarse front-to-back dependency.

Coupled with uncertainty-driven curriculum training, this shift represents a fundamental change at the decoding paradigm level, not merely swapping out the underlying model backbone.

But I have a concern.

Block boundaries could introduce new sources of subtle errors. While MinerU-Diffusion mitigates this by allowing tokens to causally attend to preceding blocks, they are strictly cut off from future blocks. Structures like headers, footers, table cells, or formulas spanning line breaks might still be disrupted if they fall near these boundaries. Such systemic fragmentation might not clearly surface through averaged evaluation metrics.

Reference:

• Paper: MinerU-Diffusion: Rethinking Document OCR as Inverse Rendering via Diffusion Decoding.

• Code: https://github.com/opendatalab/MinerU-Diffusion.

That story holds up right until a long industrial PDF arrives on your desk and makes one thing obvious: documents were never meant to be understood as confetti.

The Hidden Pitfalls of Naive RAG Chunking

Many RAG systems retrieve relevant chunks from documents and then ask an LLM to answer from that retrieved context.

However, there’s a catch: to make documents searchable, especially longer ones, they usually have to be split into smaller pieces. Traditional methods for splitting these documents can be overly simplistic, such as slicing by a fixed number of words or strictly relying on semantic meaning.

In practice, this approach often leads to problems. Real-world documents aren’t simply chunks of plain text, they often include:

• Multiple levels of headings and subheadings

• Tables, figures, and other layout-heavy elements

• Content spanning across multiple pages

• Scanned PDFs that require OCR processing

When documents are just “split by length,” it’s akin to randomly chopping up a textbook, headings end up isolated from their sections, paragraphs get separated from accompanying tables, and critical context can get lost altogether.

This fragmentation can make retrieval return disjointed evidence, which can reduce answer quality. The problem becomes especially pronounced when dealing with industrial documents, financial reports, legal contracts, and scanned materials.

Core Idea

To make RAG over long documents more faithful, three kinds of signals need to be modeled together:

• Visual layout (what the document looks like)

• Textual content (what the document says)

• Structural hierarchy (how the document is organized)

As shown in Figure 1, MultiDocFusion combines visual layout, text, and hierarchy; the name “Fusion” refers to that integration, while “MultiDoc” also reflects support for diverse document formats and corpus-level multi-document RAG scenarios.

An Intuitive Analogy

Traditional fixed-length methods are more like slicing a book at fixed token boundaries.

MultiDocFusion, on the other hand, acts like an intelligent reader: it dynamically reconstructs the document's structural tree (like building its own table of contents on the fly) and carefully groups content that naturally belongs together.

This makes it particularly effective for handling long, complex documents that reflect real-world complexities.

MultiDocFusion: How Does It Actually Work?

Think of it as a four-step process.

Step 1: Understanding the Document Layout (Document Parsing)

Read more

Today, the focus shifts to a new method for acquiring those skills.

Why Agents Need a Master Manual, Not a Pile of Sticky Notes

A skill is more than just a single file, it's a structured knowledge directory containing a root SKILL.md (the instruction manual) alongside auxiliary resources like executable scripts and reference files.

As agents tackle more specialized domains, the demand for highly specialized, comprehensive skill documentation grows, but human production simply cannot keep up.

There are also two major challenges:

First, asking a large language model to generate skills purely from its parametric knowledge often misses the real pain points, operational nuances, and common failure modes of a target domain. The resulting skills might look plausible on paper, but they rarely provide meaningful help in actual tasks.

Second, many existing online update methods based on trajectory data tend to fragment knowledge and overfit to local experience.

• Fragmentation: Many approaches extract a lesson from each trajectory, ending up with a heap of disconnected skills that are hard to search and apply.

• Sequential updates: Updating skills after every new trajectory is like trying to revise a guide without ever fully understanding the domain. It’s easy for a single trajectory to skew the knowledge.

Read more

In this post, let’s take a look at its latest progress.

The Key Constraint in Document Parsing

A recurring pattern has emerged: state-of-the-art models with different architectures and parameter sizes often make almost identical mistakes on challenging real-world PDF samples.

Simply switching to a larger model is unlikely to solve the problem. Instead, the bottleneck lies in a shared set of data-related challenges that limit progress across the board.

As a result, the key constraint on further advances in document parsing may no longer be model architecture, but the training data itself.

This data bottleneck has two main dimensions:

• First, insufficient coverage. Training datasets contain plenty of common page layouts, but long-tail scenarios such as complex nested tables, dense mathematical formulas, and unconventional multi-column layouts remain underrepresented.

• Second, there is an annotation quality paradox: many of the most informative samples, especially the difficult ones, are exactly the cases where automatic annotation is least reliable. Because of this, simply scaling raw data volume is insufficient: without better sampling and more reliable labels, it can amplify both distribution bias and annotation noise.

Given these challenges, an important question arises: if the architecture remains unchanged, can systematic data engineering and staged training still deliver significant gains, and can a more discriminative evaluation protocol reveal those gains more faithfully?

More provocatively: can a 1.2B-parameter model rely purely on data to outperform giants with over 200x more parameters?

MinerU2.5-Pro: Data Engine

Read more

They know what to do, just not always in the right order or at the right time.

Why Skills are Needed?

Over the past two years, discussions around Agents have often focused on bigger models, longer context windows, and more tools.

But the real challenge in production-level Agents isn’t whether the model can do something, it’s whether it can reliably complete a task the same way every time.

As Agent’s tasks grow longer, environments become more complex, and branches multiply, language models tend to drift. Steps get skipped, sequences get jumbled, tools are used inconsistently, and stopping conditions vary.

These challenges are known as procedural burden of agency.

Before diving into the introduction of Skills, let’s take a look at where they fit within an Agent.

Skills

Skills were created as an externalization response to this procedural burden.

A skill is not just a tool, and not merely a prompt file; it is an externalized form of procedural expertise. Tools answer the question “what actions can be performed,” protocols answer “how these actions are described and invoked,” and a Skill answers “how this type of task should be handled as a whole.”

A truly reusable Skill has at least three layers: the operational flow, decision heuristics, and governing constraints. In other words, it not only tells an Agent what to do first and what to do next, but also guides which path to take when there are branches, and defines which boundaries must never be crossed.

Read more

This post offers some interesting ideas.

Motivation

Hallucination remains a critical bottleneck for LLMs, undermining their reliability in real-world applications, especially in Retrieval-Augmented Generation (RAG) systems.

There are three core challenges:

• First, even with access to documents, models can misreport numbers, mix up timelines, or make reasoning jumps that ignore the evidence. In fields like finance, law, and healthcare, these mistakes can have serious consequences.

• Second, existing training signals are too coarse. Traditional supervised fine-tuning may teach a model to “sound convincing,” but not to ensure every fact is accurate. Standard reinforcement learning methods, including RLHF, usually provide a single score for the final answer. Even newer approaches like RL with Verifiable Reward (RLVR) are bottlenecked by the scarcity of expert annotations and the limited reasoning ceilings of external verifiers.

• Third, current verifiers suffer from confirmation bias due to "information leakage". When a judge sees the question, the source documents, and the model’s answer at the same time, there’s a strong tendency to justify the original response rather than independently verify it. This is a fundamental flaw in many hallucination detection methods.

MARCH: Three Agents, One Goal for Tighter Grounding and Fewer Hallucinations

MARCH is a framework that separates answer generation from answer verification and intentionally creates information asymmetry.

Read more

Why Traditional OCR Is No Longer Enough

Traditional OCR and many document parsing pipelines are still largely text-centric: they recover text well, but usually do not structurally parse information-dense graphics.

When confronted with charts, diagrams, icons, or UI components, they usually treat these elements as mere image fragments and store them as pixel blocks. As a result, a significant portion of structural and semantic information within documents is lost.

Figure 1 captures the main shift: traditional OCR keeps graphics as raster regions, while Multimodal OCR represents text and eligible visual symbols in a unified serialized format, using plain text, table markup, LaTeX, or SVG depending on the element type.

MOCR: Not Just Reading Text, but Reconstructing Graphics

MOCR, short for Multimodal OCR, aims to parse anything found in documents.

Read more

This post might offer an insightful idea.

Motivation

There are two common approaches to creating methodology diagrams and statistical plots for research papers today, and neither of them quite gets the job done.

• The first is code-based drawing, using tools like TikZ, Python-PPTX, or SVG. This approach works well for structured diagrams and precise layouts. But it starts to fall short when dealing with the kinds of visuals that show up in modern papers, things like intricate icons, custom shapes, and polished, detail-heavy designs. Flexibility becomes a real limitation.

• The second approach is to rely on image generation models. These can produce visually appealing results with very little effort. The problem is consistency. It is hard to guarantee that the output meets the standards of academic work, both in terms of factual accuracy and visual conventions. A figure might look good at first glance, but still require careful checking to ensure logical faithfulness and academic readability.

In practice, this means researchers often end up spending a significant amount of time manually fixing figures anyway, even after using these tools.

PaperBanana: Read, Plan, Stylize, and Sketch Science

PaperBanana is an agentic, reference-driven framework that tries to bridge this gap by synthesizing aesthetic guidelines from curated academic examples.

Read more

This post is interesting because it introduces a method that tackles that gap in a very practical, engineering-first way.

The Problems of Standard RAG

Standard RAG is often not reliable enough for hardcore tasks, like the WattBot 2025 Challenge, which demands ±0.1% numeric precision and exact source attribution. In practice, three issues show up again and again:

• Flat chunking breaks semantic boundaries and document structure, which makes precise source tracking much harder and weakens confidence in citation quality.

• A single query can easily miss relevant evidence when the wording does not line up, especially when different terms are used for the same idea.

• A single generation pass is inherently unstable. Both the answer and its citations can vary from run to run, and in some cases, terrified of hallucinating, the system may unnecessarily abstain (outputting a blank) despite the evidence being buried right there in the context.

KohakuRAG: From Flat Chunks to Citation-Ready Trees

At its core, KohakuRAG is a lightweight RAG framework built on a four-level hierarchical document index: document → section → paragraph → sentence.

It preserves document structure through bottom-up embedding aggregation, improves retrieval coverage with LLM-based query planning plus cross-query reranking, and stabilizes final outputs with abstention-aware ensemble voting during generation.

1. Offline hierarchical document indexing

The first step is document parsing.

Read more

The real shift is this: OCR is no longer just a reading task, but a document understanding problem.

Modern OCR Is About Understanding Documents

In the past, OCR usually meant one thing: recognizing text from an image.

Today, the scope of OCR has expanded far beyond basic character recognition. Real-world document processing rarely stops at reading text. It often involves layout analysis, table parsing, chart understanding, question answering, and extracting key information from complex pages. Documents are no longer just lines of text. They contain structure, hierarchy, and meaning that must be understood as a whole.

Because of this shift, OCR has gradually evolved from a character recognition tool into something much closer to a document intelligence system. Instead of only asking what characters are on the page, modern OCR systems are expected to understand how the page is organized and what the content actually represents.

Three Paradigms of Modern OCR

Current OCR paradigms can roughly be grouped into three categories:

• Pipeline OCR systems

• End-to-end OCR models

• General vision-language models.

These three approaches reflect different stages in the evolution of OCR, and each comes with its own set of trade-offs.

Traditional Pipeline OCR: Strong in Control, Weak in Fragmentation

The most classic design is the pipeline OCR system. A typical workflow starts with layout detection, followed by element-level recognition, and finally a rule-based step that assembles everything into the final output.

Read more