This post introduces pi-intelli-search [1], an open source web research extension for the Pi coding agent: a five-stage pipeline, invoked through a single tool call, that searches via Perplexity Sonar, fetches each result two ways and keeps the cleaner version, runs a per-page Large Language Model (LLM) extraction guided by a focus prompt, collates the extractions into a deduplicated summary with conflict flags, and caches everything to a local .search/ directory indexed for reuse.
Its novel feature is a dedicated secondary LLM that distils each fetched page down to its query-relevant content. The primary agent receives a focused ≈5,000-character summary instead of approximately 50,000 characters of raw content per page (10x reduction). This reduces context consumption, removes irrelevant material that can derail reasoning, lowers downstream token cost, and improves answer quality. Yet none of the widely installed Pi search extensions include either stage. They reproduce a design inherited from hosted agents, shaped by constraints that open agents do not share.
Introduction
In technical contexts (e.g. programming, data science, analytics etc), real projects routinely consume tens of millions of tokens across the lifetime of a single workstream. The industry benchmarks that quantify hallucination rates operate at a different scale entirely. SimpleQA [2], the most widely cited, spans ≈150,000 tokens across 4,326 short, independent, one-shot factual lookups. Even on that narrow evaluation, the best frontier models answer incorrectly 35-40% of the time. At project scale, with two orders of magnitude more token consumption and decisions that depend on one another rather than standing alone, hallucinations are not a risk but an issue that requires mitigation.
On SimpleQA without web access, GPT-5 (thinking mode) hallucinates on ≈40% of the questions it attempts [3]. On the broader AA-Omniscience benchmark [4], GPT-5.5 achieves the highest accuracy of any model tested (57%) but also the worst calibration: an 86% hallucination rate on questions it should abstain from [5]. Claude Opus 4.7 hallucinates on 36% of AA-Omniscience questions, an improvement driven substantially by increased abstention rather than higher accuracy [6]. None of these benchmarks measure error rates across dependent, multi-step reasoning.
Subtly wrong identifiers, plausible-but-fictional API signatures, hand-wavy version assertions, and confidently misremembered defaults accumulate. Over a project’s lifetime, the cost of these errors exceeds the cost of grounding the agent’s context with current web content. None of them stand out against the breadth of technical material in a real project. They blend in, get committed, get deployed, and surface weeks later in production.
This is what makes web research an indispensable part of any serious Agentic workflow. Search and Fetch are the mechanism by which the agent grounds its claims in something that exists outside its own model weights. If the search tooling delivers raw page dumps, the primary agent must filter relevance with its own reasoning, in the same context window that holds the rest of the work. If it delivers focused, deduplicated, source-attributed summaries, the agent has more context for the actual problem and a smaller surface for hallucinated synthesis.
The most-installed search extensions on pi.dev/packages [7] reproduce, with minor variations, the same pattern: search the web, fetch a page, return the cleaned content directly to the agent’s context. This is precisely how Claude Code [8] and OpenAI Codex [9] do it. The pattern is so widespread it reads as the obvious default, but it is shaped by constraints that hosted coding agents face and open coding agents like Pi, OpenCode, and Hermes do not. Where hosted agents are constrained, open agents have an opportunity to innovate.
pi-intelli-search exploits this opportunity by placing a per-page extraction LLM and a cross-source collation LLM between the web and the primary agent’s context. The individual components are off-the-shelf, but the architecture is novel because an open agent’s operating environment makes it both technically and economically feasible.
The Five-Stage Pipeline
intelli_research runs a self-contained pipeline inside a single tool call. The five stages are deliberate, each relying on capabilities that hosted coding agents typically cannot afford to use, in terms of capacity or cost.
| Stage | Default Model | Cost per Call | Notes |
|---|---|---|---|
| Search | Perplexity Sonar via OpenRouter | ≈$0.02 | Returns synthesised answer plus source URLs. We primarily use the sources URLs for Fetching but the answer is injected into the collation. |
| Fetch | N/A (HTTP and Markdown endpoint) | $0.00 | Dual-fetch in parallel, quality-scored, best version kept |
| Extract | MiniMax M2.7 via OpenRouter | ≈$0.03 (8 calls, 4 concurrent) | Per-page LLM extraction, guided by focusPrompt |
| Collate | MiniMax M2.7 via OpenRouter | ≈$0.005 | Cross-source deduplication, conflict flagging, source attribution |
| Cache suggest | MiniMax M2.7 via OpenRouter | ≈$0.0002 | LLM judge surfaces related previous searches |
| Model | Perplexity Sonar via OpenRouter |
| Cost | ≈$0.02 |
Returns synthesised answer plus source URLs. We primarily use the source URLs for Fetching but the answer is injected into the collation.
| Model | N/A (HTTP and Markdown endpoint) |
| Cost | $0.00 |
Dual-fetch in parallel, quality-scored, best version kept.
| Model | MiniMax M2.7 via OpenRouter |
| Cost | ≈$0.03 (8 calls, 4 concurrent) |
Per-page LLM extraction, guided by focusPrompt.
| Model | MiniMax M2.7 via OpenRouter |
| Cost | ≈$0.005 |
Cross-source deduplication, conflict flagging, source attribution.
| Model | MiniMax M2.7 via OpenRouter |
| Cost | ≈$0.0002 |
LLM judge surfaces related previous searches.
Four capabilities distinguish this pipeline. Open agents face no shared infrastructure limits, so additional LLM calls per query are feasible rather than prohibitive. The user selects models freely across providers, and capable models routinely undercut the economy tiers that hosted agents are locked to. Per-page extraction, guided by a focus prompt, removes irrelevant material before it reaches the primary agent’s context, reducing noise and improving downstream reasoning. Persistent local caching provides redundancy against hallucination by letting the agent cross-reference previous searches, and an individual researcher’s cache occupies a fundamentally different copyright position from large-scale systematic extraction.
Stage 1: Search
The search backend is Perplexity Sonar, a search-grounded model that returns a synthesised answer with inline citations rather than a bare list of URLs. Sonar is routed through OpenRouter [10] rather than the Perplexity Application Programming Interface (API) directly. OpenRouter exposes Sonar alongside thousands of other models on a single pay-as-you-go account, which avoids the Perplexity API’s $50 minimum credit top-up and consolidates spend on an account already used widely across the open coding-agent ecosystem.
The search model is configurable via the searchModel setting. If a future model surpasses Sonar for grounded search, the architecture swaps it through configuration without changing anything else.
Stage 2: Fetch
Each candidate URL is fetched twice in parallel. The first fetch uses a browser-grade Transport Layer Security (TLS) fingerprint via wreq-js [11] and pipes the resulting HyperText Markup Language (HTML) through Defuddle [12], which strips navigation, advertising, sidebars, and similar chrome (the non-content elements that surround the actual text on a web page) to produce clean Markdown. The second fetch requests the page’s Markdown variant directly using Accept: text/markdown, an <link rel="alternate"> discovery, or a .md suffix. Both candidates are scored on substantive content density (code blocks, headings, tables, prose) versus chrome noise, and the higher-scoring version is retained.
Server-rendered Markdown is not guaranteed to be cleaner than Defuddle-extracted HTML. Cloudflare’s documentation, for example, returns a Markdown variant that includes JSON Linked Data (JSON-LD) breadcrumb schema and Schema.org markup; Defuddle on the same page strips those artefacts and produces tighter output. Without a comparison step, the extension would have to guess. With it, the better source wins on every fetch.
For sites that publish an llms-full.txt file, the extension checks each fetched domain automatically by requesting https://domain/llms-full.txt. If the file exists, it is downloaded raw to the cache without LLM processing, where the agent can read or grep it directly. A small built-in list handles sites with non-standard paths (e.g. Cloudflare, Next.js, Vite).
llms-full.txt?
llms-full.txt is a convention where websites publish a single plain-text file containing their complete documentation in a format optimised for LLM consumption. It serves a similar purpose to robots.txt (which tells search engine crawlers what they may access) but is designed for language models rather than traditional web crawlers. When an llms-full.txt file is available, it typically provides cleaner, more complete content than any HTML extraction can achieve.
Stage 3: Extract
Each fetched page is passed to a configurable extraction LLM along with the original query and an optional focusPrompt. The default model is MiniMax M2.7 [13], chosen for its low cost and strong instruction-following. The extraction prompt instructs the model to keep query-relevant content verbatim (especially code, signatures, and exact figures), discard navigation and tangential discussion, and adapt to source type: official documentation preserves API surface, blog posts capture practical patterns, forum threads capture the accepted solution.
A cleaned 50,000-character page becomes approximately 3,000 to 5,000 characters of focused content after extraction. Eight pages, extracted four at a time through a bounded worker pool, deliver approximately 24,000 to 40,000 characters of relevant material to the next stage, comfortably inside any modern context window, and crucially never reach the primary agent in their raw form.
focusPrompt Parameter
Without a focusPrompt, the extraction model works generically and the collation step has less signal to work with. With one (“extract only the consistency model and transaction API”, “extract limits, timeouts, and error messages”), the extraction model knows what to keep and what to discard. The agent translates the user’s intent into the focus prompt before calling the tool, so the specificity of the extraction scales with the specificity of the original question.
Any model Pi can access works as the extraction model: built-in providers like OpenAI and Anthropic, alternative OpenRouter models, or models registered by other extensions. The choice is controlled by the extractModel setting.
Stage 4: Collate
The per-page extractions (eight at the default defaultUrls setting, up to sixteen at the maxUrls cap) are passed to a collation LLM (also MiniMax M2.7 by default, configured via collateModel) along with the original query. The collation prompt produces a single concise summary, deduplicates overlapping claims across sources, flags conflicts where sources disagree, and preserves source URLs for attribution. When sources contradict one another, the prompt instructs the model to prefer official documentation over reference material, reference material over tutorials, tutorials over blog posts, and blog posts over forum threads, surfacing the contradiction explicitly rather than silently choosing.
This is the second LLM stage that the agent’s primary context never has to absorb. Without it, the primary agent would be doing this synthesis itself, in the same context window holding the rest of the work, and against eight separately formatted sources.
Stage 5: Cache and Suggest
Every completed research session is persisted to a .search/ directory on the user’s filesystem. The structure is designed so that both the agent and the user can navigate it directly:
.search/
├── 2026-04-19-d1-worker-api-3f7a2c/
│ ├── report.md # Collated summary + source index
│ ├── query.txt # Original search query
│ ├── meta.json # Local-only telemetry sidecar
│ ├── extractions/ # Per-page LLM extractions (≈3–5K each)
│ │ ├── 01-developers-cloudflare-com.md
│ │ └── 02-developers-cloudflare-com.md
│ └── sources/ # Full page content
│ ├── 01-developers-cloudflare-com.md
│ ├── 02-developers-cloudflare-com.md
│ └── llms-full-developers-cloudflare-com.md
└── .index.json # Index of all cached searchesEach session directory carries a short hash of the full query (the -3f7a2c suffix above) so two queries that reduce to the same date and slug no longer overwrite one another. The agent receives the collated report.md in its context, but the full source content and per-page extractions remain available on disk. When the summary is insufficient, the agent can read or grep the original sources without re-fetching. The cache location is configurable via the cacheDir setting (default .search), so project-specific and shared research directories are both supported.
This cache is a compounding asset. Repeated queries on the same topic become both faster and cheaper, follow-up research can cross-reference previous reports, and contradictions against earlier findings surface naturally. No other Pi search extension retains full source content alongside structured extractions.
After each session is cached, a lightweight LLM judge compares the current query against up to twenty recent entries in .index.json and returns semantically related previous searches. These appear in the tool output as a 📚 Related cached searches table appended below the live results. The suggest stage is purely additive: it never blocks or replaces the live pipeline, and failures are silently ignored. Cost is approximately $0.0002 per query.
Hosted Agent Constraints
No widely installed Pi search extension includes per-page extraction or cross-source collation. The dominant design was copied from a class of agents that cannot afford them. Hosted coding agents serve millions of concurrent users from shared inference infrastructure. Three constraints shape how they handle web search: compute economics, context budgets, and copyright liability. Each pushes the same direction: do as little work per query as possible, return the rawest content you safely can, and store nothing.
Compute Economics at Scale
A hosted agent’s WebSearch tool typically returns a list of results with short snippets, and a WebFetch tool returns the cleaned content of a single page. Neither runs an additional LLM call to refine what is delivered to the agent. The reason is straightforward unit economics. At scale, every additional inference call multiplies infrastructure cost by the user base. Recent capacity pressure on the hosted coding agent market has been visible enough to be discussed publicly, with providers throttling requests, queueing sessions, and raising limits only after securing large-scale compute deals. In that environment, an extra LLM pass per fetched page is a luxury the unit economics do not support. The same pressure shapes context budgets: returning a 50,000-character page burns context the user is paying for, but running an extraction pass costs additional inference. Most hosted agents settle on a middle ground of truncated or summarised snippets rather than full pages, and the agent is expected to filter relevance itself, in the same context window holding the rest of the work.
Copyright Challenges
Hosted services serving millions of users have a different copyright posture from individual researchers. Reproducing substantial passages of third-party content at scale carries reputational and legal exposure that platform providers manage by limiting both the size of returned snippets and the persistence of cached content. An open extension caches pages to the user’s own machine for the user’s own research. The position is closer to a browser cache or a saved-articles folder than to a commercial content distributor. The two situations call for different design choices around quote length, full-page retention, and offline reuse.
Combined, the three constraints push hosted-agent web search toward a design that does as little processing as possible, returns truncated or partial content, and stores nothing across sessions. This is a sensible design for the constraints it serves. The design does not, however, generalise to a different operating environment, yet it often travels there anyway.
The Open Agent Advantage
Pi is an open coding agent. It runs on the user’s selected machine, calls models through the user’s own API keys, and reads and writes the user’s own filesystem. The same applies to OpenCode, Hermes, and others in this class. None of the three constraints above apply.
Compute is paid per call, by the user, against whichever provider they choose. An additional LLM pass per fetched page costs cents, not aggregate dollars across millions of users. The freedom to shop across providers matters more than it first appears. MiniMax M2.7 costs approximately $0.30 per million input tokens and $1.20 per million output tokens. Claude Haiku 4.5, the economy-tier model within Anthropic’s own lineup, costs $0.80/$4.00 for the same volume. A full state-of-the-art model from another provider can cost less than the cheapest model a hosted agent is locked to.
Context is the user’s own concern. If a sub-pipeline can deliver a 5,000-character focused summary instead of a 50,000-character page dump, the primary agent has more headroom for actual reasoning. At the hallucination rates frontier models exhibit on short factual lookups alone, removing irrelevant material before it reaches the primary context is not an optimisation but a direct mitigation. The trade-off shifts decisively in favour of pre-processing.
Caching is local, and given the hallucination rates established above, it serves as redundancy. A .search/ directory on the user’s own disk accumulates previous search results, extractions, and collated reports. The agent can cross-reference past queries, catch contradictions against earlier findings, and avoid repeating the same errors across sessions. An individual researcher caching web content to their own machine for their own use occupies a fundamentally different legal position from a platform systematically reproducing third-party content at scale. There is no redistribution, no platform-level reproduction risk, and no reason to truncate the cached source content.
The extraction and collation models are not infallible; they can miss relevant content or introduce errors of their own. Because the cache retains the full original sources alongside the LLM-processed extractions, the primary agent can fall back to the raw content whenever a summary looks incomplete or suspect. The architecture does not require the secondary models to be perfect. It requires them to be useful most of the time and verifiable all of the time.
These are the conditions pi-intelli-search was designed for. The rest of this post covers the cost structure, comparative analysis, and setup.
Cost and Model Selection
The total cost per research session at default settings, with eight pages, is approximately $0.05. The breakdown is:
| Step | Calls | Cost |
|---|---|---|
| Search (Sonar) | 1 | ≈$0.02 |
| Fetch (Defuddle + Markdown endpoint) | 8 parallel pairs | $0.00 |
| Extract (MiniMax M2.7) | 8 parallel | ≈$0.03 |
| Collate (MiniMax M2.7) | 1 | ≈$0.005 |
| Cache suggest (MiniMax M2.7) | 1 | ≈$0.0002 |
Five cents per session is not free, and that is the trade-off being made: paid extraction and collation in exchange for focused, deduplicated, cached results. Extensions without LLM processing are nominally free but the agent then spends its own reasoning tokens (and the user’s context budget) sifting raw content. The cost is moved rather than eliminated, and on the lifetime token budget of a serious technical project, the moved cost is the larger one.
The five-cent figure is reachable because the user is free to choose across providers. DeepSeek V4 Flash, Qwen 3.5-Flash, Gemini 2.0 Flash Lite, and GPT-4.1 Nano all offer one-million-token contexts in a similar or lower price band to MiniMax M2.7 ($0.30/$1.20 per million tokens). Several of these are cheaper than the economy-tier models hosted agents are limited to, and some full state-of-the-art competitors (e.g. DeepSeek V4 Pro) undercut them further still. Any of these can be substituted via the extractModel and collateModel settings. The economics of an extra LLM pass per fetched page only work when the user can choose a model appropriate to the task, and that choice is precisely what hosted agents cannot offer.
Comparative Analysis
A detailed feature-by-feature comparison against the six other most-installed Pi web search extensions is published in the project repository. The summary, across search backend, fetch strategy, LLM extraction, LLM collation, persistent caching, and cache suggestion, is that pi-intelli-search is the only extension among those compared with per-page LLM extraction, the only one with cross-source LLM collation and conflict detection, and the only one with a persistent indexed cache and cache suggestion. The other extensions take the same shape: search, fetch, return content. They are well-built tools for what they do. The point is not that they are wrong, but that the design space they occupy was inherited from a different operating environment.
Components and Setup
The extension is published on npm as @curio-data/pi-intelli-search [1] and installs through Pi’s extension system:
pi install npm:@curio-data/pi-intelli-searchIt requires Pi 0.80.8 or later. With default settings, a single OpenRouter key covers all three pipeline stages: Perplexity Sonar for search and MiniMax M2.7 for extraction and collation. On Pi 0.82.0 and later, running /login openrouter performs OAuth PKCE sign-in and stores a user-controlled key automatically with no manual key paste required. Alternatively, a key from openrouter.ai/keys can be added to ~/.pi/agent/auth.json directly. Overriding extractModel and collateModel to use a different provider (e.g. OpenAI, Anthropic) routes those stages through that provider’s key instead. All settings are namespaced under a pi-intelli-search block in ~/.pi/agent/settings.json or the project-local .pi/settings.json.
The runtime dependencies are minimal:
| Component | Purpose |
|---|---|
wreq-js |
Browser-grade TLS fingerprinting for page fetching |
defuddle |
HTML to Markdown content extraction |
linkedom |
Lightweight Document Object Model (DOM) for defuddle’s Node.js mode |
@earendil-works/pi-ai |
LLM dispatch through Pi’s native auth system |
@earendil-works/pi-coding-agent |
Extension API surface |
typebox |
JSON Schema and tool-input parameter typing |
On first load, the extension patches ~/.pi/agent/models.json to register the Perplexity Sonar models under the openrouter provider. The patch merges by model identifier, is non-destructive, and is idempotent across reloads.
Limitations
The verification-first methodology applied to a previous open source project, the unofficial Podman snap package, would be excessive for a tool of this size. The current test suite is 337 tests (unit and end-to-end), with each end-to-end script exercising a distinct pipeline behaviour in an isolated Pi environment. Every LLM call also retries transient failures (HTTP 429, 5xx, timeouts) with full-jitter exponential backoff under a hard per-call timeout, so provider rate limiting degrades gracefully instead of hanging the pipeline, which matters most on free-tier OpenRouter keys. That said, the boundaries of the current design are worth being explicit about.
- The pipeline depends on the extraction model identifying relevant content correctly. A weak extraction model will miss key details or introduce errors. The cache mitigates this: because full source content is retained alongside the LLM-processed extractions, the primary agent can fall back to the raw pages whenever a summary looks incomplete. The configurable model assignments exist precisely so the user can adjust the quality/cost balance, and the architecture does not require the extraction model to be perfect.
- The default cost of approximately $0.05 per session is not free. The question is whether to spend a small amount upfront on focused, deduplicated context or to spend more later correcting errors that entered the codebase because the agent reasoned over raw, unfiltered content. The cache reduces repeat costs over time, but a heavy-research session of dozens of queries will accumulate.
Conclusion
The widely installed Pi search extensions reproduce a design shaped by the compute economics, context budgets, and copyright liability of hosted coding agents. Those constraints are real for Claude Code and OpenAI Codex, but they do not apply to open agents like Pi. The design travelled anyway.
pi-intelli-search was built for the environment open agents actually operate in. The user selects the machine, chooses the model, and owns the cache. Per-page LLM extraction removes irrelevant material before it reaches the primary agent’s context. Cross-source collation deduplicates and flags conflicts. The persistent cache retains full sources for verification and accumulates value across sessions. At project scale, where frontier models answer short factual lookups incorrectly 36–40% of the time, the difference between raw page dumps and focused, deduplicated, verifiable summaries is the difference between mitigating hallucination and hoping it does not compound.
At Curio Data Pro, we build tools grounded in this kind of analysis: understanding what the operating environment permits, what it costs, and whether inherited constraints still apply.
The package is open source under Apache 2.0 and is the second in a series of open source tools and demonstrations from Curio Data Pro.
Version History
2026-08-22- Original.
Attribution
pi-intelli-search is a Pi extension. Pi is developed by Mario Zechner under the Earendil Works organisation. Defuddle is developed by Kepano. wreq-js is developed by sqdshguy. linkedom is developed by Andrea Giammarchi. Perplexity Sonar is developed by Perplexity. MiniMax M2.7 is developed by MiniMax. OpenRouter is operated by OpenRouter, Inc.. We gratefully acknowledge these projects and their contributors.
Images used in this post have been generated using multiple Machine Learning (or Artificial Intelligence) models and subsequently modified by the author. The text has been reviewed using Large Language Models for spelling, grammar, and word choice; however, the content, analysis, and conclusions are entirely the author’s own.
References
Citation
@online{miah2026,
author = {Miah, Ashraf},
title = {Pi-Intelli-Search: {LLM-Native} {Web} {Research} for the {Pi}
{Coding} {Agent}},
date = {2026-08-22},
url = {https://blog.curiodata.pro/posts/22-pi-intelli-search/},
langid = {en}
}