Building a RAG App on Scraped Documentation: 2026 Architecture

The 2026 Ingestion Stack

a production ingestion pipeline for documentation RAG looks like this:

1. crawl the docs site with Scrapy or Playwright (js-heavy sites need the latter)
2. strip boilerplate with trafilatura or readabilipy
3. split into chunks using heading-aware logic (512 tokens target, 64-token overlap)
4. generate embeddings with text-embedding-3-large or voyage-3-large
5. upsert into your vector store with metadata: url, doc_version, section_title, scraped_at
6. index the raw chunks into a relational table for BM25 hybrid search fallback

metadata is not optional. without doc_version, your RAG app will confidently answer questions about a v2 API using v1 docs. without scraped_at, you cannot invalidate stale chunks when the docs update.

import trafilatura
from langchain.text_splitter import MarkdownHeaderTextSplitter

def extract_and_chunk(html: str, url: str, version: str) -> list[dict]:
    text = trafilatura.extract(html, include_links=False, include_tables=True)
    splitter = MarkdownHeaderTextSplitter(
        headers_to_split_on=[("##", "h2"), ("###", "h3")]
    )
    chunks = splitter.split_text(text)
    return [
        {"content": c.page_content, "url": url, "doc_version": version,
         "section": c.metadata.get("h2", ""), "sub_section": c.metadata.get("h3", "")}
        for c in chunks
    ]

if your target docs site uses dynamic selectors that rotate on deploys (common with documentation platforms that hash CSS classes), look at self-healing scrapers with LLMs for a pattern where the extractor auto-detects the content container when the expected selector fails.

Choosing Your Vector Store

the vector store choice has real downstream effects on latency, cost, and query flexibility. for documentation RAG specifically, hybrid search (dense + sparse) matters more than raw ANN speed because keyword matching outperforms semantic search on exact API method names and error codes.

for most documentation RAG projects under 10M chunks, pgvector with ParadeDB’s pg_bm25 extension gives you hybrid search without adding another service. vector search for scraped data: pgvector vs qdrant vs weaviate benchmarks all three on realistic documentation corpora if you need numbers to back a team decision.

Query and Retrieval Design

retrieval for docs RAG needs two things most tutorials omit: version filtering and reranking.

version filtering means every query is scoped to a specific doc version (or latest). without it, a user asking “how do I authenticate?” gets results mixed across three API versions. implement this as a metadata pre-filter before ANN search, not as a post-filter, or you waste embedding compute.

reranking with a cross-encoder (Cohere Rerank 3, or cross-encoder/ms-marco-MiniLM-L-6-v2 locally) improves precision meaningfully. the vector search step retrieves top-20, the reranker selects top-5 for the LLM context window. without reranking, you will regularly pass irrelevant but semantically close chunks to the model.

key retrieval settings to tune:

• chunk retrieval count: 20 before rerank, 5 after
• max context tokens: 8k for GPT-4o, 32k for Claude Sonnet (use the budget)
• similarity threshold: 0.72 cosine minimum to drop low-confidence results
• fallback to BM25 when semantic score < threshold

Schema Extraction and Freshness

documentation RAG gets significantly more useful when you extract structured data alongside freeform text. API reference pages have consistent schemas: method name, parameters, return types, error codes. extracting these as typed objects rather than prose chunks lets you answer “what does this method return” with a lookup rather than a generative answer.

the approach at LLM-driven scraping schemas with auto-generated pydantic models applies directly here: run a small LLM pass over each API method page during ingestion to extract a structured APIEndpoint model, then store it alongside the prose chunk. at query time, route structured questions to the structured store and prose questions to the vector store.

freshness is the other hard problem. docs change constantly. a simple approach: re-crawl on a 24-hour schedule, compute a content hash per page, and only re-embed pages where the hash changed. this keeps your embedding costs near-zero on most runs while ensuring stale content doesn’t persist. tie the whole pipeline into your broader data warehouse if you are managing multiple scraped sources — building a data warehouse from scraped data covers schema design for exactly this kind of multi-source freshness tracking.

Bottom line

scrape with structure, chunk at heading boundaries, filter by version, and rerank before you pass anything to the LLM. those four steps separate documentation RAG systems that teams actually use from demos that fall apart on real questions. if you are starting fresh in 2026, pgvector plus ParadeDB covers 80% of use cases without adding a dedicated vector service. DRT covers the full stack from scraping to retrieval for engineers building on real data.

Related guides on dataresearchtools.com

• Vector Search for Scraped Data: pgvector vs Qdrant vs Weaviate (2026)
• How to Scrape and Chunk Long-Form Articles for LLM Context (2026)
• LLM-Driven Scraping Schemas: Auto-Generating Pydantic Models (2026)
• Self-Healing Scrapers with LLMs: When Selectors Break (2026)
• Pillar: Building a Data Warehouse from Scraped Data