How to Convert a Website to Clean Markdown for RAG & LLMs
Raw HTML is terrible context for an LLM: nav bars, cookie banners and footers drown the actual content and waste tokens. The fix is to crawl the page, strip the boilerplate, and keep only clean Markdown — optionally pre-chunked for embedding. Here is how to do it for a single page or a whole docs site.
- Add the start URLs. Paste the page, docs section or site you want. Set crawl depth to follow internal links (e.g. a whole /docs tree) or keep it to a single page.
- Strip to clean Markdown. The tool removes nav, headers, footers, cookie banners and scripts, and converts the main content to Markdown — headings, lists, tables and code blocks preserved.
- Chunk for RAG (optional). Turn on RAG chunking to split each page into overlapping chunks at a size that fits your embedding model, so you can embed the output directly without a post-processing step.
- Feed your pipeline. Export the Markdown/chunks as JSON and push them straight into your vector store, or pull from the dataset via API into LangChain, LlamaIndex or your own RAG stack.
Why Markdown instead of raw HTML for RAG?
Markdown keeps the meaningful structure (headings, lists, code) while dropping nav and boilerplate, so retrieval is cleaner and you spend far fewer tokens on junk.
Do I need an API key or a Firecrawl/Jina subscription?
No — it runs without any third-party key and works as a keyless alternative to Firecrawl or Jina Reader.
Can it crawl a whole documentation site?
Yes — set crawl depth and it follows internal links across the docs tree, returning one clean Markdown document per page.
Can it output chunks ready to embed?
Yes — enable RAG chunking to get overlapping chunks sized for your embedding model, no extra splitting step needed.
Run it on Apify — no install, results in minutes.
Open on Apify