Skip to content
agentusehub
Sign in+ Submit
Software & DevOpsAny

Crawl documentation into clean Markdown for AI agents

The author built a documentation crawler for their AI agents that pulled 60 pages of clean Markdown in 49 seconds.

Read the experience
Time saved
Not reported
Setup time
Not reported
Running cost
Not reported
Cost range
Not reported

Use case overview

The author built a documentation crawler for their AI agents that pulled 60 pages of clean Markdown in 49 seconds.

Read the original report ↗

Reported experience

  1. What they did

    The author built pulpie-mcp, an MCP server with four tools: fetch, save, crawl and list. It runs a local extraction model behind a backend that starts automatically. Claude was asked to crawl a 60-page docs site. It found pages through robots.txt and sitemaps, used four concurrent workers, and wrote Markdown files with frontmatter to disk instead of returning them to the context window.

  2. How it went

    Total time was about 49 seconds: roughly 10 for backend cold start and 37 for crawling and saving 60 pages. Untested on client-rendered pages, which may come back empty. PDFs aren't supported.

  3. Worth knowing

    The model is licensed CC BY-NC 4.0 (non-commercial), and it has only been tested on Linux with an NVIDIA GPU, using about 420 MB of VRAM.

Example prompts

Try with Claude Code
Build a crawler that takes the documentation site [docs URL], follows its internal links, and saves each page as clean Markdown (no nav or footer) in a folder called docs-md, one file per page. Respect robots.txt and rate limit politely. Done when you have crawled the whole site and given me a count of pages saved and a list of any that failed.

Related use cases

Software & DevOps

Port a Chrome extension to Safari

The author had ChatGPT take the Session Buddy extension source from Chrome, port it to Safari and install it, taking about 20 minutes.

ChatGPT