BackFetchDeep CrawlStart with a URL.Collect the site.
Find the pages behind a starting URL and collect their content. BackFetch Deep Crawl turns linked pages into source material for your search index, knowledge base or AI agent.
Deep CrawlSample response
client.crawl("https://example.com/docs"){
"start_url": "https://example.com/docs",
"pages": [
{
"url": "https://example.com/docs",
"markdown": "# Documentation\nGetting started."
},
{
"url": "https://example.com/docs/setup",
"markdown": "# Setup\nConfigure your project."
}
]
}SDK preview · Illustrative data
Go beyond the page you already know.
Discover linked content
Start at a documentation page or category. Follow its links to find the rest of the content you need.
Keep content tied to its source
Use each page URL alongside its extracted content. Your application can trace a result back to the original page.
Build a collection your application can read.
- Prepare documentation for a knowledge base
- Collect pages for a search index
- Gather product content across a catalog