Skip to content
Web scraping for apps and agents.
All articles

Keep the source URL with your extracted content

A page's text is more useful when you can trace it to its source. Build that link into your data before indexing content for an AI agent.

By BackFetch2 min readAI & data

An extracted paragraph can answer a question. Its source URL lets a reader check that answer. Keep both when you prepare web content for an application or an AI agent.

Store content and source together

A small record is enough to start. Include the original URL, a title and the extracted text. Add a collection timestamp if your application needs to reason about freshness.

{
  "url": "https://example.com/docs/setup",
  "title": "Setup guide",
  "content": "Configure the project before running it."
}

This is an example of a storage record, not a BackFetch API response contract. Choose the fields your own application needs.

Preserve the source when splitting text

If you split a long page into smaller chunks for indexing, copy the page URL into each chunk's metadata. You can also keep its section heading to show where the text came from.

record = {
    "url": "https://example.com/docs/setup",
    "title": "Setup guide",
    "content": "Configure the project before running it."
}

chunk = {
    "text": record["content"],
    "source_url": record["url"],
    "source_title": record["title"]
}

Check the original page before trusting a result

A URL identifies a source. It does not prove that the content is accurate or current. Your application still needs to decide which sources to trust and when to collect them again.

Use SERP to discover sources and Deep Crawl to collect linked pages for a knowledge base.

Keep reading