BackFetchYou want to collect a product catalog. If you already have every product URL, the next job is extracting the fields from those pages. If you only have the catalog homepage, you first need to find the product pages.
That distinction separates crawling from scraping.
Crawling finds pages
A crawler starts at a URL, reads the links on that page and decides which links to visit next. For a documentation site, that might mean following links to installation, configuration and API reference pages.
The important decision is the scope. Decide which domain and paths belong in the collection before following links. Search filters, calendars and pagination can generate many URLs that do not add useful content.
Scraping extracts content
A scraper reads a page and selects the information you need. An article collection might need a title, body and source URL. A product collection might need a name, description and price.
| You already have | Your next step |
|---|---|
| A list of product URLs | Extract the fields from each page |
| A catalog homepage | Discover the product URLs |
| A question or topic | Find relevant pages through search |
Keep the source URL with each result. It lets you inspect the original page when a field is missing or a value changes.
Use both for a site collection
A site collection usually combines discovery and extraction. The crawler builds the set of pages. The scraper turns their content into records your application can store.
For the BackFetch product overview, see Deep Crawl. If your starting point is a search query, see SERP.