Limiting crawls

The crawl endpoint can be limited in two ways:

maxVisits: cap how many pages are visited.
maxDepth: cap how far the crawl can move from startUrls.

Some reasons to limit crawls:

Control cost. Each page visit has network and operation cost.
Reduce runtime. Smaller crawl scope finishes faster.

Limit the number of visits

maxVisits sets a hard cap on visited pages. Example:

curl -X POST https://api.fetchfox.ai/api/crawl \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $FETCHFOX_API_KEY" \
-d '{
    "pattern":"https://pokemondb.net/pokedex/*",
    "maxVisits": 50
}'

Limit the depth

maxDepth limits crawl distance from the startUrls set. Depth is measured as:

Start URL = depth 0
Links from start URLs = depth 1
Next level = depth 2, etc.

Example:

curl -X POST https://api.fetchfox.ai/api/crawl \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $FETCHFOX_API_KEY" \
-d '{
    "pattern":"https://pokemondb.net/pokedex/*",
    "startUrls": [
      "https://pokemondb.net/pokedex/national"
    ],
    "maxDepth": 1,
    "maxVisits": 50
}'

Guides

Scrape

Agent

Visit

Crawl

Extract

Limit the number of visits

Limit the depth

​Limit the number of visits

​Limit the depth

Limit the number of visits

Limit the depth