Skip to main content
The crawl endpoint can be limited in two ways:
  • maxVisits: cap how many pages are visited.
  • maxDepth: cap how far the crawl can move from startUrls.
Some reasons to limit crawls:
  • Control cost. Each page visit has network and operation cost.
  • Reduce runtime. Smaller crawl scope finishes faster.

​
Limit the number of visits

maxVisits sets a hard cap on visited pages. Example: 
curl -X POST https://api.fetchfox.ai/api/crawl \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $FETCHFOX_API_KEY" \
-d '{
    "pattern":"https://pokemondb.net/pokedex/*",
    "maxVisits": 50
}'

​
Limit the depth

maxDepth limits crawl distance from the startUrls set. Depth is measured as:
  • Start URL = depth 0
  • Links from start URLs = depth 1
  • Next level = depth 2, etc.
Example: 
curl -X POST https://api.fetchfox.ai/api/crawl \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $FETCHFOX_API_KEY" \
-d '{
    "pattern":"https://pokemondb.net/pokedex/*",
    "startUrls": [
      "https://pokemondb.net/pokedex/national"
    ],
    "maxDepth": 1,
    "maxVisits": 50
}'