Set crawl priorities

You can use the priority parameter to control crawl scope. The most important fields are:

only: whitelist URL patterns to allow.
skip: blacklist URL patterns to avoid.

If both are provided, FetchFox uses only to constrain candidates and skip to remove unwanted URLs. Example:

curl -X POST https://api.fetchfox.ai/api/crawl \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $FETCHFOX_API_KEY" \
-d '{
    "pattern":"https://example.com/shopping/*",
    "priority": {
      "only": [
        "https://example.com/shopping/*"
      ],
      "skip": [
        "https://example.com/shopping/jeans/*",
        "https://example.com/shopping/pants/*"
      ]
    },
    "maxVisits": 50
}'

This is useful when you only want one part of a large site. Keep in mind: priority controls what the crawler chooses to visit. The final results.hits set is still based on crawl discovery and pattern matching.

Limiting crawls Extract data from URLs

Guides

Scrape

Agent

Visit

Crawl

Extract

Set crawl priorities