Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.fetchfox.ai/llms.txt

Use this file to discover all available pages before exploring further.

You can use the priority parameter to control crawl scope. The most important fields are:
  • only: whitelist URL patterns to allow.
  • skip: blacklist URL patterns to avoid.
If both are provided, FetchFox uses only to constrain candidates and skip to remove unwanted URLs. Example: 
curl -X POST https://api.fetchfox.ai/api/crawl \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $FETCHFOX_API_KEY" \
-d '{
    "pattern":"https://example.com/shopping/*",
    "priority": {
      "only": [
        "https://example.com/shopping/*"
      ],
      "skip": [
        "https://example.com/shopping/jeans/*",
        "https://example.com/shopping/pants/*"
      ]
    },
    "maxVisits": 50
}'
This is useful when you only want one part of a large site. Keep in mind: priority controls what the crawler chooses to visit. The final results.hits set is still based on crawl discovery and pattern matching.