Skip to main content
The /api/crawl endpoint supports two crawl modes. You must provide exactly one of pattern or query.

​
Pattern-based crawl

Use pattern when you already know the URL structure you want. A URL pattern is a full URL plus wildcard operators:
  • * matches any character except /
  • ** matches any character including /
URL patterns must be valid URLs with a domain. The domain may not contain wildcards. Below are a few examples of URL patterns and what they match. Example request: 
curl -X POST https://api.fetchfox.ai/api/crawl \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $FETCHFOX_API_KEY" \
-d '{
    "pattern":"https://pokemondb.net/pokedex/*",
    "maxVisits": 50
}'
Example response: 
{
  "jobId": "5ooygvit1y",
  "results": {
    "hits": [
      "https://pokemondb.net/pokedex/all",
      "https://pokemondb.net/pokedex/archaludon",
      "https://pokemondb.net/pokedex/charizard",
      "https://pokemondb.net/pokedex/corviknight",
      "https://pokemondb.net/pokedex/dipplin",
      "https://pokemondb.net/pokedex/dragapult",
      "https://pokemondb.net/pokedex/dragonite",
      "https://pokemondb.net/pokedex/eevee",
      "https://pokemondb.net/pokedex/game/legends-arceus",
      "https://pokemondb.net/pokedex/game/scarlet-violet",
      "...more results..."
    ]
  },
  "metrics": { "...cost and usage metrics..." }
}
The results.hits section contains all the matching URLs. If your pattern is too broad, switch from ** to * where you want to avoid crossing path segments.

​
Query-based crawl

Use query when you do not know the URL structure ahead of time, but you can describe the kind of page you want. With query crawl, FetchFox starts from startUrls, visits those pages, and uses page content plus links on the page to learn which URLs directly match your query and which links lead toward matching pages. query crawl requires startUrls. Example request: 
curl -X POST https://api.fetchfox.ai/api/crawl \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_API_KEY" \
-d '{
    "query":"pokemon detail pages",
    "startUrls": [
      "https://pokemondb.net/pokedex/all"
    ],
    "maxVisits": 50
}'
Use pattern when you know the URL shape. Use query when you want FetchFox to discover relevant URLs from page content. 
curl -X POST https://api.fetchfox.ai/api/crawl \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_API_KEY" \
-d '{
    "pattern":"https://pokemondb.net/pokedex/*",
    "maxVisits": 50
}'