> ## Documentation Index
> Fetch the complete documentation index at: https://docs.fetchfox.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawl using patterns or queries

> The crawl endpoint finds URLs using either a URL pattern or a query

The `/api/crawl` endpoint supports two crawl modes. You must provide exactly one of `pattern` or `query`.

## Pattern-based crawl

Use `pattern` when you already know the URL structure you want.

A URL pattern is a full URL plus wildcard operators:

* `*` matches any character *except*  `/`
* `**` matches any character *including*  `/`

URL patterns must be valid URLs with a domain. The domain may not contain wildcards.

Below are a few examples of URL patterns and what they match.

* Pattern: `https://example.com/a/*`
  * Matches
    * [https://example.com/a/page-1](https://example.com/a/page-1)
    * [https://example.com/a/page-2](https://example.com/a/page-2)
  * Does *not* match
    * [https://example.com/b/page-1](https://example.com/b/page-1)
    * [https://example.com/a/x/y/z](https://example.com/a/x/y/z)

* Pattern: `https://example.com/a/**`
  * Matches
    * [https://example.com/a/page-1](https://example.com/a/page-1)
    * [https://example.com/a/page-2](https://example.com/a/page-2)
    * [https://example.com/a/x/y/z](https://example.com/a/x/y/z)
  * Does *not* match
    * [https://example.com/b/page-1](https://example.com/b/page-1)

Example request:

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://api.fetchfox.ai/api/crawl \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $FETCHFOX_API_KEY" \
  -d '{
      "pattern":"https://pokemondb.net/pokedex/*",
      "maxVisits": 50
  }'
  ```
</CodeGroup>

Example response:

```json theme={null}
{
  "jobId": "5ooygvit1y",
  "results": {
    "hits": [
      "https://pokemondb.net/pokedex/all",
      "https://pokemondb.net/pokedex/archaludon",
      "https://pokemondb.net/pokedex/charizard",
      "https://pokemondb.net/pokedex/corviknight",
      "https://pokemondb.net/pokedex/dipplin",
      "https://pokemondb.net/pokedex/dragapult",
      "https://pokemondb.net/pokedex/dragonite",
      "https://pokemondb.net/pokedex/eevee",
      "https://pokemondb.net/pokedex/game/legends-arceus",
      "https://pokemondb.net/pokedex/game/scarlet-violet",
      "...more results..."
    ]
  },
  "metrics": { "...cost and usage metrics..." }
}
```

The `results.hits` section contains all the matching URLs.

If your pattern is too broad, switch from `**` to `*` where you want to avoid crossing path segments.

## Query-based crawl

Use `query` when you do not know the URL structure ahead of time, but you can describe the kind of page you want.

With query crawl, FetchFox starts from `startUrls`, visits those pages, and uses page content plus links on the page to learn which URLs directly match your query and which links lead toward matching pages.

`query` crawl requires `startUrls`.

Example request:

<CodeGroup>
  ```curl curl theme={null}
  curl -X POST https://api.fetchfox.ai/api/crawl \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
      "query":"pokemon detail pages",
      "startUrls": [
        "https://pokemondb.net/pokedex/all"
      ],
      "maxVisits": 50
  }'
  ```
</CodeGroup>

Use `pattern` when you know the URL shape. Use `query` when you want FetchFox to discover relevant URLs from page content.

<CodeGroup>
  ```curl curl theme={null}
  curl -X POST https://api.fetchfox.ai/api/crawl \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
      "pattern":"https://pokemondb.net/pokedex/*",
      "maxVisits": 50
  }'
  ```
</CodeGroup>