# Agent endpoint
Source: https://docs.fetchfox.ai/agent

Run scraping tasks with a single prompt

The `/api/agent` endpoint takes a natural-language `prompt` and chooses a strategy (`extract`, `crawl`, or `scrape`) for you.

## Key parameter

* `prompt`: what you want to collect

## Example

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://api.fetchfox.ai/api/agent \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "prompt": "Find product URLs on pokemondb.net and extract name and number for each pokemon."
  }'
  ```
</CodeGroup>

Typical response shape:

```json theme={null}
{
  "jobId": "abc123",
  "results": {
    "hits": [
      "https://pokemondb.net/pokedex/all",
      "https://pokemondb.net/pokedex/bulbasaur"
    ],
    "items": [
      {
        "name": "Bulbasaur",
        "number": "0001",
        "_url": "https://pokemondb.net/pokedex/bulbasaur",
        "_htmlUrl": "https://ffxyz.s3.amazonaws.com/visit/html/example.html"
      }
    ]
  },
  "metrics": { ... }
}
```

## Pattern vs query crawl behavior

The agent can route crawl-like work in two ways:

* **Pattern-based crawl**: URL wildcard matching (`pattern`).
* **Query-based crawl**: semantic page scanning (`query`) from seed URLs.

This distinction matters for behavior, but you usually only provide `prompt` and let the agent choose.


# Automatically pick a proxy
Source: https://docs.fetchfox.ai/auto-proxy

Ask FetchFox to figure out which proxy to use

Many sites [require proxies](/proxy-basics). Instead of manually choosing a tier for each domain, you can set `proxy` to `auto`.

## The auto proxy parameter

Main endpoint requests (`visit`, `crawl`, `extract`, `scrape`) support:

```json theme={null}
{
  "proxy": "auto"
}
```

Example:

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://api.fetchfox.ai/api/crawl \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $FETCHFOX_API_KEY" \
  -d '{
      "pattern":"https://pokemondb.net/pokedex/*",
      "proxy": "auto",
      "maxVisits": 50
  }'
  ```
</CodeGroup>

With `auto`, FetchFox uses domain-level history to balance reliability and cost.

`/api/agent` uses automatic proxying by default.

## What to expect

* First request on a new domain can be slower while data is established.
* Later requests on the same domain are usually faster and more stable.
* If your job still struggles on a domain, switch to an explicit `proxy` tier.

If you need hard control over cost/behavior, use an explicit tier from [Proxy basics](/proxy-basics).


# Boosted extractions
Source: https://docs.fetchfox.ai/boosted-extractions

How FetchFox speeds up repeated extractions

FetchFox can speed up repeated extractions with a boost system.

## How it works

For the first few extractions of a given page pattern and template, FetchFox extracts data by passing the page HTML to an AI model.

After it has collected a few successful examples, the backend asks a more advanced AI model to write extraction code for that pattern and template. The backend evaluates that generated code before reusing it.

Once the code has been written and accepted, later extractions can run using the generated code instead of calling an AI model for each page.

## What this means in practice

* The first few extractions are usually slower and include AI cost.
* Later extractions can run much faster.
* After boost code is in place, later runs typically have no AI cost for the extraction step.

You do not need to enable anything manually. The system learns from successful extractions automatically.


# Crawl using patterns or queries
Source: https://docs.fetchfox.ai/crawl-using-patterns

The crawl endpoint finds URLs using either a URL pattern or a query

The `/api/crawl` endpoint supports two crawl modes. You must provide exactly one of `pattern` or `query`.

## Pattern-based crawl

Use `pattern` when you already know the URL structure you want.

A URL pattern is a full URL plus wildcard operators:

* `*` matches any character *except*  `/`
* `**` matches any character *including*  `/`

URL patterns must be valid URLs with a domain. The domain may not contain wildcards.

Below are a few examples of URL patterns and what they match.

* Pattern: `https://example.com/a/*`
  * Matches
    * [https://example.com/a/page-1](https://example.com/a/page-1)
    * [https://example.com/a/page-2](https://example.com/a/page-2)
  * Does *not* match
    * [https://example.com/b/page-1](https://example.com/b/page-1)
    * [https://example.com/a/x/y/z](https://example.com/a/x/y/z)

* Pattern: `https://example.com/a/**`
  * Matches
    * [https://example.com/a/page-1](https://example.com/a/page-1)
    * [https://example.com/a/page-2](https://example.com/a/page-2)
    * [https://example.com/a/x/y/z](https://example.com/a/x/y/z)
  * Does *not* match
    * [https://example.com/b/page-1](https://example.com/b/page-1)

Example request:

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://api.fetchfox.ai/api/crawl \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $FETCHFOX_API_KEY" \
  -d '{
      "pattern":"https://pokemondb.net/pokedex/*",
      "maxVisits": 50
  }'
  ```
</CodeGroup>

Example response:

```json theme={null}
{
  "jobId": "5ooygvit1y",
  "results": {
    "hits": [
      "https://pokemondb.net/pokedex/all",
      "https://pokemondb.net/pokedex/archaludon",
      "https://pokemondb.net/pokedex/charizard",
      "https://pokemondb.net/pokedex/corviknight",
      "https://pokemondb.net/pokedex/dipplin",
      "https://pokemondb.net/pokedex/dragapult",
      "https://pokemondb.net/pokedex/dragonite",
      "https://pokemondb.net/pokedex/eevee",
      "https://pokemondb.net/pokedex/game/legends-arceus",
      "https://pokemondb.net/pokedex/game/scarlet-violet",
      "...more results..."
    ]
  },
  "metrics": { "...cost and usage metrics..." }
}
```

The `results.hits` section contains all the matching URLs.

If your pattern is too broad, switch from `**` to `*` where you want to avoid crossing path segments.

## Query-based crawl

Use `query` when you do not know the URL structure ahead of time, but you can describe the kind of page you want.

With query crawl, FetchFox starts from `startUrls`, visits those pages, and uses page content plus links on the page to learn which URLs directly match your query and which links lead toward matching pages.

`query` crawl requires `startUrls`.

Example request:

<CodeGroup>
  ```curl curl theme={null}
  curl -X POST https://api.fetchfox.ai/api/crawl \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
      "query":"pokemon detail pages",
      "startUrls": [
        "https://pokemondb.net/pokedex/all"
      ],
      "maxVisits": 50
  }'
  ```
</CodeGroup>

Use `pattern` when you know the URL shape. Use `query` when you want FetchFox to discover relevant URLs from page content.

<CodeGroup>
  ```curl curl theme={null}
  curl -X POST https://api.fetchfox.ai/api/crawl \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
      "pattern":"https://pokemondb.net/pokedex/*",
      "maxVisits": 50
  }'
  ```
</CodeGroup>


# Detached jobs
Source: https://docs.fetchfox.ai/detached-jobs

Run FetchFox requests asynchronously and poll for completion

FetchFox supports detached jobs for long-running requests.

Any endpoint can take `detach: true`. When you set it, FetchFox returns immediately with a `jobId` instead of keeping the request open until the job finishes.

Detached jobs are useful when:

* Your crawl or extraction may take a while
* Your client has a short timeout
* You want to start work in one request and check progress later

## Start a detached job

Add `detach: true` to the same request body you would normally send.

Example:

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://api.fetchfox.ai/api/scrape \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "pattern": "https://pokemondb.net/pokedex/*",
    "template": {
      "name": "Pokemon name",
      "number": "Pokemon number"
    },
    "maxVisits": 50,
    "maxExtracts": 50,
    "detach": true
  }'
  ```
</CodeGroup>

Typical response:

```json theme={null}
{
  "jobId": "abc123"
}
```

## Check job status

To check the current status of a detached job, request:

`https://api.fetchfox.ai/api/jobs/:jobId`

Example:

<CodeGroup>
  ```bash curl theme={null}
  curl https://api.fetchfox.ai/api/jobs/abc123 \
  -H "Authorization: Bearer YOUR_API_KEY"
  ```
</CodeGroup>

The job response includes a `state` field.

## Job states

* `active`: the job is still running
* `completed`: the job finished successfully
* `error`: the job failed while running

If the job is still `active`, keep polling the job endpoint until it reaches either `completed` or `error`.

## Summary

* Any FetchFox endpoint can run as a detached job.
* Enable it by adding `detach: true` to the request body.
* Use the returned `jobId` to poll `/api/jobs/:jobId`.
* Watch the `state` field to know whether the job is still running, completed, or failed.


# Extract data from URLs
Source: https://docs.fetchfox.ai/extract-from-urls

The extract endpoint takes URLs and outputs structured data

The `/api/extract` endpoint converts page content into structured items.

## A simple extraction

Key parameters:

* `url` **or** `urls` (single URL or a list)
* `template` (string or object)

The template can be a dictionary or a string:

* **Dictionary/object**: output items follow the keys you provide.
* **String**: FetchFox infers a schema from your prompt.

Example:

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST "https://api.fetchfox.ai/api/extract" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $FETCHFOX_API_KEY" \
  -d '{
     "urls": [
       "https://pokemondb.net/pokedex/bulbasaur",
       "https://pokemondb.net/pokedex/ivysaur",
       "https://pokemondb.net/pokedex/venusaur"
     ],
     "template": {
       "name": "Name of the pokemon",
       "number": "National pokedex number",
       "stats": "Base stats as a dictionary"
     }
  }'
  ```
</CodeGroup>

Response:

```json theme={null}
{
  "jobId": "j8rcgsnxq3",
  "results": {
    "items": [
      {
        "name": "Bulbasaur",
        "number": "0001",
        "stats": {
          "HP": 45,
          "Attack": 49,
          "Defense": 49,
          "Sp. Atk": 65,
          "Sp. Def": 65,
          "Speed": 45,
          "Total": 318
        },
        "_url": "https://pokemondb.net/pokedex/bulbasaur",
        "_htmlUrl": "https://ffcloud.s3.amazonaws.com/visit/html/4h0o70v9fh.html"
      },
      "...more results..."
    ]
  },
  "metrics": { "...cost and usage metrics..." }
}

```

## Extracting multiple items per URL

By default, FetchFox extracts one item per URL. To extract multiple items from each page, set `perPage` to `many`.

Example:

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST "https://api.fetchfox.ai/api/extract" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $FETCHFOX_API_KEY" \
  -d '{
     "urls": [
       "https://pokemondb.net/pokedex/bulbasaur",
       "https://pokemondb.net/pokedex/ivysaur",
       "https://pokemondb.net/pokedex/venusaur"
     ],
     "template": {
       "moveName": "Name of the pokemon move",
       "moveType": "Name of the move type",
       "movePower": "The power of the move"
     },
     "perPage": "many"
  }'
  ```
</CodeGroup>

Response:

```json theme={null}
{
  "jobId": "fjszygdh38",
  "results": {
    "items": [
      {
        "moveName": "Growl",
        "moveType": "Normal",
        "movePower": "100",
        "_url": "https://pokemondb.net/pokedex/ivysaur",
        "_htmlUrl": "https://ffcloud.s3.amazonaws.com/visit/html/xz6rjf8h2v.html"
      },
      {
        "moveName": "Growth",
        "moveType": "Normal",
        "movePower": "—",
        "_url": "https://pokemondb.net/pokedex/ivysaur",
        "_htmlUrl": "https://ffcloud.s3.amazonaws.com/visit/html/xz6rjf8h2v.html"
      },
      "...more items..."
    ]
  },
  "metrics": { "...cost and usage metrics..." }
}
```

When `perPage: "many"` is used, the response includes a `divide` artifact showing the selector used to split the page into repeated item blocks.

## Boosted extractions

The first few extractions for a given page pattern and template are usually done by sending page HTML to an AI model.

After FetchFox has a few successful examples, the backend asks a more advanced model to write extraction code for that pattern and template. Later extractions can then run that generated code directly.

That means repeated extractions usually get faster over time, and boosted runs typically have no AI cost for the extraction step.

* [Read more about boosted extractions](/boosted-extractions)


# Items per page
Source: https://docs.fetchfox.ai/items-per-page

Choose one or many extraction items per URL

Most pages on the web fall into one of two categories:

* **Detail pages** that describe a single item
* **List pages** that show multiple items

FetchFox can extract from both pages. For detail pages, you want to extract one item for each URL, and for list pages, you want to extract multiple items per page.

The **default extraction mode** is to extract **one item per URL**. In this mode, if you pass in 10 URLs, you will get exactly 10 items in your results. This works well for detail pages.

To extract multiple items per page, set `perPage` to `many`.

Example:

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST "https://api.fetchfox.ai/api/extract" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $FETCHFOX_API_KEY" \
  -d '{
     "urls": [
       "https://pokemondb.net/pokedex/bulbasaur",
       "https://pokemondb.net/pokedex/ivysaur",
       "https://pokemondb.net/pokedex/venusaur"
     ],
     "template": {
       "moveName": "Name of the pokemon move",
       "moveType": "Name of the move type",
       "movePower": "The power of the move"
     },
     "perPage": "many"
  }'
  ```
</CodeGroup>

Response:

```json theme={null}
{
  "jobId": "fjszygdh38",
  "results": {
    "items": [
      {
        "moveName": "Growl",
        "moveType": "Normal",
        "movePower": "100",
        "_url": "https://pokemondb.net/pokedex/ivysaur",
        "_htmlUrl": "https://ffcloud.s3.amazonaws.com/visit/html/xz6rjf8h2v.html"
      },
      {
        "moveName": "Growth",
        "moveType": "Normal",
        "movePower": "—",
        "_url": "https://pokemondb.net/pokedex/ivysaur",
        "_htmlUrl": "https://ffcloud.s3.amazonaws.com/visit/html/xz6rjf8h2v.html"
      },
      "...more items..."
    ]
  },
  "metrics": { "...cost and usage metrics..." }
}
```

Each URL can produce many items. FetchFox charges fees on the basis of each operation (`visit`, `crawl`, `extract`), while AI and network costs depend on model usage and page traffic. [See pricing](/pricing).


# Limiting crawls
Source: https://docs.fetchfox.ai/limiting-crawls

You can limit the number of visits and depth of crawls

The crawl endpoint can be limited in two ways:

* `maxVisits`: cap how many pages are visited.
* `maxDepth`: cap how far the crawl can move from `startUrls`.

Some reasons to limit crawls:

* **Control cost.** Each page visit has network and operation cost.
* **Reduce runtime.** Smaller crawl scope finishes faster.

## Limit the number of visits

`maxVisits` sets a hard cap on visited pages.

Example:

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://api.fetchfox.ai/api/crawl \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $FETCHFOX_API_KEY" \
  -d '{
      "pattern":"https://pokemondb.net/pokedex/*",
      "maxVisits": 50
  }'
  ```
</CodeGroup>

## Limit the depth

`maxDepth` limits crawl distance from the `startUrls` set.

Depth is measured as:

* Start URL = depth 0
* Links from start URLs = depth 1
* Next level = depth 2, etc.

Example:

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://api.fetchfox.ai/api/crawl \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $FETCHFOX_API_KEY" \
  -d '{
      "pattern":"https://pokemondb.net/pokedex/*",
      "startUrls": [
        "https://pokemondb.net/pokedex/national"
      ],
      "maxDepth": 1,
      "maxVisits": 50
  }'
  ```
</CodeGroup>


# Pricing and cost control
Source: https://docs.fetchfox.ai/pricing

FetchFox uses cost plus pricing, with many ways to lower cost

FetchFox uses [cost plus pricing](https://en.wikipedia.org/wiki/Cost-plus_pricing), which means we charge a fixed fee on top of our underlying costs. The underlying costs for each scrape are network traffic through proxies, and the cost of AI usage. We pass on these costs directly, and add a FetchFox fee based on API usage.

$$
Your\ Price = Network\ Traffic + AI\ Usage * AI\ Surcharge + FetchFox\ Fees
$$

In `metrics`:

* `cost.network` = proxy bandwidth cost
* `cost.ai` = model token cost
* `cost.fetchfox` = operation fees + AI surcharge

Credits are USD-denominated (`1000` credits = `$1.00`).

## Viewing cost metrics

Every call you make to FetchFox has a `metrics` field. This field includes a cost breakdown, showing how much of the cost for that call was for network traffic, AI usage, and FetchFox fees.

Below is an example cost breakdown.

<AccordionGroup>
  <Accordion title="Summary" icon="sparkles">
    ```json theme={null}
    {
      "cost": {
        "ai": 0.02,
        "network": 0.00019,
        "fetchfox": 0.035,
        "total": 0.05519
      },
      ... full breakdown ...
    }
    ```
  </Accordion>

  <Accordion title="Full breakdown" icon="sparkles">
    ```json theme={null}
    {
      "cost": {
        "ai": 0.02,
        "network": 0.00019,
        "fetchfox": 0.035,
        "total": 0.05519
      },
      "ai": [
        {
          "model": "openai:gpt-5.2",
          "tokens": {
            "input": 100000,
            "output": 4000,
            "total": 104000
          },
          "cost": {
            "input": 0.016,
            "output": 0.004,
            "total": 0.02
          },
          "runtime": {
            "sec": 90.4,
            "msec": 90400
          }
        }
      ],
      "network": [
        {
          "tier": "datacenter_dedicated",
          "bytes": 3800000,
          "cost": 0.00019
        }
      ],
      "fetchfox": [
        {
          "path": "crawl",
          "count": 1,
          "cost": 0.001
        },
        {
          "path": "extract",
          "count": 20,
          "cost": 0.02
        },
        {
          "path": "visit",
          "count": 10,
          "cost": 0.01
        },
        {
          "path": "surcharge",
          "count": 1,
          "cost": 0.004
        }
      ]
    }
    ```
  </Accordion>
</AccordionGroup>

In this example, you can see a summary of costs for network, AI, and FetchFox usage. You can also see a detailed breakdown showing which proxies were used for how much data, which AI models were used for how many tokens, and which FetchFox calls were used.

## FetchFox fees

FetchFox operation fees are:

| Operation | Price                               |
| --------- | ----------------------------------- |
| `visit`   | \$1.00 per 1,000 visits             |
| `crawl`   | \$1.00 per 1,000 crawl operations   |
| `extract` | \$1.00 per 1,000 extract operations |
| `captcha` | \$1.00 per 1,000 solves             |

FetchFox also applies a surcharge equal to **20% of AI usage cost**.

<CostsTable />

A [scrape call](/scrape-equals-crawl-extract) does not have its own charge, but it will incur costs from all the underlying operations.

## Underlying cost

There are two sources of underlying cost for each call to FetchFox: network traffic and AI usage.

### Network costs

Network traffic is billed per byte and depends on proxy tier. Costs vary significantly by tier.

| Proxy tier               | Cost per GB |
| ------------------------ | ----------- |
| `none`                   | \$0.025     |
| `datacenter`             | \$0.05      |
| `datacenter_shared`      | \$0.80      |
| `datacenter_dedicated`   | \$0.05      |
| `isp_dedicated`          | \$0.05      |
| `residential`            | \$8.10      |
| `residential_cdp`        | \$8.20      |
| `residential_cdp_assets` | \$8.50      |
| `unblock`                | \$2.00      |

Read more about proxies:

* [Why proxies are required to access some sites](/proxy-basics)
* [Use automatic proxy selection](/auto-proxy)

### AI costs

AI usage is billed by token usage. Larger context means higher AI cost.

Two practical ways to control AI cost:

1. Use a lower-cost model.
2. Let repeated extractions get boosted over time.

Read more about boosted extraction behavior:

* [How boosted extractions work](/boosted-extractions)


# Avoid blocks with proxies
Source: https://docs.fetchfox.ai/proxy-basics

Some sites require proxies to avoid blocks

By default, FetchFox connects using low-cost proxies. Some sites block those proxies, so you may need a different proxy tier.

To pick a proxy explicitly, set the `proxy` field in `visit`, `crawl`, `extract`, or `scrape` requests.

Example:

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://api.fetchfox.ai/api/crawl \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $FETCHFOX_API_KEY" \
  -d '{
      "pattern":"https://pokemondb.net/pokedex/*",
      "proxy": "residential",
      "maxVisits": 50
  }'
  ```
</CodeGroup>

Different domains need different proxy tiers. Start with cheaper tiers when possible.

Current proxy tiers in automatic selection and their costs:

| Proxy tier               | Cost per GB |
| ------------------------ | ----------- |
| `none`                   | \$0.025     |
| `datacenter`             | \$0.05      |
| `datacenter_shared`      | \$0.80      |
| `datacenter_dedicated`   | \$0.05      |
| `isp_dedicated`          | \$0.05      |
| `residential`            | \$8.10      |
| `residential_cdp`        | \$8.20      |
| `residential_cdp_assets` | \$8.50      |
| `unblock`                | \$2.00      |

If a site is still blocked, try `proxy: "auto"` to let FetchFox choose.

## Try multiple proxies

You can also pass an array to `proxy` and FetchFox will try multiple tiers concurrently.

Example:

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://api.fetchfox.ai/api/extract \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $FETCHFOX_API_KEY" \
  -d '{
      "urls": ["https://pokemondb.net/pokedex/pikachu"],
      "template": "pokemon name, number, and basic stats",
      "proxy": ["datacenter", "residential"]
  }'
  ```
</CodeGroup>

## Automatically pick a proxy

Pass `proxy: "auto"` to let FetchFox choose based on historical success/cost data for that domain.

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://api.fetchfox.ai/api/crawl \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $FETCHFOX_API_KEY" \
  -d '{
      "pattern":"https://pokemondb.net/pokedex/*",
      "proxy": "auto",
      "maxVisits": 50
  }'
  ```
</CodeGroup>

Read our guide on [automatic proxy selection](/auto-proxy) for more information on how this works.


# Quick start
Source: https://docs.fetchfox.ai/quickstart

Start scraping in 2 minutes with the FetchFox API

## Get your API key

First, you'll need your API key. You can find it in the FetchFox app at [<u>https://fetchfox.ai/settings/api-keys</u>](https://fetchfox.ai/settings/api-keys).

## Run a scrape

To run a scrape, use the `/api/scrape` endpoint with:

* `pattern` for URL matching
* `template` for the output shape

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://api.fetchfox.ai/api/scrape \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "pattern": "https://pokemondb.net/pokedex/*",
    "template": {
      "name": "Pokemon name",
      "number": "Pokemon number"
    },
    "maxVisits": 10,
    "maxExtracts": 10
  }'
  ```

  ```javascript javascript theme={null}
  import fetch from 'node-fetch';

  const response = await fetch('https://api.fetchfox.ai/api/scrape', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'Authorization': 'Bearer YOUR_API_KEY',
    },
    body: JSON.stringify({
      pattern: 'https://pokemondb.net/pokedex/*',
      template: {
        name: 'Pokemon name',
        number: 'Pokemon number'
      },
      maxVisits: 10,
      maxExtracts: 10
    })
  });

  const data = await response.json();
  console.log(data.results.items);
  ```

  ```python python theme={null}
  import requests

  url = "https://api.fetchfox.ai/api/scrape"
  headers = {
      "Content-Type": "application/json",
      "Authorization": "Bearer YOUR_API_KEY"
  }
  data = {
      "pattern": "https://pokemondb.net/pokedex/*",
      "template": {
          "name": "Pokemon name",
          "number": "Pokemon number"
      },
      "maxVisits": 10,
      "maxExtracts": 10
  }

  response = requests.post(url, json=data, headers=headers)
  print(response.json()['results']['items'])
  ```
</CodeGroup>

When the scrape finishes, the extracted items are in `results.items`.

## Next Steps

Now that you've run a simple scrape:

1. Learn how [crawl and extract work together](/scrape-equals-crawl-extract).
2. Learn how to [control crawl scope and cost](/limiting-crawls).


# Scrape = Crawl + Extract
Source: https://docs.fetchfox.ai/scrape-equals-crawl-extract

Scraping with FetchFox has two phases: crawl and extract

FetchFox scraping is two steps:

1. **Crawl** to find relevant URLs.
2. **Extract** to turn page content into structured items.

## Crawl for URLs

Use `/api/crawl` with a `pattern`:

* `*` matches any characters except `/`
* `**` matches any characters including `/`

Example:

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://api.fetchfox.ai/api/crawl \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $FETCHFOX_API_KEY" \
  -d '{
      "pattern":"https://pokemondb.net/pokedex/*",
      "maxVisits": 50
  }'
  ```
</CodeGroup>

Typical response:

```json theme={null}
{
  "jobId": "5ooygvit1y",
  "results": {
    "hits": [
      "https://pokemondb.net/pokedex/all",
      "https://pokemondb.net/pokedex/archaludon",
      "https://pokemondb.net/pokedex/charizard",
      "https://pokemondb.net/pokedex/corviknight",
      "https://pokemondb.net/pokedex/dipplin",
      "https://pokemondb.net/pokedex/dragapult",
      "https://pokemondb.net/pokedex/dragonite",
      "https://pokemondb.net/pokedex/eevee",
      "https://pokemondb.net/pokedex/game/legends-arceus",
      "https://pokemondb.net/pokedex/game/scarlet-violet",
      "...more results..."
    ]
  },
  "metrics": { "...cost and usage metrics..." }
}
```

The URLs are returned in `results.hits`.

## Extract from URLs to get items

Use `/api/extract` with:

* `url` or `urls`
* `template`

Example:

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST "https://api.fetchfox.ai/api/extract" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $FETCHFOX_API_KEY" \
  -d '{
     "urls": [
       "https://pokemondb.net/pokedex/bulbasaur",
       "https://pokemondb.net/pokedex/ivysaur",
       "https://pokemondb.net/pokedex/venusaur"
     ],
     "template": {
       "name": "Name of the pokemon",
       "number": "National pokedex number",
       "stats": "Base stats as a dictionary"
     }
  }'
  ```
</CodeGroup>

Typical response:

```json theme={null}
{
  "jobId": "j8rcgsnxq3",
  "results": {
    "items": [
      {
        "name": "Bulbasaur",
        "number": "0001",
        "stats": {
          "HP": 45,
          "Attack": 49,
          "Defense": 49,
          "Sp. Atk": 65,
          "Sp. Def": 65,
          "Speed": 45,
          "Total": 318
        },
        "_url": "https://pokemondb.net/pokedex/bulbasaur",
        "_htmlUrl": "https://ffcloud.s3.amazonaws.com/visit/html/4h0o70v9fh.html"
      },
      "...more results..."
    ]
  },
  "metrics": { "...cost and usage metrics..." }
}

```

## A single endpoint to crawl and extract

To run both phases in one request, use `/api/scrape`.

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST "https://api.fetchfox.ai/api/scrape" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer YOUR_API_KEY" \
    -d '{
       "pattern": "https://www.pokemon.com/us/pokedex/*",
       "template": {
         "name": "Name of the pokemon",
         "number": "National pokedex number"
       },
       "maxVisits": 50,
       "maxExtracts": 10
    }'
  ```
</CodeGroup>

Use `/api/scrape` when you want the convenience of one call. Use `/api/crawl` + `/api/extract` directly when you want fine-grained control over each phase.


# Set crawl priorities
Source: https://docs.fetchfox.ai/setting-priorities

You can control which pages FetchFox visits during a crawl

You can use the `priority` parameter to control crawl scope.

The most important fields are:

* `only`: whitelist URL patterns to allow.
* `skip`: blacklist URL patterns to avoid.

If both are provided, FetchFox uses `only` to constrain candidates and `skip` to remove unwanted URLs.

Example:

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://api.fetchfox.ai/api/crawl \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $FETCHFOX_API_KEY" \
  -d '{
      "pattern":"https://example.com/shopping/*",
      "priority": {
        "only": [
          "https://example.com/shopping/*"
        ],
        "skip": [
          "https://example.com/shopping/jeans/*",
          "https://example.com/shopping/pants/*"
        ]
      },
      "maxVisits": 50
  }'
  ```
</CodeGroup>

This is useful when you only want one part of a large site.

Keep in mind: `priority` controls what the crawler chooses to visit. The final `results.hits` set is still based on crawl discovery and pattern matching.


# Set the starting URLs
Source: https://docs.fetchfox.ai/start-urls

You can tell FetchFox which URLs to start from

When you crawl for URLs [using a pattern](/crawl-using-patterns), FetchFox needs a set of URLs to start at. Those URLs can be set in one of two ways:

* **Automatic:** You can let FetchFox determine the starting URLs for you.
* **Explicit:** You can tell FetchFox which URLs to start crawling from.

Let's look at both of these options.

## Automatically determine the starting URLs

If you do not pass `startUrls`, FetchFox generates a small seed set from your `pattern`:

* The origin (for example `https://example.com`)
* Path prefixes derived from your pattern

## Explicitly setting the starting URLs

Use `startUrls` to explicitly define the seed URLs for a crawl.

Setting `startUrls` is helpful for crawling specific parts of a large site. It is especially useful with `maxDepth`, which limits the [maximum depth of a crawl](/limiting-crawls).

For example, suppose you are scraping commits on specific repos on GitHub. You can pass target repos in `startUrls`, then set `maxDepth: 0`.

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://api.fetchfox.ai/api/crawl \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $FETCHFOX_API_KEY" \
  -d '{
      "pattern":"https://github.com/*/commit/*",
      "startUrls": [
        "https://github.com/bitcoin/bitcoin/commits/master/",
        "https://github.com/torvalds/linux/commits/master/"
      ],
      "maxDepth": 0,
      "maxVisits": 50
  }'
  ```
</CodeGroup>

The call above will find commit URLs for the target repos, without wasting time on irrelevant parts of the site.


# Visit URLs
Source: https://docs.fetchfox.ai/visit

Fetch page content with /api/visit

The `/api/visit` endpoint fetches page content and returns one or more content representations.

## Key parameters

* `url`: target page URL
* `include`: list of content types to return

## Using `include`

Use `include: [...]` to choose which content representations FetchFox should return for the page.

The default include set is:

```json theme={null}
["markdown", "html", "urls"]
```

## Include options

Supported include values:

* `raw`: raw page HTML
* `html`: normalized HTML
* `slim_html`: reduced HTML
* `markdown`: markdown conversion
* `text`: text-only content
* `urls`: links found on the page
* `images`: image URLs found on the page
* `json_ld`: JSON-LD blocks
* `json_blobs`: JSON blobs found in the page
* `timer`: timing data

## Example

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://api.fetchfox.ai/api/visit \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "url": "https://pokemondb.net/pokedex/pikachu",
    "include": ["markdown", "html", "urls"]
  }'
  ```
</CodeGroup>

Typical response shape:

```json theme={null}
{
  "results": [
    {
      "url": "https://pokemondb.net/pokedex/pikachu",
      "status": "ok",
      "markdown": { "body": "...", "link": "https://..." },
      "html": { "body": "...", "link": "https://..." },
      "urls": { "body": ["https://..."], "link": "https://..." }
    }
  ],
  "metrics": { "...": "..." },
  "links": {
    "html": "https://..."
  },
  "html": "<!doctype html>..."
}
```

Response fields from `include` are returned in camelCase (for example, `slim_html` becomes `slimHtml`).