> ## Documentation Index
> Fetch the complete documentation index at: https://help.datafetcher.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Pagination
Many API endpoints that return a list of data split that data up into separate "pages", where each page is a subset of the data. This is called pagination. It is easy to handle all types of pagination in Data Fetcher.
## How to identify your API's pagination type
Most APIs document their pagination method in a dedicated "Pagination" section or within specific endpoint documentation.
Check your API documentation for these patterns:
* **URL parameters:** "page", "offset", "limit", "cursor", or "after"
* **Request body fields:** Pagination parameters sent in the body instead of URL
* **Response fields:** "next\_url", "next\_cursor", "has\_more", or "total\_pages"
* **Headers:** Custom pagination headers like "page" or "x-page"
* **GraphQL:** Cursor-based with "first", "after", or "before" arguments
## Data Fetcher pagination types
Data Fetcher handles the most common types of pagination. Click any type below to jump to its setup instructions:
1. Page
Uses a page number parameter in the URL. For example:
* Page 1: `https://www.example.com/?page=1`
* Page 2: `https://www.example.com/?page=2`
2. Offset
Uses offset and limit parameters in the URL to specify data range. For example:
* Page 1: `https://www.example.com/?offset=0&limit=50`
* Page 2: `https://www.example.com/?offset=50&limit=100`
3. Cursor
Uses an ID from the previous response as a starting point parameter. For example:
* Page 1: `https://www.example.com`
* Page 2: `https://www.example.com/?starting_after=cus_IOCwDqeBZGGsrF`
4. Next URL
The API response includes the complete URL for the next page. Sometimes provided in response headers as a Link header.
5. Offset body
Uses offset and limit parameters in the request body. For example:
* Page 1 body: `{ "startRow": 0, "rowLimit": 1000 }`
* Page 2 body: `{ "startRow": 1000, "rowLimit": 1000 }`
6. Cursor body
Uses a cursor value from the previous response in the request body. For example:
* Page 1 body: `{ "pageToken": "" }`
* Page 2 body: `{ "pageToken": "cus_IOCwDqeBZGGsrF" }`
7. Page body
Use a page number field in the body. For example:
* Page 1 body: `{ "page": 1 }`
* Page 2 body: `{ "page": 2 }`
8. Header
Uses a header value that increments with each request. For example:
* Page 1 headers: `page: 1`
* Page 2 headers: `page: 2`
9. Path
Uses a path parameter in the URL that increments with each request. For example:
* Page 1: `https://api.example.com/stocks/1`
* Page 2: `https://api.example.com/stocks/2`
10. GraphQL Cursor
The GraphQL query takes a cursor argument, set to an ID from the previous response, to tell the API where to start the next page. For example, for this GraphQL query:
```
query($after: String) {
pullRequests(first: 100, after: $after) {
nodes {
id
createdAt
number
title
}
pageInfo {
endCursor
hasNextPage
}
}
}
```
* Page 1: `after` would be set to null
* Page 2: `after` would be set to the `endCursor` value (or the final `ID` value for some APIs).
## Select pagination type
1. On the request screen, click to open the **Advanced settings**.
2. Scroll to **Max response records**, and enter the maximum number of records you want to receive.
3. Under **Pagination**, select the pagination type using the dropdown.
## Set the number of pages to fetch
Each page that is fetched will use 1 out of your monthly workspace runs.
After selecting a pagination type, choose one of these options:
* **Fetch all pages:** Automatically paginate through every page until no more records are returned. Test this option first, as it can sometimes cause infinite runs if the API always returns data.
* Set a maximum **number of pages:** Enter a specific number of pages to fetch. Data Fetcher will stop when it reaches this limit, when no items are returned, or when a page has fewer records than the last one.
* If you want to force pagination up to this number of pages even when fewer items are returned, click the settings icon (⚙️) next to the number of page input and enable **Keep paginating when no/fewer items returned**.
## Set up pagination
### Set up page pagination
1. Set the pagination type to "Page".
2. Set the **number of pages** to fetch or select **Fetch all pages.**
3. Enter the name of the **page parameter**. This is usually "page" but some APIs may use a different name.
4. If the API supports it, you can set a page size parameter and size, so that Data Fetcher knows when to stop paginating more reliably.
1. Set the name of the **page size parameter.**
2. Set the **page size.**
### Set up offset pagination
1. Set the pagination type to "Offset".
2. Set the **number of pages** to fetch or select **Fetch all pages.**
3. Enter the name of the **Offset parameter**. This is often called "offset" but some endpoints may use a different name.
4. Enter the name of the **Limit parameter**. This is often called "limit" but some endpoints may use a different name.
5. Enter the numerical **Limit value**. This is the number of items that are fetched in each page. The offset value will be increased by this amount for each request.
6. Optionally, enter a **Starting offset**. If you leave this blank, `0` will be used.
### Set up cursor pagination
1. Make sure you import the cursor field (usually an ID) on the [response field mapping](https://help.datafetcher.com/create-requests/response-field-mapping) and run the request once to create the cursor field in the destination table.
2. Set the pagination type to "Cursor".
3. Set the **number of pages** to fetch or select **Fetch all pages.**
4. Enter the name of the **cursor parameter**.
5. Select the **Cursor field** from the destination table fields using the dropdown.
6. If the API supports it, you can set a page size parameter to help Data Fetcher know when to stop paginating.
1. Set the name of the **page size parameter**.
2. Set the **page size**.
### Set up next URL pagination
1. Make sure you import the cursor field (usually an id) on the [response field mapping](https://help.datafetcher.com/create-requests/response-field-mapping) and run the request once to create the next URL field in the destination table.
2. Set the pagination type to "Next URL".
3. Set the **number of pages** to fetch or select **Fetch all pages.**
4. Select the **Next URL field** from the destination table fields using the dropdown:
5. If the API supports it, you can set a page size parameter to help Data Fetcher know when to stop paginating.
1. Set the name of the page size parameter.
2. Set the page size.
### Set up next URL pagination using Link header
1. Set the pagination type to "Next URL".
2. Set the **number of pages** to fetch or select **Fetch all pages.**
3. Turn on **Or, use link in response header**.
### Set up offset body pagination
1. Set the pagination type to "Offset Body".
2. Set the **number of pages** to fetch or select **Fetch all pages.**
3. Enter the **Offset path** for the request body. This uses JSONPath syntax (e.g., `$.pagination.offset`).
4. Enter the **Limit path** for the request body. This uses JSONPath syntax (e.g., `$.pagination.limit`).
5. Enter the **Limit value.**
6. Optionally, enter a **Starting offset**. If you leave this blank, 0 will be used.
### Set up cursor body pagination
1. Make sure you select the cursor field (usually an ID) on the response field mapping and run the request once to create and populate the cursor field in the destination table.
2. Set the pagination type to "Cursor Body".
3. Set the **number of pages** to fetch or select **Fetch all pages.**
4. Enter the **Cursor path** for the request body. This uses JSONPath syntax (e.g., `$.pagination.cursor`).
5. Select the **Cursor field** from the destination table fields in the dropdown.
### Set up page body pagination
1. Set the pagination type to "Page Body".
2. Enter the **Page path** for the request body. This uses JSONPath syntax (e.g., `$.page`).
3. If the API supports it, you can set a **page size path** (e.g. `$.pageSize`) and **page size** (e.g. `100`), so that Data Fetcher knows when to stop paginating more reliably.
4. Enter the **Starting page**, or leave it blank to use `1` as the starting page.
### Set up header pagination
1. Set the pagination type to "Header".
2. Set the **number of pages** to fetch or select **Fetch all pages.**
3. Enter the name of **Page header**, e.g. "page".
4. Optionally, enter a **Page size header**.
5. Optionally, enter a **Page size**.
6. Optionally, enter a **Starting page**. If you leave this blank, 1 will be used.
### Set up path pagination
1. In the request screen's main **URL** field, enter the URL with `{}` around the path parameter, e.g. `https://api.example.com/stocks/{page}` .
2. Back in the pagination settings, set the pagination type to "Path".
3. Set the **number of pages** to fetch or select **Fetch all pages.**
4. Enter the **Path parameter** (e.g. "page" in this example).
5. Optionally, enter a **Page size parameter**.
6. Optionally, enter a **Page size**.
7. Optionally, enter a **Starting page**. If you leave this blank, 1 will be used.
### Set up GraphQL Cursor Pagination
1. Make sure you import the cursor field (usually an ID) on the response field mapping and run the request once to create the cursor field in the destination table.
2. In the request screen's main **Body → GraphQL → Query** section, enter the GraphQL query, including any pagination arguments.
1. There is no need to add them in the **Variables** section, as Data Fetcher will handle this automatically.
3. Set the pagination type to "GraphQL Cursor".
4. Set the **number of pages** to fetch or select **Fetch all pages.**
5. Enter the **Cursor variable** name.
6. Select the **Cursor field** from the dropdown. Typically, this is an ID field or a pagination cursor field in the response.
7. You can optionally enter a **Page size variable** or just a **Page size** value. This will help Data Fetcher know when to stop paginating if you have selected the **Fetch all pages** option.