Data Fetcher Help Center
🚀 Try for free📮 Support
  • 👋Welcome
  • Create Requests
    • Create a request
    • Use table values in requests
    • Use variables in requests
    • Response field mapping
    • Schedule a request
    • Add a request trigger
    • Troubleshoot errors
    • Run request using webhook URL
    • List of Airtable integrations
  • Custom Requests
    • Create a custom request
    • Set request method
    • Set request parameters
    • Request authorization
    • Set request headers
    • Add request body
    • Connect to a preset OAuth API
    • Create a custom OAuth connection
  • Advanced settings
    • Write modes
    • Update based on field(s)
    • Pagination
    • Response data transformations
    • Response JMESPath
    • Add fixed value to output
    • No output mode
    • Rate limit
    • Max response records
    • Run request from button field
    • Denormalize response
    • XML array handling
    • Fixed IP Address
  • Organise Requests
    • Run history
    • Duplicate a request
    • Delete a request
    • Add request description
    • Import request from cURL command
    • Export requests as JSON file
    • Import requests from JSON file
  • Create Sequences
    • Create a sequence
    • Schedule a sequence
    • Run sequence from button field
    • Run sequence using webhook URL
  • Account
    • Upgrade workspace
    • Authorize Data Fetcher and Airtable
    • Add user to workspace
    • Roles and permissions
    • View monthly usage
    • Update name or email address
    • Update payment card
    • Update billing information
    • View invoices
    • Switch workspace
  • Get a Google Maps API key
Powered by GitBook
On this page
  • How to identify your API's pagination type
  • Data Fetcher pagination types
  • Select pagination type
  • Set the number of pages to fetch
  • Set up page pagination
  • Set up offset pagination
  • Set up cursor pagination
  • Set up next URL pagination
  • Set up next URL pagination using Link header
  • Set up offset body pagination
  • Set up cursor body pagination
  • Set up header pagination
  • Set up path pagination
  • Set up GraphQL Cursor Pagination
  1. Advanced settings

Pagination

Last updated 10 days ago

Many API endpoints that return a list of data split that data up into separate "pages", where each page is a subset of the data. This is called pagination.

How to identify your API's pagination type

Most APIs document their pagination method in a dedicated "Pagination" section or within specific endpoint documentation.

Check your API documentation for these patterns:

  • URL parameters: "page", "offset", "limit", "cursor", or "after"

  • Request body fields: Pagination parameters sent in the body instead of URL

  • Response fields: "next_url", "next_cursor", "has_more", or "total_pages"

  • Headers: Custom pagination headers like "page" or "x-page"

  • GraphQL: Cursor-based with "first", "after", or "before" arguments

Data Fetcher pagination types

Data Fetcher handles the most common types of pagination. Click any type below to jump to its setup instructions:

Uses a page number parameter in the URL. For example:

  • Page 1: http://example.com/?page=1

  • Page 2: https://www.example.com/?page=2

Uses offset and limit parameters in the URL to specify data range. For example:

  • Page 1: https://www.example.com/?offset=0&limit=50

  • Page 2: https://www.example.com/?offset=50&limit=100

Uses an ID from the previous response as a starting point parameter. For example:

  • Page 1: http://www.example.com

  • Page 2: https://www.example.com/?starting_after=cus_IOCwDqeBZGGsrF

The API response includes the complete URL for the next page. Sometimes provided in response headers as a Link header.

Uses offset and limit parameters in the request body. For example:

  • Page 1 body: { "startRow": 0, "rowLimit": 1000 }

  • Page 2 body: { "startRow": 1000, "rowLimit": 1000 }

Uses a cursor value from the previous response in the request body. For example:

  • Page 1 body: { "pageToken": "" }

  • Page 2 body: { "pageToken": "cus_IOCwDqeBZGGsrF" }

Uses a header value that increments with each request. For example:

  • Page 1 headers: page: 1

  • Page 2 headers: page: 2

Uses a path parameter in the URL that increments with each request. For example:

  • Page 1: https://api.example.com/stocks/1

  • Page 2: https://api.example.com/stocks/2

The GraphQL query takes a cursor argument, set to an ID from the previous response, to tell the API where to start the next page. For example, for this GraphQL query:

query($after: String) {
  pullRequests(first: 100, after: $after) {
    nodes {
      id
      createdAt
      number
      title
    }
    pageInfo {
      endCursor
      hasNextPage
    }
  }
}
  • Page 1: after would be set to null

  • Page 2: after would be set to the endCursor value (or the final ID value for some APIs).

Select pagination type

  1. On the request screen, click to open the Advanced settings.

  2. Scroll to Max response records, and enter the maximum number of records you want to receive.

  3. Under Pagination, select the pagination type using the dropdown.

Set the number of pages to fetch

Each page that is fetched will use 1 out of your monthly workspace runs.

After selecting a pagination type, choose one of these options:

  • Fetch all pages: Automatically paginate through every page until no more records are returned. Test this option first, as it can sometimes cause infinite runs if the API always returns data.

  • Set a maximum number of pages: Enter a specific number of pages to fetch. Data Fetcher will stop when it reaches this limit, when no items are returned, or when a page has fewer records than the last one.

    • If you want to force pagination up to this number of pages even when fewer items are returned, click the settings icon (⚙️) next to the number of page input and enable Keep paginating when no/fewer items returned.

Set up page pagination

  1. Set the number of pages to fetch or select Fetch all pages.

  2. Enter the name of the page parameter. This is usually "page" but some APIs may use a different name.

  1. If the API supports it, you can set a page size parameter and size, so that Data Fetcher knows when to stop paginating more reliably.

    1. Set the name of the page size parameter.

    2. Set the page size.

Set up offset pagination

  1. Set the number of pages to fetch or select Fetch all pages.

  2. Enter the name of the Offset parameter. This is often called "offset" but some endpoints may use a different name:

  1. Enter the name of the Limit parameter. This is often called "limit" but some endpoints may use a different name:

  1. Enter the numerical Limit value. This is the number of items that are fetched in each page. The offset value will be increased by this amount for each request.

  1. Optionally, enter a Starting offset. If you leave this blank, 0 will be used.

Set up cursor pagination

  1. Set the number of pages to fetch or select Fetch all pages.

  2. Enter the name of the cursor parameter:

  1. Select the Cursor field from the output table fields using the dropdown.

  2. If the API supports it, you can set a page size parameter to help Data Fetcher know when to stop paginating.

    1. Set the name of the page size parameter.

    2. Set the page size.

Set up next URL pagination

  1. Set the number of pages to fetch or select Fetch all pages.

  2. Select the Next URL field from the output table fields using the dropdown:

  3. If the API supports it, you can set a page size parameter to help Data Fetcher know when to stop paginating.

    1. Set the name of the page size parameter.

    2. Set the page size.

Set up next URL pagination using Link header

  1. Set the number of pages to fetch or select Fetch all pages.

  2. Turn on Or, use link in response header.

Set up offset body pagination

  1. Set the number of pages to fetch or select Fetch all pages.

  2. Enter the Offset path for the request body. This uses JSONPath syntax (e.g., $.pagination.offset).

  3. Enter the Limit path for the request body. This uses JSONPath syntax (e.g., $.pagination.limit).

  4. Enter the Limit value.

  5. Optionally, enter a Starting offset. If you leave this blank, 0 will be used.

Set up cursor body pagination

  1. Make sure you import the cursor field (usually an ID) on the response field mapping and run the request once to create the cursor field in the output table.

  2. Set the pagination type to "Cursor Body".

  3. Set the number of pages to fetch or select Fetch all pages.

  4. Enter the Cursor path for the request body. This uses JSONPath syntax (e.g., $.pagination.cursor).

  5. Select the Cursor field from the output table fields in the dropdown.

Set up header pagination

  1. Set the number of pages to fetch or select Fetch all pages.

  2. Enter the name of Page header, e.g. "page".

  3. Optionally, enter a Page size header.

  4. Optionally, enter a Page size.

  5. Optionally, enter a Starting page. If you leave this blank, 1 will be used.

Set up path pagination

  1. In the request screen's main URL field, enter the URL with {} around the path parameter, e.g. https://api.example.com/stocks/{page} .

  2. Back in the pagination settings, set the pagination type to "Path".

  3. Set the number of pages to fetch or select Fetch all pages.

  4. Enter the Path parameter (e.g. "page" in this example).

  5. Optionally, enter a Page size parameter.

  6. Optionally, enter a Page size.

  7. Optionally, enter a Starting page. If you leave this blank, 1 will be used.

Set up GraphQL Cursor Pagination

  1. Make sure you import the cursor field (usually an ID) on the response field mapping and run the request once to create the cursor field in the output table.

  2. In the request screen's main Body → GraphQL → Query section, enter the GraphQL query, including any pagination arguments.

    1. There is no need to add them in the Variables section, as Data Fetcher will handle this automatically.

  3. Set the number of pages to fetch or select Fetch all pages.

  4. Enter the Cursor variable name.

  5. Select the Cursor field from the dropdown. Typically, this is an ID field or a pagination cursor field in the response.

    1. You can optionally enter a Page size variable or just a Page size value. This will help Data Fetcher know when to stop paginating if you have selected the Fetch all pages option.

to "Page".

to "Offset".

Make sure you import the cursor field (usually an ID) on the and run the request once to create the cursor field in the output table.

to "Cursor".

Make sure you import the cursor field (usually an id) on the and run the request once to create the next URL field in the output table.

to "Next URL".

to "Next URL".

to "Offset Body".

to "Header".

to "GraphQL Cursor".

response field mapping
response field mapping
1. Page
2. Offset
3. Cursor
4. Next URL
5. Offset Body
6. Cursor Body
7. Header
8. Path
9. GraphQL Cursor
Set the pagination type
Set the pagination type
Set the pagination type
Set the pagination type
Set the pagination type
Set the pagination type
Set the pagination type
Set the pagination type