Data Fetcher Help Center
🚀 Try for free📮 Support
  • 👋Welcome
  • Create Requests
    • Create a request
    • Use table values in requests
    • Use variables in requests
    • Response field mapping
    • Schedule a request
    • Add a request trigger
    • Troubleshoot errors
    • Run request using webhook URL
    • Pre-built integrations
  • Custom Requests
    • Create a custom request
    • Set request method
    • Set request parameters
    • Request authorization
    • Set request headers
    • Add request body
    • Connect to a preset OAuth API
    • Create a custom OAuth connection
  • Advanced settings
    • Write modes
    • Update based on field(s)
    • Pagination
    • Response data transformations
    • Response JMESPath
    • Add fixed value to output
    • No output mode
    • Rate limit
    • Max response records
    • Run request from button field
    • Denormalize response
    • XML array handling
    • Fixed IP Address
  • Organise Requests
    • Run history
    • Duplicate a request
    • Delete a request
    • Add request description
    • Import request from cURL command
    • Export requests as JSON file
    • Import requests from JSON file
  • Create Sequences
    • Create a sequence
    • Schedule a sequence
    • Run sequence from button field
    • Run sequence using webhook URL
  • Account
    • Upgrade workspace
    • Authorize Data Fetcher and Airtable
    • Add user to workspace
    • Roles and permissions
    • View monthly usage
    • Update personal details
    • Update payment card
    • Update billing information
    • View invoices
    • Switch workspace
  • Get a Google Maps API Key
Powered by GitBook
On this page
  • Select pagination type
  • Set the number of pages to fetch
  • Set up page pagination
  • Set up offset pagination
  • Set up cursor pagination
  • Set up next URL pagination
  • Set up next URL pagination using Link header
  • Set up offset body pagination
  • Set up cursor body pagination
  • Set up header pagination
  • Set up path pagination
  • Set up GraphQL Cursor Pagination
  1. Advanced settings

Pagination

Last updated 6 months ago

Many API endpoints that return a list of data split that data up into separate "pages", where each page is a subset of the data. This is called pagination.

Data Fetcher handles the most common types of pagination:

A page parameter is set in the URL to tell the API which page number of the data you want. e.g.

  • Page 1:http://example.com/?page=1

  • Page 2:https://www.example.com/?page=2

Offset and limit parameters are set in the URL to tell the API which slice of the data you want. e.g.

  • Page 1:https://www.example.com/?offset=0&limit=50

  • Page 2:https://www.example.com/?offset=50&limit=100

After the first request, a parameter in the URL is set to an ID from the previous response to tell the API where to start the next page. e.g.

  • Page 1:http://www.example.com

  • Page 2:https://www.example.com/?starting_after=cus_IOCwDqeBZGGsrF

After the first request, the API returns a field which contains the entire URL for the next page of data. This URL is used to make the next request. Sometimes this will be in the response headers as a header called Link . The examples here could be any of the previous three examples. The difference is how the next URL is accessed.

Offset and limit parameters are set in the request body to tell the API which slice of the data you want. e.g.

  • Page 1 body: { "startRow": 0, "rowLimit": 1000 }

  • Page 2 body: { "startRow": 1000, "rowLimit": 1000 }

After the first request, a field value in the request body is set to an id from the previous response to tell the API where to start the next page. e.g.

  • Page 1 body: { "pageToken": "" }

  • Page 2 body: { "pageToken": "cus_IOCwDqeBZGGsrF" }

A header is sent with each request. The value of the header is incremented each time. e.g.

  • Page 1 headers: page 1

  • Page 2 headers: page 2

A path parameter in the request URL is incremented with each request to determine the page. For example, in the URL https://api.example.com/stocks/{page} the {page} path parameter is replaced with the page. e.g.

  • Page 1:https://api.example.com/stocks/1

  • Page 2:https://api.example.com/stocks/2

The GraphQL query takes a cursor argument, set to an ID from the previous response, to tell the API where to start the next page. E.g., for this GraphQL query:

query($after: String) {
  pullRequests(first: 100, after: $after) {
    nodes {
      id
      createdAt
      number
      title
    }
    pageInfo {
      endCursor
      hasNextPage
    }
  }
}
  • Page 1: after would be set to null

  • Page 2: after would be set to the endCursor value (or the final id value for some APIs).

Some API documentation will have a Pagination section that explains the pagination type you should use for all endpoints. If not, it will be explained in the documentation for the specific API endpoint.

Select pagination type

  • Under Pagination, select the pagination type using the dropdown. You can set it to "None" to turn off pagination.

Set the number of pages to fetch

Each page that is fetched will use 1 out of your monthly workspace runs.

  • Enable Fetch all pages to paginate through every page until no/fewer items are returned. This can sometimes cause infinite runs if the API response always returns some data, so you will need to test it.

  • Alternatively, you can enter a maximum number of pages to fetch:

Set up page pagination

  • Enter the name of the page parameter. This is nearly always called "page" but some endpoints may use a different name:

If the API supports it, you can set a page size parameter and size, so that Data Fetcher knows when to stop paginating more reliably.

  • Set the name of the page size parameter:

  • Set the page size:

Set up offset pagination

  • Enter the name of the Offset parameter. This is often called "offset" but some endpoints may use a different name:

  • Enter the name of the Limit parameter. This is often called "limit" but some endpoints may use a different name:

  • Enter the numerical Limit value. This is the number of entities that are fetched in each page. The offset value will be increased by this amount for each request.

  • Optionally, enter a Starting offset. If you leave this blank, 0 will be used.

Set up cursor pagination

  • Enter the name of the cursor parameter:

  • Select the Cursor field from the output table fields using the dropdown:

If the API supports it, you can set a page size parameter to help Data Fetcher know when to stop paginating.

  • Set the name of the page size parameter:

  • Set the page size:

Set up next URL pagination

  • Select the Next URL field from the output table fields using the dropdown:

If the API supports it, you can set a page size parameter to help Data Fetcher know when to stop paginating.

  • Set the name of the page size parameter:

  • Set the page size:

Set up next URL pagination using Link header

  • Enable "Or, use link in response header".

Set up offset body pagination

  • Enter the Limit value.

  • Optionally, enter a Starting offset. If you leave this blank, 0 will be used.

Set up cursor body pagination

  • Select the Cursor field from the output table fields in the dropdown:

Set up header pagination

  • Enter the name of Page header, e.g. "page".

  • Optionally, enter a Page size header.

  • Optionally, enter a Page size.

  • Optionally, enter a Starting page. If you leave this blank, 1 will be used.

Set up path pagination

  • In the URL input, enter the URL with {} around the path parameter, e.g. https://api.example.com/stocks/{page} .

  • Enter the Path parameter, e.g. "page" in this example.

  • Optionally, enter a Page size parameter.

  • Optionally, enter a Page size.

  • Optionally, enter a Starting page. If you leave this blank, 1 will be used.

Set up GraphQL Cursor Pagination

  • In the Body -> GraphQL -> Query section, enter the GraphQL query, including any pagination parameters.

  • There is no need to add the pagination parameters in the GraphQL -> Variables section. Data Fetcher will do this automatically.

  • Enter the Cursor variable name.

  • Select the Cursor field. You should already have created this field in Airtable and / or mapped the relevant response field to this field. Typically, this is an ID field or a pagination cursor field in the response.

  • You can optionally enter a Page size variable or just a Page size value. This will help Data Fetcher know when to stop paginating if you have selected the Fetch all pages option.

On the create request screen, click to open the advanced settings.

After , these options will become visible:

When you've entered a maximum number of pages, Data Fetcher will stop paginating whenever no/fewer items are returned. You can force pagination up to the maximum by clicking next to the number input and enabling Keep paginating when no/fewer items returned on the modal that opens.

to "Page".

.

to "Offset".

.

Make sure you import the cursor field (usually an ID) on the and once to create the cursor field in the output table.

to "Cursor".

.

Make sure you import the cursor field (usually an id) on the and once to create the next URL field in the output table.

to "Next URL".

.

to "Next URL".

.

to "Offset Body".

.

Enter the Offset path for the request body. This uses .

Enter the Limit path for the request body. This uses .

Make sure you import the cursor field (usually an id) on the and once to create the cursor field in the output table.

to "Cursor Body".

.

Enter the Cursor path for the request body. This uses .

to "Header".

.

to "Path".

.

to "GraphQL Cursor".

.

JSONPath
JSONPath
JSONPath
1. Page
2. Offset
3. Cursor
4. Next URL
5. Offset Body
6. Cursor Body
7. Header
8. Path
9. GraphQL Cursor
selecting a pagination type
Set the pagination type
Set the number of pages to fetch
Set the pagination type
Set the number of pages to fetch
Set the pagination type
Set the number of pages to fetch
Set the pagination type
Set the number of pages to fetch
Set the pagination type
Set the number of pages to fetch
Set the pagination type
Set the number of pages to fetch
Set the pagination type
Set the number of pages to fetch
Set the pagination type
Set the number of pages to fetch
Set the pagination type
Set the number of pages to fetch
Set the pagination type
Set the number of pages to fetch
response field mapping
response field mapping
response field mapping
run the request
run the request
run the request