Pagination
Last updated
Last updated
Many API endpoints that return a list of data split that data up into separate "pages", where each page is a subset of the data. This is called pagination.
Most APIs document their pagination method in a dedicated "Pagination" section or within specific endpoint documentation.
Check your API documentation for these patterns:
URL parameters: "page", "offset", "limit", "cursor", or "after"
Request body fields: Pagination parameters sent in the body instead of URL
Response fields: "next_url", "next_cursor", "has_more", or "total_pages"
Headers: Custom pagination headers like "page" or "x-page"
GraphQL: Cursor-based with "first", "after", or "before" arguments
Data Fetcher handles the most common types of pagination. Click any type below to jump to its setup instructions:
Uses a page number parameter in the URL. For example:
Page 1: http://example.com/?page=1
Page 2: https://www.example.com/?page=2
Uses offset and limit parameters in the URL to specify data range. For example:
Page 1: https://www.example.com/?offset=0&limit=50
Page 2: https://www.example.com/?offset=50&limit=100
Uses an ID from the previous response as a starting point parameter. For example:
Page 1: http://www.example.com
Page 2: https://www.example.com/?starting_after=cus_IOCwDqeBZGGsrF
The API response includes the complete URL for the next page. Sometimes provided in response headers as a Link header.
Uses offset and limit parameters in the request body. For example:
Page 1 body: { "startRow": 0, "rowLimit": 1000 }
Page 2 body: { "startRow": 1000, "rowLimit": 1000 }
Uses a cursor value from the previous response in the request body. For example:
Page 1 body: { "pageToken": "" }
Page 2 body: { "pageToken": "cus_IOCwDqeBZGGsrF" }
Uses a header value that increments with each request. For example:
Page 1 headers: page: 1
Page 2 headers: page: 2
Uses a path parameter in the URL that increments with each request. For example:
Page 1: https://api.example.com/stocks/1
Page 2: https://api.example.com/stocks/2
The GraphQL query takes a cursor argument, set to an ID from the previous response, to tell the API where to start the next page. For example, for this GraphQL query:
Page 1: after
would be set to null
Page 2: after
would be set to the endCursor
value (or the final ID
value for some APIs).
On the request screen, click to open the Advanced settings.
Scroll to Max response records, and enter the maximum number of records you want to receive.
Under Pagination, select the pagination type using the dropdown.
After selecting a pagination type, choose one of these options:
Fetch all pages: Automatically paginate through every page until no more records are returned. Test this option first, as it can sometimes cause infinite runs if the API always returns data.
Set a maximum number of pages: Enter a specific number of pages to fetch. Data Fetcher will stop when it reaches this limit, when no items are returned, or when a page has fewer records than the last one.
If you want to force pagination up to this number of pages even when fewer items are returned, click the settings icon (⚙️) next to the number of page input and enable Keep paginating when no/fewer items returned.
Set the number of pages to fetch or select Fetch all pages.
Enter the name of the page parameter. This is usually "page" but some APIs may use a different name.
If the API supports it, you can set a page size parameter and size, so that Data Fetcher knows when to stop paginating more reliably.
Set the name of the page size parameter.
Set the page size.
Set the number of pages to fetch or select Fetch all pages.
Enter the name of the Offset parameter. This is often called "offset" but some endpoints may use a different name:
Enter the name of the Limit parameter. This is often called "limit" but some endpoints may use a different name:
Enter the numerical Limit value. This is the number of items that are fetched in each page. The offset value will be increased by this amount for each request.
Optionally, enter a Starting offset. If you leave this blank, 0
will be used.
Set the number of pages to fetch or select Fetch all pages.
Enter the name of the cursor parameter:
Select the Cursor field from the output table fields using the dropdown.
If the API supports it, you can set a page size parameter to help Data Fetcher know when to stop paginating.
Set the name of the page size parameter.
Set the page size.
Set the number of pages to fetch or select Fetch all pages.
Select the Next URL field from the output table fields using the dropdown:
If the API supports it, you can set a page size parameter to help Data Fetcher know when to stop paginating.
Set the name of the page size parameter.
Set the page size.
Set the number of pages to fetch or select Fetch all pages.
Turn on Or, use link in response header.
Set the number of pages to fetch or select Fetch all pages.
Enter the Offset path for the request body. This uses JSONPath syntax (e.g., $.pagination.offset
).
Enter the Limit path for the request body. This uses JSONPath syntax (e.g., $.pagination.limit
).
Enter the Limit value.
Optionally, enter a Starting offset. If you leave this blank, 0 will be used.
Make sure you import the cursor field (usually an ID) on the response field mapping and run the request once to create the cursor field in the output table.
Set the pagination type to "Cursor Body".
Set the number of pages to fetch or select Fetch all pages.
Enter the Cursor path for the request body. This uses JSONPath syntax (e.g., $.pagination.cursor
).
Select the Cursor field from the output table fields in the dropdown.
Set the number of pages to fetch or select Fetch all pages.
Enter the name of Page header, e.g. "page".
Optionally, enter a Page size header.
Optionally, enter a Page size.
Optionally, enter a Starting page. If you leave this blank, 1 will be used.
In the request screen's main URL field, enter the URL with {}
around the path parameter, e.g. https://api.example.com/stocks/{page}
.
Back in the pagination settings, set the pagination type to "Path".
Set the number of pages to fetch or select Fetch all pages.
Enter the Path parameter (e.g. "page" in this example).
Optionally, enter a Page size parameter.
Optionally, enter a Page size.
Optionally, enter a Starting page. If you leave this blank, 1 will be used.
Make sure you import the cursor field (usually an ID) on the response field mapping and run the request once to create the cursor field in the output table.
In the request screen's main Body → GraphQL → Query section, enter the GraphQL query, including any pagination arguments.
There is no need to add them in the Variables section, as Data Fetcher will handle this automatically.
Set the number of pages to fetch or select Fetch all pages.
Enter the Cursor variable name.
Select the Cursor field from the dropdown. Typically, this is an ID field or a pagination cursor field in the response.
You can optionally enter a Page size variable or just a Page size value. This will help Data Fetcher know when to stop paginating if you have selected the Fetch all pages option.
to "Page".
to "Offset".
Make sure you import the cursor field (usually an ID) on the and run the request once to create the cursor field in the output table.
to "Cursor".
Make sure you import the cursor field (usually an id) on the and run the request once to create the next URL field in the output table.
to "Next URL".
to "Next URL".
to "Offset Body".
to "Header".
to "GraphQL Cursor".