Pagination

Many API endpoints that return a list of data split that data up into separate 'pages', where each page is a subset of the data. This is called pagination.
Data Fetcher handles the most common types of pagination:
​1. Page ​
A page parameter is set in the URL to tell the API which page number of data you want. e.g.
Page 1:http://example.com/?page=1
Page 2:https://www.example.com/?page=2
​2. Offset ​
Offset and limit parameters are set in the URL to tell the API which slice of data you want. e.g.
Page 1:https://www.example.com/?offset=0&limit=50
Page 2:https://www.example.com/?offset=50&limit=100
​3. Cursor ​
After the first request, a parameter in the URL is set to an id from the previous response to tell the API where to start the next page. e.g.
Page 1:http://www.example.com
Page 2:https://www.example.com/?starting_after=cus_IOCwDqeBZGGsrF
​4. Next URL ​
After the first request, the API returns a field which contains the entire URL for the next page of data. This URL is used to make the next request. Sometimes this will be in the response headers as a header called Link . The examples here could be any of the previous three examples. The difference is how the next URL is accessed.
​5. Offset Body​
Offset and limit parameters are set in the request body to tell the API which slice of data you want. e.g.
Page 1 body: { "startRow": 0, "rowLimit": 1000 }
Page 2 body: { "startRow": 1000, "rowLimit": 1000 }
​6. Cursor Body​
After the first request, a field value in the request body is set to an id from the previous response to tell the API where to start the next page. e.g.
Page 1 body: { "pageToken": "" }
Page 2 body: { "pageToken": "cus_IOCwDqeBZGGsrF" }
Some API documentation will have a "Pagination" section which explains the type you should use. If not, it will be explained in the documentation for the API endpoint.
Select pagination type
On the create request screen, click 
 to open the advanced settings.
Under Pagination, select the pagination type using the dropdown. You can set it to 'None' to turn off pagination.
Set the number of pages to fetch
After selecting a pagination type, these options will become visible:
Turn on "Fetch all pages" to paginate through every page until no data is returned. This can sometimes cause infinite runs to be made if the API response always returns some data, so you will need to test it.
Alternatively, you can enter a fixed number of pages to fetch using the input:
Set up page pagination
​Set the pagination type to 'Page'.
​Set the number of pages to fetch.
Enter the name of the page parameter. This is nearly always called 'page' but some endpoints may use a different name:
​
 
If the API supports it, you can set a page size parameter name and size, so that Data Fetcher knows when to stop paginating more reliably.
Set the name of the page size parameter:
 
​
Set the page size:
 
​
​
Set up offset pagination
​Set the pagination type to 'Offset'.
​Set the number of pages to fetch.
Enter the name of the Offset parameter. This is often called 'offset' but some endpoints may use a different name:
​
 
Enter the name of the Limit parameter. This is often called 'limit' but some endpoints may use a different name:
​
 
Enter the numerical Limit value. This is the number of entities that are fetched in each page. The offset value will be increased by this amount for each request.
​
 
Optionally, enter a Starting offset. If you leave this blank, 0 will be used.
Set up cursor pagination
Make sure you import the cursor field (usually an id) on the response field mapping and have run the request once to create the cursor field in the output table.
​Set the pagination type to 'Cursor'.
​Set the number of pages to fetch.
Enter the name of the cursor parameter:
​
 
Select the Cursor field from the output table fields using the dropdown:
If the API supports it, you can set a page size parameter to help Data Fetcher know when to stop paginating.
Set the name of the page size parameter:
 
​
Set the page size:
 
​
Set up next URL pagination
Make sure you import the cursor field (usually an id) on the response field mapping and have run the request once to create the next URL field in the output table.
​Set the pagination type to 'Next URL'.
​Set the number of pages to fetch.
Select the Next URL field from the output table fields using the dropdown:
If the API supports it, you can set a page size parameter to help Data Fetcher know when to stop paginating.
Set the name of the page size parameter:
 
​
Set the page size:
 
​
Set up next URL pagination using Link header
​Set the pagination type to 'Next URL'.
​Set the number of pages to fetch.
Enable "Or, use link in response header".
Set up offset body pagination
​Set the pagination type to 'Offset Body'.
​Set the number of pages to fetch.
Enter the Offset path for the request body. This uses JSONPath.
Enter the Limit path for the request body. This uses JSONPath.
Enter the Limit value.
Optionally, enter a Starting offset. If you leave this blank, 0 will be used.
Set up cursor body pagination
Make sure you import the cursor field (usually an id) on the response field mapping and have run the request once to create the cursor field in the output table.
​Set the pagination type to 'Cursor Body'.
​Set the number of pages to fetch.
Enter the Cursor path for the request body. This uses JSONPath.
Select the Cursor field from the output table fields in the dropdown:

Advanced settings - Previous

Update based on field

Next - Advanced settings

Response data transformations

Last modified 1mo ago

Copy link

On this page

Select pagination type

Set the number of pages to fetch

Set up page pagination

Set up offset pagination

Set up cursor pagination

Set up next URL pagination

Set up next URL pagination using Link header

Set up offset body pagination

Set up cursor body pagination

​1. Page ​

​2. Offset ​

​3. Cursor ​

​4. Next URL ​

​5. Offset Body​

​6. Cursor Body​

Select pagination type

Set the number of pages to fetch

Set up page pagination

Set up offset pagination

Set up cursor pagination

Set up next URL pagination

Set up next URL pagination using Link header

Set up offset body pagination

Set up cursor body pagination

1. Page

2. Offset

3. Cursor

4. Next URL

5. Offset Body

6. Cursor Body