Pagination

Most API endoints that return a list of data split that data up into 'pages', and return one page at a time, rather than everything at once. This is known as pagination and can be handled within Data Fetcher.
Data Fetcher handles the most common types of pagination:
1. Page 
A page parameter is set in the URL to tell the API which page of data you want. e.g.
Page 1:http://example.com/?page=1
Page 2:https://www.example.com/?page=2
2. Offset 
Offset and limit parameters are set in the URL to tell the API which slice of data you want. e.g.
Page 1:https://www.example.com/?offset=0&limit=50
Page 2:https://www.example.com/?offset=50&limit=100
3. Cursor 
After the first request, a parameter in the URL is set to an entity id from the previous response to tell the API where to start. e.g.
Page 1:http://www.example.com
Page 2:https://www.example.com/?starting_after=cus_IOCwDqeBZGGsrF
4. Next URL 
After the first request, the API returns a field which contains the entire URL for the next page of data. This URL is used to make the next request. Sometimes this will be in the response headers as a header called Link . 
5. Offset Body
Offset and limit parameters are set in the request body to tell the API which slice of data you want. e.g.
Page 1 body: { "startRow": 0, "rowLimit": 1000 }
Page 2 body: { "startRow": 1000, "rowLimit": 1000 }
6. Cursor Body
Note: Cursor Body pagination currently does not work for manual runs, only for scheduled runs. This will be fixed soon!
After the first request, a field value in the request body is set to an entity id from the previous response to tell the API where to start. e.g.
Page 1 body: { "pageToken": "" }
Page 2 body: { "pageToken": "cus_IOCwDqeBZGGsrF" }
Select pagination type
Click 
  to open the advanced options.
Under Pagination, select the pagination type you want from the dropdown. You can also set it to 'None' to turn off pagination.
Set the number of pages to fetch
After selecting a pagination type, these options will become visible:
​
Turn on 
and Data Fetcher will make requests for each page until no data is returned. This can sometimes cause infinite requests to be made if the API response always returns some data.
Alternatively, you can enter the number of pages you want to fetch:
.
Set up page pagination
​Set the pagination type to 'Page'.
​Set the number of pages to fetch.
Enter the name of the page parameter. This is nearly always called 'page' but some endpoints may use a different name:
​
 
Set up offset pagination
​Set the pagination type to 'Offset'.
​Set the number of pages to fetch.
Enter the name of the Offset parameter. This is often called 'offset' but some endpoints may use a different name:
​
 
Enter the name of the Limit parameter. This is often called 'limit' but some endpoints may use a different name:
​
 
Enter the numerical Limit value. This is the number of entities that are fetched in each request. The offset value will be increased by this amount for each request.
​
 
Optionally, enter a Starting offset. If you leave this blank, 0 will be used.
Set up cursor pagination
You will need to have run the request at least once to have created the fields in the output table.
​Set the pagination type to 'Cursor'.
​Set the number of pages to fetch.
Enter the name of the cursor parameter:
​
 
Select the cursor field from the output table fields in the dropdown:
Set up next URL pagination
You will need to have run the request or created the response field mapping to have created the fields in the output table.
​Set the pagination type to 'Next URL'.
​Set the number of pages to fetch.
Select the next URL field from the output table fields in the dropdown:
Set up next URL pagination using Link header
​Set the pagination type to 'Next URL'.
​Set the number of pages to fetch.
Enter the name of the offset parameter. This is often called 'offset' but some endpoints may use a different name:
Set up offset body pagination
​Set the pagination type to 'Offset Body'.
​Set the number of pages to fetch.
Enter the Offset path for the request body. This uses JSONPath.
Enter the Limit path for the request body. This uses JSONPath.
Enter the Limit value.
Optionally, enter a Starting offset. If you leave this blank, 0 will be used.
Set up cursor body pagination
Note: Cursor Body pagination currently does not work for manual runs, only for scheduled runs. This will be fixed soon!
You will need to have run the request or created the response field mapping to have created the fields in the output table.
​Set the pagination type to 'Cursor Body'.
​Set the number of pages to fetch.
Enter the Cursor path for the request body. This uses JSONPath.
Select the Cursor field from the output table fields in the dropdown: