Pagination

Many API endpoints that return a list of data split that data up into separate "pages", where each page is a subset of the data. This is called pagination.

How to identify your API's pagination type

Most APIs document their pagination method in a dedicated "Pagination" section or within specific endpoint documentation.

Check your API documentation for these patterns:

• URL parameters: "page", "offset", "limit", "cursor", or "after"

• Request body fields: Pagination parameters sent in the body instead of URL

• Response fields: "next_url", "next_cursor", "has_more", or "total_pages"

• Headers: Custom pagination headers like "page" or "x-page"

• GraphQL: Cursor-based with "first", "after", or "before" arguments

Data Fetcher pagination types

Data Fetcher handles the most common types of pagination. Click any type below to jump to its setup instructions:

1. Page

Uses a page number parameter in the URL. For example:

• Page 1: http://example.com/?page=1

• Page 2: https://www.example.com/?page=2

2. Offset

Uses offset and limit parameters in the URL to specify data range. For example:

• Page 1: https://www.example.com/?offset=0&limit=50

• Page 2: https://www.example.com/?offset=50&limit=100

3. Cursor

Uses an ID from the previous response as a starting point parameter. For example:

• Page 1: http://www.example.com

• Page 2: https://www.example.com/?starting_after=cus_IOCwDqeBZGGsrF

4. Next URL

The API response includes the complete URL for the next page. Sometimes provided in response headers as a Link header.

5. Offset Body

Uses offset and limit parameters in the request body. For example:

• Page 1 body: { "startRow": 0, "rowLimit": 1000 }

• Page 2 body: { "startRow": 1000, "rowLimit": 1000 }

6. Cursor Body

Uses a cursor value from the previous response in the request body. For example:

• Page 1 body: { "pageToken": "" }

• Page 2 body: { "pageToken": "cus_IOCwDqeBZGGsrF" }

7. Header

Uses a header value that increments with each request. For example:

• Page 1 headers: page: 1

• Page 2 headers: page: 2

8. Path

Uses a path parameter in the URL that increments with each request. For example:

• Page 1: https://api.example.com/stocks/1

• Page 2: https://api.example.com/stocks/2

9. GraphQL Cursor

The GraphQL query takes a cursor argument, set to an ID from the previous response, to tell the API where to start the next page. For example, for this GraphQL query:

query($after: String) {
  pullRequests(first: 100, after: $after) {
    nodes {
      id
      createdAt
      number
      title
    }
    pageInfo {
      endCursor
      hasNextPage
    }
  }
}

• Page 1: after would be set to null

• Page 2: after would be set to the endCursor value (or the final ID value for some APIs).

Select pagination type

1. On the request screen, click to open the Advanced settings.

2. Scroll to Max response records, and enter the maximum number of records you want to receive.

3. Under Pagination, select the pagination type using the dropdown.

Set the number of pages to fetch

Each page that is fetched will use 1 out of your monthly workspace runs.

After selecting a pagination type, choose one of these options:

• Fetch all pages: Automatically paginate through every page until no more records are returned. Test this option first, as it can sometimes cause infinite runs if the API always returns data.

• Set a maximum number of pages: Enter a specific number of pages to fetch. Data Fetcher will stop when it reaches this limit, when no items are returned, or when a page has fewer records than the last one.

  • If you want to force pagination up to this number of pages even when fewer items are returned, click the settings icon (⚙️) next to the number of page input and enable Keep paginating when no/fewer items returned.

Set up page pagination

1. Set the pagination type to "Page".

2. Set the number of pages to fetch or select Fetch all pages.

3. Enter the name of the page parameter. This is usually "page" but some APIs may use a different name.

1. If the API supports it, you can set a page size parameter and size, so that Data Fetcher knows when to stop paginating more reliably.

   1. Set the name of the page size parameter.

   2. Set the page size.

Set up offset pagination

1. Set the pagination type to "Offset".

2. Set the number of pages to fetch or select Fetch all pages.

3. Enter the name of the Offset parameter. This is often called "offset" but some endpoints may use a different name:

1. Enter the name of the Limit parameter. This is often called "limit" but some endpoints may use a different name:

1. Enter the numerical Limit value. This is the number of items that are fetched in each page. The offset value will be increased by this amount for each request.

1. Optionally, enter a Starting offset. If you leave this blank, 0 will be used.

Set up cursor pagination

1. Make sure you import the cursor field (usually an ID) on the response field mapping and run the request once to create the cursor field in the output table.

2. Set the pagination type to "Cursor".

3. Set the number of pages to fetch or select Fetch all pages.

4. Enter the name of the cursor parameter:

1. Select the Cursor field from the output table fields using the dropdown.

2. If the API supports it, you can set a page size parameter to help Data Fetcher know when to stop paginating.

   1. Set the name of the page size parameter.

   2. Set the page size.

Set up next URL pagination

1. Make sure you import the cursor field (usually an id) on the response field mapping and run the request once to create the next URL field in the output table.

2. Set the pagination type to "Next URL".

3. Set the number of pages to fetch or select Fetch all pages.

4. Select the Next URL field from the output table fields using the dropdown:

5. If the API supports it, you can set a page size parameter to help Data Fetcher know when to stop paginating.

   1. Set the name of the page size parameter.

   2. Set the page size.

Set up next URL pagination using Link header

1. Set the pagination type to "Next URL".

2. Set the number of pages to fetch or select Fetch all pages.

3. Turn on Or, use link in response header.

Set up offset body pagination

1. Set the pagination type to "Offset Body".

2. Set the number of pages to fetch or select Fetch all pages.

3. Enter the Offset path for the request body. This uses JSONPath syntax (e.g., $.pagination.offset).

4. Enter the Limit path for the request body. This uses JSONPath syntax (e.g., $.pagination.limit).

5. Enter the Limit value.

6. Optionally, enter a Starting offset. If you leave this blank, 0 will be used.

Set up cursor body pagination

1. Make sure you import the cursor field (usually an ID) on the response field mapping and run the request once to create the cursor field in the output table.

2. Set the pagination type to "Cursor Body".

3. Set the number of pages to fetch or select Fetch all pages.

4. Enter the Cursor path for the request body. This uses JSONPath syntax (e.g., $.pagination.cursor).

5. Select the Cursor field from the output table fields in the dropdown.

Set up header pagination

1. Set the pagination type to "Header".

2. Set the number of pages to fetch or select Fetch all pages.

3. Enter the name of Page header, e.g. "page".

4. Optionally, enter a Page size header.

5. Optionally, enter a Page size.

6. Optionally, enter a Starting page. If you leave this blank, 1 will be used.

Set up path pagination

1. In the request screen's main URL field, enter the URL with {} around the path parameter, e.g. https://api.example.com/stocks/{page} .

2. Back in the pagination settings, set the pagination type to "Path".

3. Set the number of pages to fetch or select Fetch all pages.

4. Enter the Path parameter (e.g. "page" in this example).

5. Optionally, enter a Page size parameter.

6. Optionally, enter a Page size.

7. Optionally, enter a Starting page. If you leave this blank, 1 will be used.

Set up GraphQL Cursor Pagination

1. Make sure you import the cursor field (usually an ID) on the response field mapping and run the request once to create the cursor field in the output table.

2. In the request screen's main Body → GraphQL → Query section, enter the GraphQL query, including any pagination arguments.

   1. There is no need to add them in the Variables section, as Data Fetcher will handle this automatically.

3. Set the pagination type to "GraphQL Cursor".

4. Set the number of pages to fetch or select Fetch all pages.

5. Enter the Cursor variable name.

6. Select the Cursor field from the dropdown. Typically, this is an ID field or a pagination cursor field in the response.

   1. You can optionally enter a Page size variable or just a Page size value. This will help Data Fetcher know when to stop paginating if you have selected the Fetch all pages option.