Following Pagination in HTTP Transformer

General
1

Hello,

I’m currently working on an integration where I have a waterfall structure of data that I need to retrieve. in the incoming I retrieve the “grandparents”. Then furthermore for each grandparent I need to get all the parents and for those all the children. Since the HTTP Subscriber can follow pagination but the HTTP transformer can not. What would my best bet be to retrieve all this data. All 3 calls have pagination.

3

Hi Bas,

Thanks for the detailed question! You’re right that the HTTP Transformer doesn’t natively follow pagination like the HTTP Subscriber does. However, there are a couple of transformers that could help you handle this within your transformation flow as a workaround:

1. While Loop Transformer The While Loop Transformer allows you to repeat an HTTP call until a defined condition is met, for example, until there is no nextPage or the result set is empty. This makes it possible to keep fetching paginated results directly inside a transformer, without needing an external scheduler or a separate route.

2. Global Variable Setter Transformer To carry pagination state (e.g. page, offset, or cursor) across iterations within the loop, you can use the Global Variable Setter Transformer. It lets you store a temporary variable (e.g. @{globals.next_page}) that persists for the duration of the process and can be referenced as a placeholder in your HTTP Transformer calls.

Suggested Approach:

  • Use your HTTP Subscriber (With Merger Transformer) to fetch the grandparents (pagination handled natively).
  • For each grandparent, use a While Loop Transformer + HTTP Transformer to fetch all parents across pages, tracking the current page/cursor via Global Variable Setter.
  • For each parent, apply the same pattern to fetch all children.

Important Considerations – Memory & Execution Time Since you’re dealing with a 3-level waterfall structure where each level has its own pagination, please be mindful of the following:

  • Memory usage – All data retrieved across iterations is held in memory during the process. The more grandparents, parents, and children you have, and the more pages each level requires, the more memory will be consumed per task run. Monitor this carefully, especially if the datasets are large.
  • Execution duration – Each HTTP call within the loop adds to the total processing time. With multiple levels of nested pagination, the number of calls can grow quickly (e.g. 10 grandparents × 5 pages of parents × 10 pages of children = potentially hundreds of HTTP calls per task). Make sure your task timeout settings allow sufficient time to complete.
  • Rate limiting – Be aware of any API rate limits on the third-party system, as a high volume of sequential calls may trigger throttling.

It’s a good idea to test with a small subset of data first to estimate the total call volume and resource impact before running it at full scale.

Attached below the very simple sample configuration for while-loop with Global variables and HTTP Transformer, you can try to import it to your environment and let us know whether it’s works.

sample-while-loop_transformer.ndjson (2.5 KB)