Hi all,
I have an API calls which gets items using the following URL:
https://xxx.com/API/1/50
In this URL, the 1 is the offset and the 50 is the pagesize. Which means in the next call the offset should be 51, after that 101 etc.
Normally I would use the follow pagination from the incoming, however the offset and pagesize are not in the body but directly in the URL.
Optimally; the API response returns the total number of items via a “count” property:
"Count": 1554022
What is the most optimal way to set up this incoming so that it paginates correctly for every item and does not continue after the count value?
Hi @j.wiegmann
A way to do this is to create the URL in the incoming with placeholders.
You would need a specific transformer to handle the pagination and addition the offset, each time the incoming will get triggered.
And construct the URL using the above logic.
When it gets to the end and no more data will come trough, reset the offset back to 1, so you start fresh.
Best is to set the cronJob to run more frequently, and of course use caching if required to filter the data.
Hi @catalin,
Thank you for the reply.
Your answer makes sense, however I struggle to find a way to “loop” through a transformer multiple times. I’ve tried using a conditional transformer “if offset < count; add pagesize to offset” but I can’t seem to get it to run additional times.
Hi @j.wiegmann.
The loop will work by running the incoming through a scheduler. So, we can track the process by using the transformer
and then for each incoming, we read this value first by using the Input transformer
we then potentially run the subscriber like this.
Please let me know if it works or if you have further issues, please raise a ticket to support@alumio.com
I got it to work using the above explanation and some help from Support. Thank you all!
1 Like
