We’re running into issues where we need to do some transformation before the pagination takes place.
Let’s say we’re calling an API endpoint using an ‘offset’ parameter for pagination. Currently, Alumio supports this but there’s an issue there: the ‘offset’ param is calculated by Alumio by using the expected entities value. But often, you’ll pre-filter the dataset before they become tasks (i.e. by filtering out everything with no changes).
So what happens is Alumio expects 10.000 entities, but only 100 are delivered because 99.900 are filtered. So Alumio guesses it’s the last page and doesn’t ask the next page (with an offset of 10.000).
I suppose there are multiple ways around this, but the option I’d love is a transformer that’s ‘in between’ the result coming from the Subscriber and the pagination logic.
So in the example above, I could:
- Count the number in the array of the result (I expect 10.000, so anything less than that will result in it being the ‘last page’)
- If the result is 10.000, I create a node called “nextPage” with the URL to call with the added param (offset = 10.000)
- I set the Subscriber to use Pagination = Next Page
This would also solve another issue I sometimes encounter: when for example syncing pricelists, I use one call to get all the active ‘pricelist headers’ (with the code for the pricelist), and then a paginated call for all the prices within the pricelist. Currently there’s no way to create pagination based on other params then the page-number (or offset), but that’s exactly what I’d need in this case.
Thanks in advance!