"Pagination Transformer" on HTTP Calls

Hi Alumio,

We’re running into issues where we need to do some transformation before the pagination takes place.

Let’s say we’re calling an API endpoint using an ‘offset’ parameter for pagination. Currently, Alumio supports this but there’s an issue there: the ‘offset’ param is calculated by Alumio by using the expected entities value. But often, you’ll pre-filter the dataset before they become tasks (i.e. by filtering out everything with no changes).

So what happens is Alumio expects 10.000 entities, but only 100 are delivered because 99.900 are filtered. So Alumio guesses it’s the last page and doesn’t ask the next page (with an offset of 10.000).

I suppose there are multiple ways around this, but the option I’d love is a transformer that’s ‘in between’ the result coming from the Subscriber and the pagination logic.

So in the example above, I could:

  • Count the number in the array of the result (I expect 10.000, so anything less than that will result in it being the ‘last page’)
  • If the result is 10.000, I create a node called “nextPage” with the URL to call with the added param (offset = 10.000)
  • I set the Subscriber to use Pagination = Next Page

This would also solve another issue I sometimes encounter: when for example syncing pricelists, I use one call to get all the active ‘pricelist headers’ (with the code for the pricelist), and then a paginated call for all the prices within the pricelist. Currently there’s no way to create pagination based on other params then the page-number (or offset), but that’s exactly what I’d need in this case.

Thanks in advance!

Hi @floris

Thank you for your valuable feedback.

You are correct that Alumio calculates the next offset parameter by using the “Expected number of items” field value. You are also correct that Alumio stops fetching the next page when it receives entities less than the number set in “Expected number of items” field.

Therefore, we will pass your feedback on to the team for further discussion of the implementation possibilities. We will be sure to get back to you once we have an update.

Feel free to let us know if you have any other feedback or ideas. Once again, thank you for your feedback.