Combine entities but use pagination

iPaaS Core Incoming Configuration
1

We use the pagination feature to get multiple entities from a remote service. Therefor we must define the pattern for the items. And that produces tasks per item. However I want to process in bulk/batch per page, so keep one task per page. Is that achievable ?

2

Hi @b.hoekman

Welcome to the Alumio forum.

If you use the pagination method “Increase query parameter” or “Increase body parameter”, unfortunately, it’s required for you to define the pattern for the items, and the items will be separated. Therefore, it’s
not achievable at the moment.

Could you please confirm whether you want to publish multiple entities in a batch (single call/write) to the outgoing system?

5

Hi @Gugi , yes our outgoing system supports batching, i.e. sending stock records of multiple SKU’s in a single http call.
So if it was possible to keep the batches of 100 items together we would save approx 12.000 http calls.

6

Thank you for confirming.

Unfortunately, it is not yet possible to group a page of entities as a single task if you use the “Increase query/body parameter” pagination. However, we will forward this matter to the team for further discussion.

Could you please provide us with the information below?

  • Is this issue blocking your integration?
  • Could you please let us know what is the name of the system/platform of the data source (incoming) that you paginate?
7

No it is not blocking! It is just performance.
We get the data from Picqer WMS, but it is basically applicable to any source I would say.
There is a list of items, this list is used to count the next offset, the items are splitted up.
Would be nice that you independently can configure the batch size, similar to the CSV reader.

8

Thank you for the answers.

We have passed this on to the team for further discussion. We will let you know once we have an update.

12

Hi @b.hoekman

Thank you for your feature request and feedback.

We’re pleased to inform you that your suggestion has been accepted by our development team. However, we cannot commit to a specific timeline for its implementation just yet.

Thank you for your patience and for helping us improve our product!