Table Of Contents
- Use Case
- Application of List: split into Chunks Mapper
Use Case
Let’s say you have a lot of data with prices and you need to push 1 million prices. If you push them one by one, it will take too long, but if you push 1 million at once, it won’t work as it would take too long to process. We need to find the solution somewhere in-between.
Instead of sending requests with 1 million prices or 1 million requests, we can send, for example, 10,000 requests with 100 prices. For this, we can use the value mapper List: split into chunks, and we can define the Length of each batch.
List: split into chunks mapper gives a possibility to send multiple price changes, products, or customers with one request.
For example, for some systems, it can take 4 seconds to process one price. But if you send 100 prices in the same request, it only takes 7 seconds. Even though it takes 3 seconds longer, it can process 99 more prices, which is more efficient.
Application of List: split into Chunks Mapper
You need to build up an entity transformer and use List: split into chunks mapper within it.
For this, create a new transformer:
In Settings select Data, transform data using mappers and conditions
Transformers: Value mapper
In the Accessors field choose one of the following:
Key accessor - is used to target the key of the entity data.
Pattern accessor - is used to target the root of the entity data.
In Mappers choose List: Split into Chunks
In the Length field define the size of each chunk by putting an integer.
In the example transformer in the screenshot below, an HTTP transformer is asked to pull the data and then split it into batches.
As we can see, the data pulled from the URL defined in an HTTP transformer was split into chunks in the Result field/