Hi Alumio,
I had a case where I wanted to change the structure of a very large array (40.000+). This took hours, so I tried finding out where the actual problem was.
It turned out the problem is in the ‘Move using a Pattern’. Basically my config was something like this:
array.*
array.$1.key
Functionally in this particular case ‘Copy using a Pattern’ is the same as the ‘Move using a Pattern’. Since (either when doing this change or reverting back) you’re overriding the original value, they both lead to the same result. So it’s really a toss-up if a user will use the one or the other.
However, the ‘Move using a Pattern’ is significantly slower. It seems to scale badly with an increased array.
I’ve made an easy test:
- Chain Multiple
- Data Transformer
- Value Setter, with key
arrayand valuestring, and with the mapperList: fill with valuewith a configurable upper limit - Again a Data Transformer (so you’ll see it in the logs)
- Here either the Copy using a Pattern or Move using a Pattern, with the
array.*array.$1.key - Again a Data Transformer (so you’ll see it in the logs)
Outcome:
Upper limit of 1.000 items
Copy: 29ms
Move: 350ms
Upper limit of 5.000 items
Copy: 254ms
Move: 8000ms
Upper limit of 10.000 items
Copy: 222ms
Move: 18000ms
Upper limit of 20.000 items
Copy: 491ms
Move: crashes, can’t do this in a test-run of a transformer
Upper limit of 50.000 items
Copy: 1309ms
Move: crashes, can’t do this in a test-run of a transformer
Although when it’s known that one transformer is slower than another, that’s not really a big issue, it should be well known. But I’m guessing it won’t be very difficult to improve the performance of this Move, and will benefit everybody.
If it’s not possible to significantly improve performance of the ‘Move Using a Pattern’, then please add the note “NOTE: This is much slower than a copy!” to your documentation!