Bug: Move using a Pattern significantly slower than Copy using a Pattern

Alumio Feedback
1

Hi Alumio,

I had a case where I wanted to change the structure of a very large array (40.000+). This took hours, so I tried finding out where the actual problem was.

It turned out the problem is in the ‘Move using a Pattern’. Basically my config was something like this:
array.*
array.$1.key

Functionally in this particular case ‘Copy using a Pattern’ is the same as the ‘Move using a Pattern’. Since (either when doing this change or reverting back) you’re overriding the original value, they both lead to the same result. So it’s really a toss-up if a user will use the one or the other.

However, the ‘Move using a Pattern’ is significantly slower. It seems to scale badly with an increased array.

I’ve made an easy test:

  • Chain Multiple
  • Data Transformer
  • Value Setter, with key array and value string, and with the mapper List: fill with value with a configurable upper limit
  • Again a Data Transformer (so you’ll see it in the logs)
  • Here either the Copy using a Pattern or Move using a Pattern, with the array.* array.$1.key
  • Again a Data Transformer (so you’ll see it in the logs)

Outcome:
Upper limit of 1.000 items
Copy: 29ms
Move: 350ms

Upper limit of 5.000 items
Copy: 254ms
Move: 8000ms

Upper limit of 10.000 items
Copy: 222ms
Move: 18000ms

Upper limit of 20.000 items
Copy: 491ms
Move: crashes, can’t do this in a test-run of a transformer

Upper limit of 50.000 items
Copy: 1309ms
Move: crashes, can’t do this in a test-run of a transformer

Although when it’s known that one transformer is slower than another, that’s not really a big issue, it should be well known. But I’m guessing it won’t be very difficult to improve the performance of this Move, and will benefit everybody.

If it’s not possible to significantly improve performance of the ‘Move Using a Pattern’, then please add the note “NOTE: This is much slower than a copy!” to your documentation!

1 Like
2

I’ve found another one with a similar issue: Key Filters using a pattern with an array.

So for example a Key Filter with a Pattern: array.*.key. I’ve compared this to a Node, Transform Nodes with a Key Filter without the *.

I’ve used my previous test scenario:
Upper limit of 5.000
Key Filter: 16671ms
Node Transformer with Key Filter: 222ms

Upper limit of 10.000
Key Filter: 67000ms (yes, more than a minute!)
Node Transformer with Key Filter: 488ms

3

Hi @floris,

Thank you for your feedback. We will pass this to the corresponding team to further checking.