Memory issue: Move Between Accessors

Alumio Feedback
1

Hi Alumio,

We don’t often use the ‘move between accessors’, but it’s very useful in a lot scenario’s. Especially when you have a structure like this:

{
  "values": {
    "name": {
      "data": "Product X"
    },
    "ean": {
      "data": "203958093285"
    }
}

And you’d like a structure like this:

{
  "values": [
    {
      "key": "name",
      "value": {
        "data": "Product X"
      }
    },
    {
      "key": "ean",
      "value": [
        {
          "data": "203958093285"
        }
      ]
    }
  ]
}

However, the Move Between Accessors uses a lot of memory and is fairly slow.

move-between-accessors-memory-issue-not-working-example_transformer.ndjson (1.0 KB)

Attached a small transformer. It creates an array of objects (with one key) with 5.000 items and then uses the Move Between Accessors. It breaks due to memory issues.

3

Hi @floris

Thank you for your valuable feedback and for reporting this issue. We appreciate you taking the time to share your findings with us.

We acknowledge that this transformer is indeed quite heavy. Therefore, we will escalate this issue to our Production team for further investigation. We will provide an update as soon as we receive feedback from our development team.

Thank you for your patience.

5

Hi @floris

Can you let us know whether you have any plans to use this case for your projects in the near future? and using large dataset too?

6

Hi Arief,

Pretty much continually, but we have ways around it. No real hurry from us.