Manipulating and iterating over arrays

iPaaS Transformers General
1

So, I have two sets, as arrays, of options, one existing (that I looked up in the destination system) and one new (from the exporting system) - here simplified:

{
    "id":"1"
    "existingOptions":[
        {
            "propId":"1",
            "optionId":"2"
        },
        {
            "propId":"2",
            "optionId":"34"
        }
    ],
    "newOptions":[
        {
            "propId":"1",
            "optionId":"2"
        },
        {
            "propId":"3",
            "optionId":"5"
        }
    ]
}

I want to compare and find which objects need to be created, updated and deleted at the destination system. Since node transformers don’t have access to outside objects I will copy the data I need into each one of them. So, here, copy root id into each newOptions object, and each of the arrays into each of the other array objects.

$.newOptions[*].id = $.id
$.existingOptions[*].newOptions = $.newOptions
$.newOptions[*].existingOptions = $.existingOptions

Now that we have the necessary data accessible inside each node, we can iterate through them (with the “Node, transform nodes” transformer) and set some flags (as key-values).

First we go through the newOptions, and set “method” keys:
if propId exists in existingOptions set “PUT” (update), else set “POST” (create).

Secondly, in existingOptions, I need to find which propId’s are NOT in the newOptions. There, add the key “method”: “DELETE”.
$.existingOptions[?(!($.propId in $.newOptions[*].propId))].method = "DELETE"

Then, copy that object to the newOptions.
$.newOptions = $.newOptions + $.existingOptions[0]

expected result JSON:

{
    "id": "1",
    "existingOptions": [
        {
            "propId": "1",
            "optionId": "2",
            "newOptions": [
                {
                    "propId": "1",
                    "optionId": "2"
                },
                {
                    "propId": "3",
                    "optionId": "5"
                }
            ]
        },
        {
            "propId": "2",
            "optionId": "34",
            "newOptions": [
                {
                    "propId": "1",
                    "optionId": "2"
                },
                {
                    "propId": "3",
                    "optionId": "5"
                }
            ]
        }
    ],
    "newOptions": [
        {
            "propId": "1",
            "optionId": "2",
            "id": "1",
            "existingOptions": [
                {
                    "propId": "1",
                    "optionId": "2",
                 },
                {
                    "propId": "2",
                    "optionId": "34",
                  }
            ],
            "method": "PUT"
        },
        {
            "propId": "3",
            "optionId": "5",
            "id": "1",
            "existingOptions": [
                {
                    "propId": "1",
                    "optionId": "2",
                 },
                {
                    "propId": "2",
                    "optionId": "34",
                }
            ],
            "method": "POST"
        },
        {
            "propId": "2",
            "optionId": "34",
            "id": "1",
            "method": "DELETE"
        }
    ]
}

Now, finally I can make a new iteration over the newOptions and make those requests with a HTTP transformer.

Hope I got this right - quite a long way to do some simple programming using just JSON. Are there other shorter, better and/or easier ways to this same thing in Alumio?

PS The goal for this this is to go through product variant properties inside a product sync. route using transformers. The properties and options themselves will already have been synced first.

2

Hi @kjetil

Thank you for your thorough explanation of your case. You are right that you can do the approach you mentioned. We will also introduce the way to do it using Group Records and Operator Transformer.

Since we depend on the propId properties to define the method by comparing them between the existingOptions and newOptions arrays, we can group the objects of each array and set the propId as the key of the grouped objects.

image
image880×269 10.3 KB

We can do the same with the newOptions array.

This way, we will have propId-grouped objects of existingOptions and newOptions, such as below.

  "existingOptions": {
    "1": [
      {
        "propId": "1",
        "optionId": "2"
      }
    ],
    "2": [
      {
        "propId": "2",
        "optionId": "34"
      }
    ]
  },
  "newOptions": {
    "1": [
      {
        "propId": "1",
        "optionId": "2"
      }
    ],
    "3": [
      {
        "propId": "3",
        "optionId": "5"
      }
    ]
  }
}

Next, we will use Operator Transformer to map which objects need to be created, updated, or deleted.

In order to get the objects we need to create (the POST method), we should use “Array Diff Keys” operator, set the newOptions as the left path and existingOptions as the right path. We can set the Destination to a property, such as call.POST. Please see the below table.

| Method  | Operator             | Left Path       | Right Path      | Destination |
|---------|----------------------|-----------------|-----------------|-------------|
| POST    | Array Diff Keys      | newOptions      | existingOptions | call.POST   |
| PUT     | Array Intersect Keys | newOptions      | existingOptions | call.PUT    |
| DELETE  | Array Diff Keys      | existingOptions | newOptions      | call.DELETE |

We will have the below data as the result of the above Operator Transformers.

{
  "call": {
    "POST": {
      "3": [
        {
          "propId": "3",
          "optionId": "5"
        }
      ]
    },
    "PUT": {
      "1": [
        {
          "propId": "1",
          "optionId": "2"
        }
      ]
    },
    "DELETE": {
      "2": [
        {
          "propId": "2",
          "optionId": "34"
        }
      ]
    }
  }
}

Then, we should map the method, which is a key at this moment, into a property using Move data between accessors. The Source accessor should be “Key accessor” and the Root is “call”. The Destination accessor should be “Structure accessor” with “method” as the Key, “properties” as the Value, and “call” the “Root”. The result of the transformer will be such below.

{
  "call": [
    {
      "method": "POST",
      "properties": {
        "3": [
          {
            "propId": "3",
            "optionId": "5"
          }
        ]
      }
    },
    {
      "method": "PUT",
      "properties": {
        "1": [
          {
            "propId": "1",
            "optionId": "2"
          }
        ]
      }
    },
    {
      "method": "DELETE",
      "properties": {
        "2": [
          {
            "propId": "2",
            "optionId": "34"
          }
        ]
      }
    }
  ]
}

We can then copy the id property into each object in pattern call.*.properties.*.* using the Recursively copy values to children.

In order to copy the method, we should use a Node, transform nodes entity transformer using the pattern call.*. Inside the node, we can use Recursively copy values to children to copy the method property into each object using pattern properties.*.*. The entity will be such below.

{
  "call": [
    {
      "method": "POST",
      "properties": {
        "3": [
          {
            "propId": "3",
            "optionId": "5",
            "id": "1",
            "method": "POST"
          }
        ]
      }
    },
    {
      "method": "PUT",
      "properties": {
        "1": [
          {
            "propId": "1",
            "optionId": "2",
            "id": "1",
            "method": "PUT"
          }
        ]
      }
    },
    {
      "method": "DELETE",
      "properties": {
        "2": [
          {
            "propId": "2",
            "optionId": "34",
            "id": "1",
            "method": "DELETE"
          }
        ]
      }
    }
  ]
}

Finally, we can use a new Node, transform nodes entity transformer using the pattern call.*.properties.*.* and submit the HTTP request using HTTP transformer there.

I hope my explanation is clear to you. It may be not so short solution, but this is an alternative way to achieve your objective. Feel free to let me know if you have any questions.

3

I get this array to work in my entity transformer. But how do you advice to do in the outgoing config? All the “work” (requests) are sent and done within the transformer, so there is eventually no requests left to do in the outgoing config itself. I guess one option would be to put some kind of dummy request there or maybe put the transformer on the incoming config, and just not do any outgoing for this route. Or do you have a solution to this kind of scenario?

My [simplified] schedule would be to
route 1: add the products - incoming from system A outgoing to B.
route 2: add product properties (from array structure) - this can not be done before 1. - i.e. when they are created the product ID would be missing - incoming from storage (saved to storage in route 1).

4

You can set up the publisher to do nothing using the “No action publisher”. Please let me know if this is not the case.