Process incoming response data directly

Hi, is there a way to process (convert) response data in an incoming config before it is split up into entities by Alumio?

Hi @ben,

Could you clarify what kind of process you are referring to? Are you talking about mapping, or something else?

Hi Rian, of course.

I am using the Snowflake Subscriber to get data from Snowflake. In this case customers.

Snowflake returns the data like this:

{
    "resultSetMetaData": {

        "rowType": [
            {
                "name": "ID",
            },
            {
                "name": "EMAIL",
            },
            {
                "name": "FIRSTNAME",
            },
            {
                "name": "LASTNAME",
            }
        ]
    },
    "data": [
        [
            "123",
            "[email protected]",
            "John",
            "Doe"
        ],
        [
            "124",
            "[email protected]",
            "Mary",
            "Doe"
        ],
        [
            "125",
            "[email protected]",
            "Peter",
            "Johansen"
        ],

    ]
}

So, the keys are in the “rowType” array, and the values in the “data” array.

I want Alumio to process the customers one per task obviously.

So basically I want to combine the keys and values to a customers array first:

{
    "customers": [
            {
                "ID": "123",
                "EMAIL": "[email protected]",
                "FIRSTNAME": "John",
                "LASTNAME": "Doe"
            },
            {
                "ID": "124",
                "EMAIL": "[email protected]",
                "FIRSTNAME": "Mary",
                "LASTNAME": "Doe"
            },
            {
                "ID": "125",
                "EMAIL": "[email protected]",
                "FIRSTNAME": "Peter",
                "LASTNAME": "Johansen"
            }
    ]
}

so I can simply enter “customers” in the Pattern to items field:

Basically I need a “transform response before it is spit up into instances” transformer. Maybe Alumio has one already somewhere, but I can’t find it.

Hi @ben , you can achieve that by using the Code Transformer like this:

How/where exactly do I apply this transformer?

This is the flow:

  1. The incoming config does a request.
  2. It gets a response.
  3. That response is transformed (using your code for example) to an array of customers.
  4. I have put “customers” in the Pattern to items field.
  5. Alumio processes each customer in a task.

How do I do step 3?

Hi, you can create a new transformer for the code transformer I provided.

Then put that transformer in the incoming. If you can share the link to your configuration via DM or a support ticket, we can create an example config for the incoming.

Rian and Ariel did a really great job helping me out here, so just to follow up on this:

The solution in this specific case is to use two Entity Transformers on the Incoming Config.

The first to transform the response body into a pattern that the second can use to “Branche from a pattern”.

Since these transformers work on the response body and not on the Entity this seems a bit counterintuitive, but it works.

Bear in mind that using a Code Transformer in the first might lead to memory issues, since the whole response will be passed to it (unless you pre-process it first).

For any other purposes, the original (general) question is still valid I guess: how to transform the response body before anything is done by the Entity Transformers. Alumio will have a look at it.

Hi @ben ,

Thank you for your valuable feedback!