Feature request: entity transformers after outgoing publishers in configs

iPaaS General Feature Requests
1

Dear Alumio team,

In the past few integrations, we hardly used the routing/incoming→outgoing path system in Alumio. It is purely due to the fact, that in almost all cases, we had to perform actions based on the response of the outgoing entity’s publisher - thus we could only use incoming configurations with merger&http transformers for these occasions (which somewhat defeats the purpose of the outgoing configurations, since we use incoming for that as well).

It would be nice to have the option to run entity transformers AFTER the publisher of an outgoing configuration too - not only before.

An example:

Lets say, that I have products coming in from System A. I’m getting them using an incoming config (subscriber is irrelevant), save then to a storage for later use, and I also want them to integrate into System B. Based on whether this product exists in System B or not, I would send different HTTP requests (lets say POST vs PATCH, url would change too).

The goal, is to somehow retrieve whether the product exists in System B or not WITHOUT causing unnecessary requests to System B.

By using outgoing configs and routes, we have the following possibilities:

The best approach would be to use the product storage, in which a product has either a saved ID from System B, or not. If it does, then we only PATCH, if it doesn’t, we POST. For this approach, we need to save the ID contained by the response of the outgoing publisher into the product storage - Which we cannot do now as far is I know

The above method would be convinient.

The worse approach is to query down the product from System B with entity transformers between incoming and outgoing config of the route - and decide what to do based on the response. This method - while working flawlessly - creates an unnecessary overhead to System B, thus making the sync less efficient (especially when we are talking about systems with 100k+ entities)

Since the first approach is not feasible, and the second creates unnecessary requests, we have to default back to using an incoming config for this job as well - thus getting stripped from features like tasks, easier parallel processing using routes etc.

I hope my request is clear, in case of any questions, feel free to comment!

Best Regards,

Gergő Borsodi - Backend Developer
7digits

2

Hi @gerg.borsodi

Thank you for letting us know your concern.

We have a publisher called “No action publisher”, which allows you to do nothing in the publisher part of an outgoing configuration, as Publisher is a must-fill part of an outgoing configuration. With “No action publisher” set, you can freely do the PATCH/POST call to System B in the Entity transformers part of the outgoing configuration, and put any transformers after the HTTP call as many as you need.

image
image1070×503 19.5 KB

Please let us know if this works for your case.

3

I think you misunderstood me. I would need to use a real publisher (like http publisher), but I would also need to be able to run transformers AFTER the publisher has finished - and transform the data it returned.

4

@gerg.borsodi Could you please let us know why you would need a real publisher? Is there any functionality in the publisher that is not replaceable by any existing transformers?

5

Yeah sure. My goal to achieve is the following:

I would need to run transformers after an outgoing config’s publisher has finished.

For example, I have an outgoing configuration with an HTTP Publisher. This uploads some inflowing data to a remote HTTP server, using a POST request. With the current workflow, after the publisher has sent the HTTP POST request to the remote server, the task is finished/terminated (based on the response status of the publisher).

The problem is, that if the publisher finishes successfully, then I would need to save for example the ID from the publisher’s response to a storage, associating the uploaded data with the remote ID. With the current Alumio workflow, this isn’t possible using Outgoing Configurations as far is I know, since I cannot run an additional transformer AFTER the publisher finished to handle the publisher’s response data. The only way is to leave the outgoing configs out of the equasion, and use incoming configs solely, with HTTP transformers serving as HTTP publishers.

Ot another problem: If the publisher fails for some reason, and the payload could be changed so that the publisher would successfully run; then my only option is to add redundancy using solely incoming configs again. WIth transformers after the publishers, I could add error-handling, and redundant HTTP calls, so that a task could recover from the error.

Summarized, my feature request is to be able to run additional transformers inside an outgoing config AFTER the publisher, so that the executing order is as follows:

Current: [data] ==> Outgoing Config pre-publisher transformers ==> publisher ==> end task

Desired: [data] ==> Outgoing Config pre-publisher transformers ==> publisher ==> post-publisher transformers ==> task end

The solution you suggested also works, but I think it kinda defeats the purpose of publishers. :smiley:

Feel free to ask if my proposition is not clear.

6

We understand your concern about the suggested solution, which you see as “defeating the purpose of publishers”. That’s a fair point, and it’s exactly why we introduced the “no action publisher” option earlier. It’s meant to give you the flexibility to move the HTTP call into the transformer chain when the publisher step becomes too restrictive for use cases like yours.

This way, you still keep the conceptual separation, but you’re not blocked when you need post-request transformations (e.g., saving IDs, processing the response, etc).

Please let us know if you have any further questions.

1 Like
7

Thank you for your answer, and sorry for my pickiness - I will def. use this method more often.

8

No worries at all. We believe your feedback is valuable, and it helps us understand where the product can feel limiting in real scenarios.

We’re glad the approach works for you, and if you run into any other edge cases or ideas, feel free to share them anytime. We appreciate the constructive discussion!