As an Alumio integrator, I could really benefit from a sample data function that (temporarily) saves real-time data into transformer tests as it passes through transformers. This gives a good overview of what real-world data passes through each transformer of a route when enabled, and could benefit troubleshooting.
This should probably be a toggle for a single passing stream of data / entity at which point it disables again to minimize the impact on performance. The result would be automatically created tests of the last processed data in each transformer it passed through.
Iām afraid that I didnāt understand what you needed. Could you please explain in more detail? So far, what we understand is that you wanted temporary data for each transformer result, is that correct?
I do understand: by logging the last data as it went through the transformer (and possibly failed) this would allow for far easier debugging and keeping tests up to date with actual sample dataā¦
@t.vanderveen maybe the datapoints are what you need?
Or are you unable to get what you need with that?
On second thought, do you mean a snapshot of the already existing datapoints and results when a task gets failed?
Maybe when a ādebugā toggle has been enabled in the route for example.
Hi @r.candrian@Alumio_Bot , thank you for your reply. I understand the principles of the datapoints and transformer testers, itās more a way to attach real-world data to them. I am under the impression that datapoints can be used to test each transformer based on a transformer test input data, and not actual data flowing through the route, right? Or are you telling me that primed datapoints also use the actual dataflow and not only the input data from the transformertests?
Maybe a scenario story would clarify my intended use case more;
If a client calls me that there is a data problem on an existing configuration when pushing a specific product for example, we could debug these occurrences far easier when we could toggle that real data is being collected as transformertest input over a limited timeperiod. That would enable us to push that product manually en really check what happened in each passed transformer. There are a lot of use cases where this could be helpful in my opinion. I think the ādebugā toggle you mentioned is along the my train of thought in this case.
Please let me know if you need me to elaborate some more!
@Alumio_support
Could you please create a feature request that contains the following?
As Alumio user Iād like to have a toggle on the route called āDebug modeā.
When enabled, all data points are activated, whenever a task fails;
Iād like to be able to see the actual data on all available datapoints for that tasks like a snapshot.
To browse the transformations and to debug.
When āDebug modeā is deactivated the snapshots can be deleted.
Optional, āDebug modeā has a dropdown with amount of failed tasks to keep snapshots of (1,3,10).