TABLE OF CONTENTS
- Introduction
- Entity Transformers
- Chain Multiple Entity Transformers
- Filter Previously Stored Entities
- Getting Branches From A Pattern
- Data Transformers
- Delete Keys From Storage
- Get All From Storage
- Conditional Transformer
- Copy Using A Pattern
- Get Entity From Storage
- Get From Storage
- Group Records
- HTTP Transformer
- Key Filter
- Key Mapper
- Merger Transformer
- Move Using A Pattern
- Operator Transformer
- Recursively Copy Values To Children
- Value Filter, Value Mapper, and Value Setter
- Write Content To Storage
- Write To Storage
- Wrapping Up!
Introduction
A transformer can be defined as a miniature reusable logic unit that is applied on certain data. The transformer’s input can be derived from various data sources. If the data is suitable, the transformer logic will be applied to it. Multiple transformers can be linked together and the sequence of the distinct transformers determines the output. They can be added to various stages as well, such as incoming/outgoing configurations and even data routes. Alumio comes packed with industry-leading transformers and here is a comprehensive guide for you to use them!
Entity Transformers
Entity transformers are a crucial piece of data transformation jigsaw that paves the way for easy data manipulation to suit the best use cases. Alumio offers the best-in-class entity transformers that are aimed at solving sequence to sequence tasks while managing long-range dependencies. From chain multiple entity transformers to filtering previously stored entities to obtaining branches from a pattern, these transformers can accomplish a variety of tasks.
Let’s take a closer look!
More often than not, you might require to manipulate a data transformer to suit a certain objective. Nevertheless, you might still require a prototype transformer on top of that. In such a scenario, the best course of action is to add a chain of multiple entity transformers instead of adding a data transformer right away.
While creating an entity transformer, you can select Chain Multiple Entity Transformers from the Settings. Upon selection, you will have the option to add data transformers (for example, transforming data using mappers and conditions) while following it up by adding another entity transformer (for example, getting branches from a pattern).
This approach will allow you to incorporate data manipulations on the data transformers. For instance, you can include a value setter, set up its configuration using a result key (objects, numbers, arrays).
The subsequent entity transformer can simply include the result key and filter out irrelevant data that are not required.
In many cases, you might scramble for ways to boost the performance of an integration while designing it. For instance, let’s consider a case of a fileserver (FTP) having an XML file that contains thousands of products. Given its nature, detecting changes can be a challenging task as the XML file has no query functionality. Well, Alumio can sail you through this challenge by allowing you to utilize storage and building a delta filtering functionality, thank you to a fine-tuned Filter Previously Stored Entities transformer.
This transformer bestows on you the capability to filter the already-processed entities using a dedicated storage called Variant Cache. As soon as you implement the transformer and run a test, you will see that the storage has stored the cache and it will be displayed.
In case you change the cache and run it again, the storage will instantaneously capture the change and update it.
Thus, the entire process of storing a cache system within the Alumio integration becomes a cakewalk.
It is a common practice for the users to opt for multiple individual processings over batch-processing since the former offers better maintenance and governance over the integrations. While it sounds cool, accomplishing it on-the-ground is not that easy since a single interaction with a foreign system can result in the retrieval of multiple data entities.
Alumio comes armed with one of the finest transformers, namely Get Branches From A Pattern, which makes it pretty easy to disintegrate a single API into numerous smaller tasks.
You can select a pattern and the transformer will pick up all of the objects within the pattern on-the-go and split all of the tests.
Data Transformers
Alumio comes packed with various kinds of transformers. Users can perform a variety of actions such as deleting keys from storage, copying using a pattern, getting entities from storage, writing to storage, and a plethora of other tasks.
In scenarios where you want to eliminate a certain entity/entities from a storage, the Delete Keys From Storage transformer comes to the rescue.
Here’s how you can utilize the transformer.
Step 1: Click on Add Data Transformer and select the Delete Keys From Storage transformer.
Step 2: You simply need to select the storage next.
Step 3: Select the Pattern you want to eliminate.
Step 4: The key (entity you want to delete) will be deleted from the storage by the transformer. You can check this by clicking on the Run Test button.
This transformer comes in handy in scenarios where you have to retrieve each and every entity from a storage.
Here’s how you can implement the transformer.
Step 1: Click on Add Data Transformer and select the Get All From Storage transformer.
Step 2: Select the Storage from where you wish to pull data.
Step 3: Select the DestinationPath (entities from where you wish to add data). It will retrieve the entity keys which contain values inside the storage.
When you are working with data that demand conditional based interaction, Conditional Transformer is the best option. They are derived from elementary if, else, and then statements commonly used in programming. It is a widely utilized data transformer where you can set up a myriad of conditional transformations.
Here is the step-by-step process to use the Conditional Transformer.
Step 1: Click on Add Data Transformer and select the Conditional Transformer.
Step 2: Next, you can choose from various condition options present.
Step 3: Select the Accessor.
Step 4: Choose the Key and follow it up by adding customized conditions.
Step 5: If you click on the Run Test button, you will see that the transformer has implemented the conditions on the data.
This transformer is predominantly used when there is a requirement of copying objects to another key.
Here’s how you can implement the transformer.
Step 1: Click on Add Data Transformer and select the Copy Using A Pattern transformer.
Step 2: You have to select the Pattern (origin key) first.
Step 3: Next, you have to select the Replacement (copy destination). The best part is that you can copy multiple objects while implementing this transformer.
All you need to do is utilize the * notation after the variable in the field of Pattern and $1 notation after the copy destination in the Replacement field.
Step 4: If you click on the Run Test button, you will see that the transformer has copied the objects to the destination key (provided in the Replacement field).
Its function is to improve data quality by enriching the mainframe data objects from a wide range of sources. Now, how can you implement the transformer?
Here’s how you can implement the transformer.
Step 1: Click on Add Data Transformer and select the Get Entity From Storage transformer.
Step 3: Select the Storage (or storages) from where you wish to import the entity.
Step 4: Select the iDPath.
Step 5: The transformer writes data (entities) from multiple sources to storage and integrates them when the final report is made.
Step 6: If you click on the Run Test button, you will see that reading of data from any storage utilizing an Identifier.
This transformer pulls large objects from storage containing one or multiple fields.
Here’s how you can implement the transformer.
Step 1: Click on Add Data Transformer and select the Get From Storage transformer.
Step 2: Next, you have to select the Storage and StoragePath.
Step 3: Subsequently, you have to specify the DestinationPath.
Step 4: If you click on the Run Test button, you will see that the required data has been transferred from the selected storage to the destination.
There will be many scenarios where you have to group objects by an Identifier (for example, grouping products by their SKUs) and the Group Records transformer is useful in such scenarios.
Here’s how you can implement the transformer.
Step 1: Click on Add Data Transformer and select the Group Records transformer.
Step 2: Next, select the Pattern and Path.
Step 3: Specify the Destination where the grouped data will be reflected.
Step 4: On clicking the Run Test button, you will see that the transformer will waste no time in grouping records based on the input provided.
The function of an HTTP Transformer is to interact with an API end-point with the implementation of an entity transformer. This can be used to collect data from an endpoint based on previously obtained data.
Here’s how to do it.
Step 1: Click on Add Data Transformer and select the Key Filter transformer.
Step 2: Fill the Request URL section.
Step 3: Fill the Request Method.
Step 4: Select the HTTP Client.
Step 5: On clicking the Run Test button, you will see that the transformer has pulled the specified data.
Note: While utilizing an HTTP transformer, the data retrieved from the HTTP transformer becomes the new reality. In effect, it means that the previous data is no longer reflected in the Entity Data.
(To prevent this from happening a Merger Transformer can be implemented to merge data together.)
What happens many times is that there is a heap of information, of which, a majority portion is not required. The Key Filter transformer plays a key role in eliminating the chaos by filtering out the irrelevant keys.
Here’s how you can implement the transformer.
Step 1: Click on Add Data Transformer and select the Key Filter transformer.
Step 2: Select Key Accessor under Accessor.
Step 3: Type in the irrelevant keys that are not required in the Keys field.
Voila! The mentioned keys will be removed (or filtered) from the object.
The best part is that you can utilize the transformer to eliminate keys using conditions or no conditions at all.
Here is the step-by-step guide to do it.
Step 1: Click on Add Data Transformer and select the Key Filter transformer.
Step 2: Select Key Accessor under Accessor.
Step 3: Instead of selecting keys from Add Keys, you have to click on Add Conditions and select a condition. For example, you can select Value one of list.
Step 4: Next, you have to click on Add Value and specify the value. Let’s suppose you add the value String and specify its value.
Step 5: On clicking the Run Test button, you will see that the transformer has removed the string from the object
Let’s imagine a scenario where data coming from a system (let’s say A) is not consistent with another system (let’s say B). The Key Mapper Transformer facilitates smooth compatibility by making the keys of both the systems consistent.
Here’s how you can implement the transformer.
Step 1: Click on Add Data Transformer and select Key Mapper transformer.
Step 2: Select Key Accessor under Accessor.
Step 3: Click on Add Mappers under Mappers and select Dictionary Map.
Step 4: Click on Add Map and then fill up the fields you want to map for a certain data type. You can add multiple maps depending on your requirement.
Step 4: The keys will be mapped and updated by the transformer. You can visualize the same by clicking on the Run Test button.
Merger transformer helps in combining data pulled from a source with the current data ( both being wrapped within an entity data).
For this example, let’s use a merger transformer in combination with a HTTP transformer. You can refer to what we had discussed under HTTP Transformer (refer point 8).
Here’s how you can add a merger transformer.
Step 1: You have to temporarily remove the HTTP transformer first.
Step 2: Select Merger Transformer under Data Transformers.
Step 3: Now select a HTTP Transformer under it, and follow the same steps mentioned in point 8 (above).
Step 4: Fill in the field of Template. If you want to merge both data under an object with a new key, mention the key. For example, “xxxx” : “&{@}”.
Step 5: On clicking the Run Test button, you can visualize that the previous data has been recorded. It’s followed by the new key xxxx containing the object obtained from the merger transformer.
In many cases, you will require moving or mapping objects to a different key. For instance, consider moving a variable called name to new_name.
Here’s how to do it.
Step 1: Click on Add Data Transformer and select Move Using A Pattern transformer.
Step 2: Fill in the fields of Pattern and Replacement.
Step 3: On clicking the Run Test button, you will see that the data has been replaced accordingly.
Note: While utilizing the Move Using A Pattern Transformer, the * notation is applied to move multiple objects. At the same time, $1 is used in the Replacement field for such cases.
The Operator Transformer, as the name suggests, allows you to conduct multiple operations on two datasets. There are mathematical operators, bitwise operators, array merging/comparing, and much more.
Here’s how you can implement the transformer.
Step 1: Click on Add Data Transformer and select Operator transformer.
Step 2: Select the Operator. For example, you can select Addition, Array Merge, etc.
Step 3: There are two fields named Data Container Right Path and Data Container Left Path. Enter the two data sets on which you want to run the operation as per your requirement.
Step 4: Next, you have to fill the field of the Data Container Destination Path. This is the new key that will reflect the operated data.
Step 5: The transformer will run the desired operation and store the operated data in the new key. You can visualize the same by clicking on the Run Test button.
It is a common practice in eCommerce to have numerous variations of a product. For example, a product can be sold in many colors and different sizes. However, there can be a parent data that is consistent among all the variations. This transformer allows you to copy such parent data to all the product variations seamlessly.
Here’s how you can implement the transformer.
Step 1: Click on Add Data Transformer and select Recursively Copy Values To Children transformer.
Step 2: Fill the field of Pattern to children within a parent by specifying the variants (or children).
Step 3: Fill the field of Mapping of values from parent to child by specifying the parent data (placeholders) that will be copied to its variants (or children)
Step 4: Upon clicking the Run Test button, you will observe that the transformer has copied the required data (mentioned in Step 3) to the variants.
These 3 are value-based transformers that are very useful.
The Value Filter transformer can be utilized to filter values based on a specified condition. For example, it can remove a value when it returns a certain value or when it is empty.
Here’s how you can implement the transformer.
Step 1: Click on Add Data Transformer and select Value Filter transformer.
Step 2: Select Pattern Accessor under Accessor.
Step 3: Specify the Pattern.
Step 4: Select the Conditions that will trigger the transformer.
Step 5: Upon clicking the Run Test button, you will observe that the transformer has filtered out the conditions you had mentioned.
Alumio offers the best-in-class integrations and it demands seamless movement of data with cross-system compatibility. The Value Mapper transformer can transform a data type into another (such as a string to an integer) and value manipulations a cakewalk.
Here’s how you can implement the transformer.
Step 1: Click on Add Data Transformer and select Value Mapper transformer.
Step 2: Select the Accessor. For example, Key Accessor.
Step 3: Specify the Keys you wish to map and transform.
Step 4: Next, select a Mapper (the data type you want the key to be transformed into). For example, you can select Cast: Float.
Step 5: Click on the Run Test button. You will observe that the transformer has transformed the data (into float in this case).
The Value Setter transformer can quickly prototype a data object, execute mathematical calculations of numerous fields at one go, and even integrate one/multiple fields.
Here, we have explored how you can combine two strings (or fields0 together.
Step 1: Click on Add Data Transformer and select Value Setter transformer.
Step 2: Set up the Value Setter configuration next.
Step 3: Specify the Key (where the combined strings will be saved, let’s say X)
Step 4: Next, select the Value (Strong in this case). Place two value holders with the strings that you want to combine. Let’s consider two strings, String A and String B. In this case, you have to input: &{A} &{B}
Step 5: Upon clicking the Run Test button, you will observe that the transformer has saved the combined value of A & B in X.
In some cases, you might require mapping values across several systems. The Write Content To Storage Transformer does exactly that and this is how you can use it.
Step 1: Click on Add Data Transformer and select Write Content To Storage transformer.
Step 2: Select Key Accessor under Accessor.
Step 3: Specify the key/keys in the Keys field.
Step 4: Select a storage in the Storage field.
Step 5: Specify the fields of Storage Path and Source Path.
Step 6: Upon clicking the Run Test button, you will observe that the transformer has saved the key (in step 3) having all the data (as specified in step 5).
Note: The basic difference between the Write Content To Storage transformer and Write To Storage transformer is that you can specify a key in the former where the data goes to, or is retrieved from.
The storage can be utilized for many purposes which demands data to be saved in the current state so that it can be re-utilized later. The use cases can be saving mutation dates, saving objects, pagination, and much more.
Here’s how you can implement the transformer.
Step 1: Click on Add Data Transformer and select Write To Storage transformer.
Step 2: Select the Storage (where you want the data to be saved)
Step 3: Specify the fields of Storage Path (data key) and Source Path (data destination).
Step 4: Upon clicking the Run Test button, you will observe that the transformer has written the desired data to the storage destination.
Wrapping Up!
Alumio iPaaS is the best integration platform with an extensive range of features that provides you the ability to create, sustain, and manage integrations effortlessly. The in-built transformers of Alumio play a critical role to facilitate such integrations and deliver a smooth experience. This article-cum-guide has covered the entirety of Alumio transformers that should allow you to slice and dice your data with ease and grace!