A Quick Talk On Generic Cloud App Integration With Informatica Cloud

The Informatica Cloud Integration platform is the industry's complete cloud integration platform as a service (iPaaS). Informatica has been continuously playing as the leader role in this space for the 4th consecutive year (Gartner's margic quadrant for enterprise integration platform). It's definitely your top choice for data integration management.

With the emerging SaaS application on the market, it's essential for integration platform to provide a flexible and efficient connector to integrate various cloud applications in a generic manner. Indeed there are many specific connectors built for individual applications by Informatica (Tier 1 and Tier 2 connectors) or vendors themselves (OEM connectors). But let's face it: most cloud applications evolves quickly on new features and their APIs get updated quicker than you realize. How quickly the connectors can turn around with the most recent API updates? The answer is definitely the connector update lags behind, as most connectors get updated once a year or longer than that.

For what's worth, Informatica introduced the Rest V2 connector in fall of 2016. The V2 connector is an successor of its ancestor which is just named as "Rest Connector". It allows to connect with the latest Rest API of the cloud application directly to read or write data. You can fully leverage it no matter the cloud application is a source, a target or a lookup system. It offers great improvements on flexibility, efficiency, ease of use besides performance. Considering the majority use of Rest API as the industry standard for emerging cloud services, this Rest V2 connector can fill in the role of a generic connector for cloud applications with Rest API. In this article, I will reveal the great features and caveats followed by uncovering some hidden gems of the Rest V2 connector that many may not have realized yet.

One of the great features of the Rest V2 is the introduction of hierarchy data support. As we know, the Rest API deals with either XML or JSON data which are hierarchy data structure. Before the appearance of Rest V2, transformation of hierarchy and relational data is cumbersome and even requiring writing your own script to transform the data as part of the integration process. At the Informatica fall release of 2016, the same time Rest V2 is introduced, the Hierarchy Builder/Parser is brought to life to fill the gap. Hierarchy builder is to build hierarchy data from relational data and hierarchy parse does the opposite way. Let's take a look at how hierarchy builder/parser works.

First, a hierarchy schema needs to be created. It is used by hierarchy builder/parser to present the data structure. To create a hierarchy schema, one will need to have a XML schema file (.xsd) or a JSON sample file (.json). Note the XML schema file is not necessarily provided by the vendor and one has to create it based on the XML payload.

After the creation of hierarchy schema, a hierarchy builder/parser can be used in a data mapping to map hierarchy data to relational data or vice versa.

That seems quite easy, compared to writing your own script to transform. But let's look at the Rest V2 connector as it's even better.

The core of Rest V2 connector configuration is the Swagger support. I have to thumb up on Informatica on the betting of Swagger because of its popularity and great potential as industry standard for API framework. Not all vendor provides swagger definition for their API. But no worry, Informatica provides this tool to you right inside your Informatica cloud web console. It's under Configure -> Swagger files. If you don't see this option, please contact with your Informatica support to enable it. You will need to have Rest V2 connector license to use it.

While creating the Swagger file, you basically defines the API details - URL, verb (POST/GET etc), authentication (username/password, token, etc), Headers, Query parameters, Body and sample response file. A couple of caveats in this process are:

• The JSON response file is required even you are dealing with XML response. In this case, you will need to convert the XML response in JSON format. There are many online tools to do that.
• If your raw body is XML format and you have issues creating the Swagger, after making sure nothing is wrong on your end, you may want to contact Informatica support requesting them to create the Swagger for you. Informatica support does have their own internal tool to create swagger file. Informatica will improve the Swagger tool based on my conversation with them and most likely it's no longer an issue soon.

After the Swagger file, create the Rest V2 connection which is quite easy as majority of the info needed for the API is filled in the Swagger file.

Now the Rest V2 Connector is ready to use in the data mapping. The Rest V2 can be used a source to read data or the target to write data in this case. Of course the API you put in the initial Swagger file has to comply with read or write operation. The hierarchy data mapping can be achieved in the Rest V2 Connector as a source/target itself. No need to use hierarchy builder or parser.

To save the best to the last, the great power of using Rest V2 connector is to use it as a midstream. Let's work on a use case: say I need to pull data from A system and push to B system via Rest API. If I were using Rest V2 as the target for B system. At the end of the flow, how do I know the data is successfully taken in B? I will need to start another pipeline or another data mapping task to examine the log file and take action from here. When using it as a midstream, the response of the Rest call can be transmitted to the next step for follow up action in the same streamline flow. This is a differentiator and made me believe all Rest V2 connector should not serve as target but midstream instead.

Another hidden gem on the Rest V2 connector as a midstream, is the capability of processing multiple Rest calls. This is definitely a great sweet feature. For any Rest call, one payload corresponds to one request and generates one response. If you have multiple payloads to be processed in one integration, essentially you need a loop function to loop through individual records and make the Rest call for each record. Without Rest V2 connector, this is impossible as there is no such loop function in Informatica data integration. The alternative would be to use Informatica Cloud Real Time (ICRT) which provides a workflow type of designer that a task can be jumped from one step to another based on certain condition - basically to achieve loop function. With Rest V2 connector, there is no additional configurations needed and it can process multiple records in one step - taking multiple records as inputs, make Rest calls for each record, and generate all response as outputs. While using it as a midstream, we can easily follow up on actions with the Rest call outputs in the same streamline flow.