Friday, August 4, 2017

A Quick Talk On Generic Cloud App Integration With Informatica Cloud

The Informatica Cloud Integration platform is the industry's complete cloud integration platform as a service (iPaaS). Informatica has been continuously playing as the leader role in this space for the 4th consecutive year (Gartner's margic quadrant for enterprise integration platform). It's definitely your top choice for data integration management.

With the emerging SaaS application on the market, it's essential for integration platform to provide a flexible and efficient connector to integrate various cloud applications in a generic manner. Indeed there are many specific connectors built for individual applications by Informatica (Tier 1 and Tier 2 connectors) or vendors themselves (OEM connectors). But let's face it: most cloud applications evolves quickly on new features and their APIs get updated quicker than you realize. How quickly the connectors can turn around with the most recent API updates? The answer is definitely the connector update lags behind, as most connectors get updated once a year or longer than that.

For what's worth, Informatica introduced the Rest V2 connector in fall of 2016. The V2 connector is an successor of its ancestor which is just named as "Rest Connector". It allows to connect with the latest Rest API of the cloud application directly to read or write data. You can fully leverage it no matter the cloud application is a source, a target or a lookup system. It offers great improvements on flexibility, efficiency, ease of use besides performance. Considering the majority use of Rest API as the industry standard for emerging cloud services, this Rest V2 connector can fill in the role of a generic connector for cloud applications with Rest API. In this article, I will reveal the great features and caveats followed by uncovering some hidden gems of the Rest V2 connector that many may not have realized yet.

One of the great features of the Rest V2 is the introduction of hierarchy data support. As we know, the Rest API deals with either XML or JSON data which are hierarchy data structure. Before the appearance of Rest V2, transformation of hierarchy and relational data is cumbersome and even requiring writing your own script to transform the data as part of the integration process. At the Informatica fall release of 2016, the same time Rest V2 is introduced, the Hierarchy Builder/Parser is brought to life to fill the gap. Hierarchy builder is to build hierarchy data from relational data and hierarchy parse does the opposite way. Let's take a look at how hierarchy builder/parser works.

First, a hierarchy schema needs to be created. It is used by hierarchy builder/parser to present the data structure. To create a hierarchy schema, one will need to have a XML schema file (.xsd) or a JSON sample file (.json). Note the XML schema file is not necessarily provided by the vendor and one has to create it based on the XML payload.


After the creation of hierarchy schema, a hierarchy builder/parser can be used in a data mapping to map hierarchy data to relational data or vice versa.



That seems quite easy, compared to writing your own script to transform. But let's look at the Rest V2 connector as it's even better.

The core of Rest V2 connector configuration is the Swagger support. I have to thumb up on Informatica on the betting of Swagger because of its popularity and great potential as industry standard for API framework. Not all vendor provides swagger definition for their API. But no worry, Informatica provides this tool to you right inside your Informatica cloud web console. It's under Configure -> Swagger files. If you don't see this option, please contact with your Informatica support to enable it. You will need to have Rest V2 connector license to use it.

While creating the Swagger file, you basically defines the API details - URL, verb (POST/GET etc), authentication (username/password, token, etc), Headers, Query parameters, Body and sample response file. A couple of caveats in this process are:

  • The JSON response file is required even you are dealing with XML response. In this case, you will need to convert the XML response in JSON format. There are many online tools to do that.
  • If your raw body is XML format and you have issues creating the Swagger, after making sure nothing is wrong on your end, you may want to contact Informatica support requesting them to create the Swagger for you. Informatica support does have their own internal tool to create swagger file. Informatica will improve the Swagger tool based on my conversation with them and most likely it's no longer an issue soon.



After the Swagger file, create the Rest V2 connection which is quite easy as majority of the info needed for the API is filled in the Swagger file.


Now the Rest V2 Connector is ready to use in the data mapping. The Rest V2 can be used a source to read data or the target to write data in this case. Of course the API you put in the initial Swagger file has to comply with read or write operation. The hierarchy data mapping can be achieved in the Rest V2 Connector as a source/target itself. No need to use hierarchy builder or parser.

To save the best to the last, the great power of using Rest V2 connector is to use it as a midstream. Let's work on a use case: say I need to pull data from A system and push to B system via Rest API. If I were using Rest V2 as the target for B system. At the end of the flow, how do I know the data is successfully taken in B? I will need to start another pipeline or another data mapping task to examine the log file and take action from here. When using it as a midstream, the response of the Rest call can be transmitted to the next step for follow up action in the same streamline flow. This is a differentiator and made me believe all Rest V2 connector should not serve as target but midstream instead.

Another hidden gem on the Rest V2 connector as a midstream, is the capability of processing multiple Rest calls. This is definitely a great sweet feature. For any Rest call, one payload corresponds to one request and generates one response. If you have multiple payloads to be processed in one integration, essentially you need a loop function to loop through individual records and make the Rest call for each record. Without Rest V2 connector, this is impossible as there is no such loop function in Informatica data integration. The alternative would be to use Informatica Cloud Real Time (ICRT) which provides a workflow type of designer that a task can be jumped from one step to another based on certain condition - basically to achieve loop function. With Rest V2 connector, there is no additional configurations needed and it can process multiple records in one step - taking multiple records as inputs, make Rest calls for each record, and generate all response as outputs. While using it as a midstream, we can easily follow up on actions with the Rest call outputs in the same streamline flow.

18 comments:

Andrew Son said...

Helpful post, continue sharing more about Informatica.
Informatica Training in Chennai | Informatica Training center Chennai

shalu said...

Thanks a lot for explaining practically. Fantastic Post! RPA Training in Chennai. Get more information Robotics Process Automation Training in Chennai

Chandini Mandal said...


I appreciate your work on Informatica. It's such a wonderful read on Informatica.Keep sharing stuffs like this. I am also educating people on similar technologies so if you are interested to know more you can watch this:-
https://www.youtube.com/watch?v=56vMQ1lG-vc

Akash said...

Informatica training and Robotics Process Automation training is very popular nowadays. There are more institutes providing RPA training in Chennai. Learn to shine your skills.

Vicky Ram said...

Wonderful post!!Thank you for sharing this info with us.
Keep updating I would like to know more updates on this topic
Very useful content, I would like to suggest this blog to my friends.


Best Informatica Training Institute In Chennai


Informatica institutes in Chennai

Nikitha Raja said...

Super very nice post and thanks for sharing this useful information
Informatica Training in Chennai | Informatica Training Institute in Chennai

Informatica Read JSON said...

https://zappysys.com/blog/read-json-informatica-import-rest-api-json-file/

JSON / REST API is becoming more and more popular each day as everyone embrace cloud-centric services. Informatica Read JSON is primarily focused on Informatica users who want to do XML SOAP/ JSON / REST API Integration in Informatica. However many tips and techniques described here that will help to understand how to integrate XML SOAP / JSON / REST API in other ETL / Reporting apps such as Tableau, Power BI, SSRS, Talend, Excel and many more.

Nancy Rai said...

https://zappysys.com/blog/read-json-informatica-import-rest-api-json-file/

JSON / REST API is becoming more and more popular each day as everyone embrace cloud-centric services. Informatica Read JSON is primarily focused on Informatica users who want to do XML SOAP/ JSON / REST API Integration in Informatica. However many tips and techniques described here that will help to understand how to integrate XML SOAP / JSON / REST API in other ETL / Reporting apps such as Tableau, Power BI, SSRS, Talend, Excel and many more.

Vicky Ram said...

Thanks for sharing your knowledge keep going...

Informatica Training center Chennai

Informatica Training Institute in Chennai

sathyaramesh said...

You have provided a nice article, Thank you very much for this one. And I hope this will be useful for many people. And I am waiting for your next post keep on updating these kinds of knowledgeable things
Informatica course in Chennai
Informatica Training center in Chennai

SRI said...

Thanks for the informative article. This is one of the best resources I have found in quite some time. Nicely written and great info. I really cannot thank you enough for sharing.
rpa training in Chennai | rpa training in bangalore

Vicky Ram said...

Thank you for sharing this post.

Article submission sites

Education

Anexas Europe said...

Great Article… I love to read your articles because your writing style is too good, its is very very helpful for all of us. Do check Six Sigma Training in Bangalore | Six Sigma Training in Dubai & Get trained by an expert who will enrich you with the latest trends.

sumathi s said...

I ‘d mention that most of us visitors are endowed to exist in a fabulous place with very many wonderful individuals with very helpful things.
fire and safety course in chennai

Swetha Gauri said...

Really cool post, highly informative and professionally written and I am glad to be a visitor of this perfect blog, thank you for this rare info!
nebosh course in chennai

likitha said...

I like your blog, I read this blog please update more content on hacking,
Nice post
Tableau Online Course

Manju NaIdu said...

i like it informatica Online Training

Anexas Europe said...

The post was amazing. It showcases your knowledge on the topic. Thanks for Posting.
CPHQ Online Training in Kabul. Get Certified Online|
CPHQ Training Classes in Al Farwaniyah