Wednesday, May 30, 2018

CONCUR INTEGRATION WITH INFORMATICA CLOUD: A TECHNICAL REVIEW

Article originally published here.

Concur is one of the industry’s most widely used travel and expense management solutions, used by more than 32,000 organizations globally. For companies that are implementing or have implemented Concur, one critical step necessary to fully unleash the power of the platform is the integration process.
After implementing Concur, companies often spend a large amount of time on tedious manual work, such as adding new users on new hire events, user off-boarding, manager changes, department changes, etc. These "maintenance" tasks seem trivial at first, but they actually require a major time commitment from Concur administrators, who could be focusing on other critical tasks. Besides, managing Concur’s interaction with other systems, such as HR or financial ledger platforms, introduces more challenges, as these tasks are not straightforward and solutions are not directly provided by any of the applications involved in the integration.
Does this sound familiar? Many companies experience these challenges on a daily basis. Without a sound integration solution in place, managing such a powerful and efficient travel and expense system can be challenging.
Informatica Cloud is the industry-leading integration platform-as-a-service (iPaaS) and data management solution, designed to connect with enterprise-ready data and applications across cloud, on-premise, big data, social, and mobile environments. Primitive Logic is proud to be an award-winning Informatica Partner (Outstanding Partner of the Year 2016Cloud Emerging Growth Partner of the Year 2015), with experience implementing hundreds of integrations. In this article, I’ll share our experience and the technical insights we gained in leveraging Informatica Cloud for successful Concur integrations.

Understand Integration Goals

When it comes to data integration, regardless of the applications involved, your first task is to understand the business and technical goals you want to achieve. You’ll also want to consider the other applications that will have to integrate with Concur.
To better understand the business goals of the integration — and the challenges involved — consider the following questions:
  • What business value (or return of investment) will this integration bring, in both the short and long term?
  • What is the current process we’re looking to integrate? What are the pain points?
  • What's the end state of this automated integration? Are we looking to roll out the integration in phases?
Gaining a better understanding of your business goals and how they align with technical needs is an important first step in designing a solid integration approach. To understand your technical needs, consider the following questions:
  • What are the data objects involved with the integration? Specifically, what types of data (data attributes on these objects) are involved? Are they structured, semi-structured, or unstructured?
  • How should the integration be triggered — real-time, event-based, scheduled, or on-demand?
  • What's the size of the data in scope? Will we need to run the integration in bulk mode?
  • How do we want to be notified on the outcome of the integration, and what information should this notification include?
  • How would this integration recover from exceptions — human intervention, auto-recovery, or a combination of both?
Answering these questions will help you further define the scope of the integration process and narrow down your design options for satisfying your requirements and achieving your goals.

Match the Approach to the Need

Let’s review the integration options around Concur — the thoughts behind each option, the reasons why each should be used in different scenarios, and how each ties back to the business and technical needs we discussed in the previous section.

Concur Connectors by iPaaS

Most iPaaS offerings provide pre-built application connectors that are ready to use — an important consideration when choosing your platform. Informatica provides two versions of Concur connectors, ConcurV1 and ConcurV2.
It's rare for a provider to offer multiple versions of connectors at the same time; however, with Concur connectors, there is a good reason for it. Most connectors are built by the iPaaS vendor on top of the API offered by the solution provider. Concur, as the solution provider, has multiple versions of its API (Concur API Portal) running at the same time for different sets of functions. That's why we see two versions of Concur connectors from Informatica.
It's essential to understand which Concur data objects are in the scope of your integration, as you can then evaluate the right connector for the right operation for these objects. For example, for the User object in Concur, different API endpoints offer different sets of functions:
  • Version 1 API offers three operations on the User object: read, create, and update. It also supports updating the user password (separate from other User attribute updates).
  • Version 3 API offers only the search operation on the User object. (Note: There is no Version 2 API for the User object.)
As noted, the two versions of API for User objects are designed for different services. Version 3 API provides robust search capability, while only limited user information can be retrieved in the output. Version 1 API on the User object is not as performant as Version 3, but provides a full span of user information with read, update, and create capabilities. To match the Concur connectors, Informatica wraps the Version 1 User API in ConcurV1 and Version 3 in ConcurV2.
As you can see, Concur Rest API is the backbone of all Concur connectors. However, since Concur connectors do not inherit all API functions, they pose several limitations:
  • Connectors lack support for custom attributes. Going back to the example of the User object in Concur, if you create a user with additional custom attributes that are not listed in the API reference doc, the connector will not work.
  • The connector can only deal with one object at a time. If you update users and user profiles in sequence (a common use case), the connector will not work.
  • Connector provisioning always incurs a licensing cost.

Concur Rest API

If you encounter limitations using Concur connectors for the above use cases, you may want to work with the Concur Rest API directly and design customized solutions to address your specific use cases.
Advantages of working with the Concur Rest API include
  • Custom Attributes: It’s common to set up custom attributes in Concur to fit records into your business use cases, which leaves the Concur Rest API as the only option for automating processes.
  • Working with Multiple Objects in Parallel or in Sequence: Most business processes involve multiple data objects. For example, for a new hire event, three objects need to be created: a user, a user profile, and a travel profile. The ability to orchestrate multiple object operations is critical for governing data integrity in an automated manner and keeping future maintenance work to a minimum.
  • Cost Reduction: As I mentioned above, each connector comes with a cost. Using generic connectors and the Concur Rest API or custom connectors (discussed in more detail below) are good alternatives to expensive application-specific connectors, leading to long-term savings.
  • Performance Benefits: With Rest API, you can choose to deal only with the data of interest instead of all data exposed by the API. This reduces the data size and results in improved performance of the overall transactions.
Informatica offers two options for implementing Concur Rest API:
  • Informatica Rest V2 Connector: This generic connector can be configured and set up to work with any Rest API endpoint. It supports a variety of authentication options and utilizes swagger definition to configure the data objects and operations involved in the transaction. The Rest connector can work as a source, a target, or a mid-stream in any typical integration operation of the Rest API call, such as read, write, or delete. It automatically converts hierarchy data from the application to relational data or vice versa, which can subsequently be used in downstream applications. (I recently published a technical talk with in-depth focus on the Rest V2 connector.)
  • Informatica Cloud Application Integration (ICAI, formerly Informatica Cloud Real Time): This option allows you to construct the Concur Rest API as a service in a service-oriented integration process, encompassing event processing, service orchestration and process management. ICAI allows you to streamline complex business use cases in a workflow-like process and invoke them in real or near-real time. As an output of the integration process, it also provides flexibility around custom error handling and customized notification content.

Concur SFTP

Concur SFTP can be used for both inbound and outbound integration, for data imports as well as exports. It is also a viable option for bulk data loads and extracts.
Concur SFTP works as either a source or a target within Informatica cloud integration and also interacts with other applications. You can implement it as a service in Informatica Cloud Application Integration and monitor it in real time, invoking an integration process upon a new file dropped on the SFTP site.
Although information on the data objects that the SFTP supports is not publicly accessible, it's clear that not all objects are supported. Concur SFTP is also an option with a relatively low cost.

Concur App Center Connectors

Concur manages a list of connectors in the Concur App Center, most of which were created by travel and expense partners. These connectors are designed mainly for consumers to import their personal data and aren't suitable for enterprise-level application or system integrations.

Avoiding Common Caveats

To conclude this overview, I'd like to share some caveats, or “gotchas,” that can complicate Concur integrations.

Skipping the Sandbox

To save on cost, some companies choose not to have a sandbox and to provision only the Concur production instance. This can create a problem, as Concur could be dealing with multiple sets of data that could become corrupted, and other integrated systems generally have both production and sandbox instances. By omitting the sandbox, you run the risk of production data becoming corrupted. Besides, certain objects in Concur cannot be deleted but only be disabled or set to “inactive.” This increases the amount of redundant, “noisy” data, which can be difficult to manage and may affect data integration with other systems, as data integrity will be compromised.
There is no ideal solution in this case other than getting a sandbox, even temporarily for integration design, development, and testing. Otherwise, specific considerations need to be taken on architecting and design to minimize the injection of redundant data — such as introducing a data flag to categorize individual transactions (update on existing data vs. create with new data) — so that we can reset these data accordingly after development and testing. Additionally, good planning is needed for test data to minimize the impact of the noisy data and avoid impacting data integrity.

Custom Fields Created in a Web Browser

As we discussed above, it's a common practice to customize the Concur application by adding custom data fields, which administrators can do through the Concur application in a web browser. However, not all custom fields created by web browsers are necessarily supported right away by the Concur Rest API. As a result, the API may not support all the custom attributes that are in use at any given time. You can submit a support ticket, but it could take some time before Concur releases an API update.

API Changes

Finally, the backbone of application integration is the underlying API, which evolves over time. It could definitely affect your integration if the API version is deprecated or even updated with different behavior during the process. Be mindful and stay on top of new API releases.
If your organization is planning to implement Concur, keep in mind that the integration process is a vital step that must be carefully and mindfully planned. Begin by clarifying your business goals and technical needs, then select the approach that will fit your requirements (and be sure to avoid the “gotchas”). Executing a well-designed integration will help ensure that your organization gets the most benefit out of this powerful platform.

Friday, August 4, 2017

A Quick Talk On Generic Cloud App Integration With Informatica Cloud

The Informatica Cloud Integration platform is the industry's complete cloud integration platform as a service (iPaaS). Informatica has been continuously playing as the leader role in this space for the 4th consecutive year (Gartner's margic quadrant for enterprise integration platform). It's definitely your top choice for data integration management.

With the emerging SaaS application on the market, it's essential for integration platform to provide a flexible and efficient connector to integrate various cloud applications in a generic manner. Indeed there are many specific connectors built for individual applications by Informatica (Tier 1 and Tier 2 connectors) or vendors themselves (OEM connectors). But let's face it: most cloud applications evolves quickly on new features and their APIs get updated quicker than you realize. How quickly the connectors can turn around with the most recent API updates? The answer is definitely the connector update lags behind, as most connectors get updated once a year or longer than that.

For what's worth, Informatica introduced the Rest V2 connector in fall of 2016. The V2 connector is an successor of its ancestor which is just named as "Rest Connector". It allows to connect with the latest Rest API of the cloud application directly to read or write data. You can fully leverage it no matter the cloud application is a source, a target or a lookup system. It offers great improvements on flexibility, efficiency, ease of use besides performance. Considering the majority use of Rest API as the industry standard for emerging cloud services, this Rest V2 connector can fill in the role of a generic connector for cloud applications with Rest API. In this article, I will reveal the great features and caveats followed by uncovering some hidden gems of the Rest V2 connector that many may not have realized yet.

One of the great features of the Rest V2 is the introduction of hierarchy data support. As we know, the Rest API deals with either XML or JSON data which are hierarchy data structure. Before the appearance of Rest V2, transformation of hierarchy and relational data is cumbersome and even requiring writing your own script to transform the data as part of the integration process. At the Informatica fall release of 2016, the same time Rest V2 is introduced, the Hierarchy Builder/Parser is brought to life to fill the gap. Hierarchy builder is to build hierarchy data from relational data and hierarchy parse does the opposite way. Let's take a look at how hierarchy builder/parser works.

First, a hierarchy schema needs to be created. It is used by hierarchy builder/parser to present the data structure. To create a hierarchy schema, one will need to have a XML schema file (.xsd) or a JSON sample file (.json). Note the XML schema file is not necessarily provided by the vendor and one has to create it based on the XML payload.


After the creation of hierarchy schema, a hierarchy builder/parser can be used in a data mapping to map hierarchy data to relational data or vice versa.



That seems quite easy, compared to writing your own script to transform. But let's look at the Rest V2 connector as it's even better.

The core of Rest V2 connector configuration is the Swagger support. I have to thumb up on Informatica on the betting of Swagger because of its popularity and great potential as industry standard for API framework. Not all vendor provides swagger definition for their API. But no worry, Informatica provides this tool to you right inside your Informatica cloud web console. It's under Configure -> Swagger files. If you don't see this option, please contact with your Informatica support to enable it. You will need to have Rest V2 connector license to use it.

While creating the Swagger file, you basically defines the API details - URL, verb (POST/GET etc), authentication (username/password, token, etc), Headers, Query parameters, Body and sample response file. A couple of caveats in this process are:

  • The JSON response file is required even you are dealing with XML response. In this case, you will need to convert the XML response in JSON format. There are many online tools to do that.
  • If your raw body is XML format and you have issues creating the Swagger, after making sure nothing is wrong on your end, you may want to contact Informatica support requesting them to create the Swagger for you. Informatica support does have their own internal tool to create swagger file. Informatica will improve the Swagger tool based on my conversation with them and most likely it's no longer an issue soon.



After the Swagger file, create the Rest V2 connection which is quite easy as majority of the info needed for the API is filled in the Swagger file.


Now the Rest V2 Connector is ready to use in the data mapping. The Rest V2 can be used a source to read data or the target to write data in this case. Of course the API you put in the initial Swagger file has to comply with read or write operation. The hierarchy data mapping can be achieved in the Rest V2 Connector as a source/target itself. No need to use hierarchy builder or parser.

To save the best to the last, the great power of using Rest V2 connector is to use it as a midstream. Let's work on a use case: say I need to pull data from A system and push to B system via Rest API. If I were using Rest V2 as the target for B system. At the end of the flow, how do I know the data is successfully taken in B? I will need to start another pipeline or another data mapping task to examine the log file and take action from here. When using it as a midstream, the response of the Rest call can be transmitted to the next step for follow up action in the same streamline flow. This is a differentiator and made me believe all Rest V2 connector should not serve as target but midstream instead.

Another hidden gem on the Rest V2 connector as a midstream, is the capability of processing multiple Rest calls. This is definitely a great sweet feature. For any Rest call, one payload corresponds to one request and generates one response. If you have multiple payloads to be processed in one integration, essentially you need a loop function to loop through individual records and make the Rest call for each record. Without Rest V2 connector, this is impossible as there is no such loop function in Informatica data integration. The alternative would be to use Informatica Cloud Real Time (ICRT) which provides a workflow type of designer that a task can be jumped from one step to another based on certain condition - basically to achieve loop function. With Rest V2 connector, there is no additional configurations needed and it can process multiple records in one step - taking multiple records as inputs, make Rest calls for each record, and generate all response as outputs. While using it as a midstream, we can easily follow up on actions with the Rest call outputs in the same streamline flow.

Wednesday, August 24, 2016

WebCenter Portal and Jdeveloper Version Contrast

For easy reference, here is a list of version contrast between patchset, WebCenter Portal, Jdeveloper with corresponding WebCenter extensions.

(the info is extracted from Oracle doc 1627574.1)


Patchset + Bundle PatchWebCenter PortalJDeveloperWebCenter Extension
12c12.2.1.0.012.2.1.0.0Framework: 12.2.1.0.0.151013.2001
Core: 12.2.1.0.0.151005.0601
dot9 + BP311.1.1.9.16011911.1.1.9.011.1.1.9.0.160107.0701
dot9 + BP211.1.1.9.811.1.1.9.011.1.1.9.0.151005.2311
dot9 + BP111.1.1.9.211.1.1.9.011.1.1.9.0.150701.0100
dot911.1.1.9.011.1.1.9.0*11.1.1.9.0.150521.2301*
dot8 + BP811.1.1.8.811.1.1.7.011.1.1.8.0.150618.0701
dot8 + BP711.1.1.8.711.1.1.7.011.1.1.8.0.150331.0703
dot8 + BP611.1.1.8.611.1.1.7.011.1.1.8.0.141219.1406
dot8 + BP511.1.1.8.511.1.1.7.011.1.1.8.0.141001.1208
dot8 + BP411.1.1.8.411.1.1.7.011.1.1.8.0.140627.1403
dot8 + BP311.1.1.8.311.1.1.7.011.1.1.8.0.140404.0207
dot8 + BP211.1.1.8.211.1.1.7.011.1.1.8.0.131217.0204
dot8 + BP111.1.1.8.111.1.1.7.011.1.1.8.0.130926.1405
dot811.1.1.8.011.1.1.7.011.1.1.8.0.130707.1955
PS6 + BP111.1.1.7.111.1.1.7.011.1.1.7.0.130702.0214
PS611.1.1.7.011.1.1.7.011.1.1.7.0.130301.0422
PS5 + BP611.1.1.6.611.1.1.6.011.1.1.6.0.130326.0044
PS5 + BP511.1.1.6.511.1.1.6.011.1.1.6.0.121221.0259
PS5 + BP411.1.1.6.411.1.1.6.011.1.1.6.0.120928.1201
PS5 + BP311.1.1.6.311.1.1.6.011.1.1.6.0.120626.1201
PS511.1.1.6.011.1.1.6.011.1.1.6.0.111214.0952
PS4 + BP411.1.1.5.411.1.1.5.011.1.1.4.0.120306.2202
PS411.1.1.5.011.1.1.5.011.1.1.4.0.110415.0606
PS311.1.1.4.011.1.1.4.011.1.1.4.0.101223.1848
PS2 + BP311.1.1.3.311.1.1.3.011.1.1.3.0.101208.0400
PS211.1.1.3.011.1.1.3.011.1.1.3.0.100707.1000
PS1 + BP411.1.1.2.411.1.1.2.011.1.1.2.0.100510.1800
PS111.1.1.2.011.1.1.2.011.1.1.2.0.100311.1200
 * Guide to Developing Portals with Oracle WebCenter Portal and Oracle JDeveloper refers to the vesion of WebCenter Portal extensions as 11.1.1.8.0 (Chapter 2, 2.2.2 Installing the WebCenter Portal Extension for JDeveloper). These extensions were renamed to be in sync with the version of JDeveloper used for Development.

Wednesday, April 20, 2016

Overview of "Profile" and "Folder" in WebCenter Content

In this post, I will go over the basics of two features in WebCenter Content (WCC) - Profile and Folder - and then talk about their own design considerations and usages, and finally discuss their differences.

In WCC, the Profile (a.k.a metadata profile or content profile, to differentiate from user profile) is an approach of metadata modeling. The profile is a powerful tool that can be used to manage the metadata fields to achieve efficient processing of content items, such as check-in, update, search, etc.

Essentially, content profiles consist of set of rules that manage the display of the metadata fields of content items. Through content profiles, you can control what metadata fields should be displayed/hidden, required or not, read-only/editable, initialized with default value or not, and grouped based on the action to the content – whether you are viewing the content information, checking in, updating or searching. Profile provides the capability to customize the user interface of the metadata presented to the users. This is important as the profile can not only improve the user experience but also improve the data quality and accuracy. If a user is presented with too many irrelevant fields when dealing with a content item, the experience could be daunting and very likely the user would not provide quality data input. It doesn't need to take long before your content system filled with more and more irrelevant data and is abused in a sense. Bad profiling could definitely hurt searching experience as well. Consider a profile is defined in a way that captures irrelevant and inaccurate data, the overall search output would be misleading.

One of the frustrations I heard from customers was users wanted multiple profiles on a content item. In WCC, only one profile can be associated with a content item. You may find it a drawback from your unique business use cases or the way you handle the content processing. However, it’s designed in this way for a purpose. You can consider the profile is to define the type of a content item. Any content item belongs to a certain type but not more than one types. The capability of aggregating multiple profiles into one content item could very much likely lead to data abuse and subsequently many other undesired outcomes. Another frustration was “specific resources are required to build or update the profiles”. From what we have discussed, a new profile should be created only if there is a new type of content item required in the system. A profile should stay as static as possible and be updated only if the type of the content item changes with your business context. If the business context doesn’t require any new types of content or any existing types to be retired, the profile should not be built or updated frequently. You may need to look into the initial design and definition of the profiles to match the business needs. Since profile is the approach of metadata modeling, the proper metadata design is essential for daily content management. Metadata in WCC should match your enterprise taxonomy to achieve best outcome on content organization.

Speaking of the number of the profiles, there is no good or bad number. It just needs to fit your business needs. The 50-100 range is the average number of profiles for all US WCC implementations (the statistics is not published anywhere but from a technical summit with Oracle WCC team). The extreme case I have seen with a client is over thousands of profiles in the WCC system and it works just fine. There is a performance caveat with very high number of the profiles in the WCC system. I encountered such performance issue in one of my WCC implementations and the issue has been addressed by Oracle team. For details, please check here.

Folder, in WCC, is a way to structure and organize content items. It’s worthy to note that in WCC the folders are standalone “virtual” structures. Content items are not physically stored in any folder. Every content item in a folder has a metadata field (xCollectionId) to store a numeric folder ID that links the content item to a folder. It behaves like a symbolic link in WCC system.

Content folders offer a conventional hierarchy structure that provides easy access to a content item in WCC. They are just like the directories on your local laptop that point to virtual locations of the content system. With folders in WCC, you can just perform actions like you do in the conventional file system. Quoted in Oracle documentation, “The familiar folder and file model provides a framework for organizing and accessing content stored in the repository. Functionally, folders and files are very similar to those in a conventional file system. You can copy, move, rename, and delete folders and files. You can also create shortcuts to folders or files so you can access a content item from multiple locations in the hierarchy. You can think of the files in the Folders interface as symbolic links or pointers to content items in the repository. The operations you perform in the Folders interface, such as searching or propagating metadata, effectively operate on the associated content items.” 

The hierarchical folder interface is achieved by a component installed in WCC. This component is called FrameworkFolders. It is a scalable enterprise solution and is intended to replace the earlier Contribution Folder interface (called Folders_g component). For a comparison of FrameworkFolders and Folders_g, you can visit this link for more details.
  • There are different types of folders can be used to organize content to fit your different needs. Traditional Folders: it’s the general folder we have discussed that you use to organize your content just like the one you use in your computer.
  • Query Folders: it’s a folder you can create based on a search/query result. It contains collections of document based on the search criteria you defined. You can save the query folder just like you create a regular folder.
  • Retention Folder: it’s a type of query folder with retention rules.

Comparison

Conceptually, the Folder and Profile are distinct on their functionality and their design purpose. Profile can be considered as a way to define a content “type”. Folder, like the conventional folder in your laptop, is a way to aggregate and organize content. You can store content items with the same profile in the same folder or different folders. You can have a folder containing content items with the same profile or different profiles. I will use an example to better illustrate the usage of the profile and folder.  Say in your company you have the following types of content items: legal document, sales document, and reports. You also have the following departments: HR, IT and Sales. All departments may have their own legal documents and reports. Sales document would almost fall into the sales department not the other two. In this case, you may want to take the content types as the profiles and aggregate the content into folders as per the departments. You don’t want to create profile based on departments because the department can have all kinds of content items and it’s not just one static type. If somehow you define the profile as per department, you will find yourself in a way that has to create/update profiles all the time.

Folder and Profile do reveal some similarities in ways that how content is managed. You can manage content items based on a folder or a profile.  For example, you can search content items either by a folder or a profile; you can batch process content items in a folder or a profile, such as manage workflow, govern security, update content information, etc. On the other hand, folder and profile do have many differences from the way they are designed.
  • Folder, just like a file, can have its own metadata. You can also propagate metadata from a folder to the subfolder and the content items within it. But for profile, it is a way to manage metadata and you cannot apply metadata on top of a profile.
  • Folder has its own security. Each folder has an owner who can modify its metadata and delete it if needed. But the folder owner doesn’t have any additional privileges over the content items inside the folder. Profile has little to do with security directly. But since Profile is to manage the metadata, it could manage profile indirectly. The “Security Group” and “Account” metadata can be used to manage security of a content item.
  • With folder, you can perform basic content retention scheduling by creating a retention query folder, assigning retention attributes to the folder, and then configuring the retention schedule. There is also a specific folder type – retention folder in WCC – which is based on query folder with rules for content retention. Since Profile is to manage the metadata, it has little to do with retention directly.
  • In workflow, actions can be applied on top of content items either within a folder or associated with a profile. In this perspective, folder and profile have similar effects.
  • WCC doesn’t have standalone tagging service. But you can create custom metadata for this purpose. Folder has its own metadata, so you can apply the custom tag on top of a folder. Profile, again, as a way to model metadata, can be used to manage any metadata field, including the tagging.
In a quick summary, Profile and Folder are two different concepts in WCC. Although they may reveal some level of similarities in how content can be managed, their design basis are quite distinct. Profile can be considered as a way to define the "type" of a content item and provide a customizable user interface for users to manage their content. Folder provides a virtual hierarchical structure just like the conventional file system in your computer to help to organize and manage content. They should be used and designed as per the essentials of their function, to avoid inefficient content management.