Blog

Integration is layered, like an onion

Building layers of an application integration

More and more business processes are embedded in specific (cloud) applications designed for one or some tasks. The user of these applications is often the only one who synchronises, interprets and integrates information from different systems. Application integration provides optimisation of business processes with the aim of saving time, reducing the error margin and/or cost savings.

Application integration seems to be a matter of connecting application endpoints. In practice this is actually one of the last steps and the most technically oriented. There are quite a few steps to be taken before you get to the actual connection of endpoints. We recently went through all the steps in an Exact related project and based on this example we like to share our view on how to get to the core of the integration. Let’s peel the onion off together.

The onion summarized

Integration checklist

In this checklist we take you through the practical steps of the integration process. 

Each step from this article is translated into 13 steps towards successful integration.

The 1st layer

What is the desired end result?

In the case at hand, ProductFlow wants to “build” an integration with Exact Online.

ProductFlow develops software that can be summarised as a combination of a PIM (Product Information Management) with an OMS (Order Management System) that integrates seamlessly with popular marketplaces. ProductFlow ensures the recording of the correct product content, its distribution to marketplaces and the synchronisation of the order with associated statuses.

Exact Online is a very extensive financial and logistics system (and more) in which the entire logistics and financial process is supported. Exact Online has an extensive set of APIs that support full integration.

The desired end result is;

The 2nd layer

Is there an API available?

API image
A simple Google search will probably tell you if the to be integrated applications have an API. If that answer is not immediately clear, find the supplier and get the answer there. If there is no API available, we are often looking at an (obsolete?) on premise application. It is more challenging to integrate with those types of applications. As a fallback, you can consider whether batch exchange over (s)FTP can be a useful alternative.
The 3rd layer

Is there API documentation available?

An API without proper documentation feels like a 3,000 piece puzzle with no picture of the end result. Look in the documentation not only for possibilities, but also for limitations that may exist. Pay extra attention to API limits because they may limit you later if there is large amounts of data or high frequencies that the API does not allow. That documentation is therefore essential!

The 4th layer

Collect the Credentials

You don’t (generally) get access to an API just like that. Sometimes you just need an API key, sometimes more complex steps are requested to access the API (for example Amazon Seller API). To avoid disappointment later in the process, always test whether you can access the APIs of the applications to be integrated with the credentials.

The 5th layer

Which functional areas are integrated?

logistics employee

The combination of these applications is used in this case by companies that are active on one or more marketplaces for B2C sales. To support the process, the following functional areas have been identified;

The 6th layer

The functionality of the applications

Application knowledge is essential, on the one hand to understand the process steps from layer 2, on the other hand to understand the invisible business rules (the 7th layer). Going through the entire associated process in the application contributes to a successful integration because results can be better validated and there is a better understanding of what needs to be taken into account. This exercise will also bring other issues to light like technical documentation that is not up to date, data that is visible in the interface but not described in the API, or that the relationship between field names in the interface is difficult or impossible to trace to the characteristics in the result output from the API.

When entering an order in Exact Online, Exact knows an order, delivery and invoice debtor. The debtor, contact person and delivery address must exist or are created during the entry.

When submitting that data via API, you must first check whether the debtor from the ProductFlow order already exists, for example by checking the presence of the email address. If it exists, you can retrieve a debtor’s unique identifier (GUID) and save it for later in the process. You should do the same for the contacts. For the addresses, however, it is wise to create the current delivery address stated on the marketplace as a new address and link it in the order.

If certain information is not available when integrating an order, you will have to create it first, the debtor is not yet in Exact Online, for example. Here you can already see a conceptualization of the process in various integrations, in Dovetail we call this a flow. Insight into how information flows within the applications and what information must be available at any time is a precondition for a successful integration.

The 7th layer

Know the business rules

The example above about creating a debtor, contact person and address via API is about business rules associated with the application that you need to know to prevent an old delivery address from being used for the current order.

Another example is price calculations where you have to distinguish between price master data and a price that is determined in the transaction on the basis of discounts, graduated scales or mix/match rules. A price determined in a transaction is only available in a calculation in combination with other items and other variables.

The 8th layer

Data mapping

Not all alligators are crocodiles!

mapping in dovetail

So, each application has its own naming convention for fields, both on the interface, in the database and within the API. History, advancing insight and (non) consistency means that logic is not always logical. So, the chance that the article number in both applications is called ItemID is not that big… That’s why we have to make sure that both sides of the integration speak the same language by means of mapping.

The 9th layer

Configuring the integration flows

Flow to support article about opinionated components

With all the preparations done, you can start configuring the integration flow. Within Dovetail you do not need any programming knowledge, you do not need to be able to code.
The Dovetail toolbox is well stocked with everything you need to build the integration.

It is common to build a separate flow for the authentication and to record variables that you can keep “valid” and continue to use later in the process.

You build up the flows step by step until full integration between the systems is achieved.

The 10th layer

Error Handling

Equally important is the handling of errors. Configuring the happy flow is not that difficult, but what do we do if something doesn’t go as expected? And do we actually know what could go wrong? The problem is that we don’t know what we don’t know. And the problem with that is, that we have to make an economic trade-off in the amount of time we spend looking for problems. So we make pragmatic trade-offs and control error handling for everything we can reasonably foresee;

Within Dovetail, every flow has a so-called error route. If something goes wrong in the flow, you will fall back on this error route in which you have previously determined what needs to be done.

We also say, integration is iteration, every next version is better!

The 11th Layer

Optimization

It can always be better! Undoubtedly, we now know a number of things that we did not know before. And we have probably fixed a number of errors, which reduces the chance of new errors. As mentioned, integration is iteration.

What is used a lot on a daily basis can always be improved. Realize that a user may not report errors, but “just” fix them themselves. Not realizing that not only he (or she) solves this for himself but that colleagues probably do too. Of course, that was not the purpose of integration! Walk around, ask how things are going and whether the colleagues can go through their process without manual adjustments.

The 12th Layer

Management

Invisible, but every cloud application is maintained in terms of infrastructure, updates and upgrades of operating systems, updates and upgrades of underlying frameworks, hot-fixes and updates of the application(s). All in all, that is work for a team of DevOps, back- and front-end developers, product managers, consultancy and support specialists.

However, it is important to keep an eye on the impact it has on your integration flows. If an effect of an update is that a field name has changed in application A that is part of an integration, that might have effect on all the steps in the flow.

Ensure you keep up with the pace of the updates from the applications. Having exception management in place will support you in maintaining a stable integration.

The 13th layer

Change management

Because there is always room for improvement, according to us or according to the user, because the user’s landscape changes, because new functionality is needed.
Every change requires our people to read and analyse the existing situation.

Requests for change are handled by means of an RfC, a Request for Change in which our consultant assesses the possibilities, the consequences in functionality, time and money.

The moral of the story

Integrating applications is more than developing a technical integration. There are a lot of steps before it, and a few more after it. That is why we say “Integration is complex, but not difficult”.

When peeling the onion, chances are you will have to shed a tear… We try to prevent that by telling the entire story and thereby clarifying expectations.

That is what we call Integration made easy

Check out our Integration checklist

Related resources

Democratise integration

What is the influence of democratising integrat...

What are the benefits of using an embedded iPaaS platform?

An embedded integration platform, or embedded i...

Dovetail walk through

A walkthrough the Dovetail application

An introduction into Integration made Easy by Dovetail

In this video we provide you with an introducti...