More and more business processes are embedded in specific (cloud) applications designed for one or some tasks. The user of these applications is often the only one who synchronises, interprets and integrates information from different systems. Application integration provides optimisation of business processes with the aim of saving time, reducing the error margin and/or cost savings.
Application integration seems to be a matter of connecting application endpoints. In practice this is actually one of the last steps and the most technically oriented. There are quite a few steps to be taken before you get to the actual connection of endpoints. We recently went through all the steps in an Exact related project and based on this example we like to share our view on how to get to the core of the integration. Let’s peel the onion off together.
The onion summarized
In this checklist we take you through the practical steps of the integration process.
Each step from this article is translated into 13 steps towards successful integration.
The 1st layer
What is the desired end result?
In the case at hand, ProductFlow wants to “build” an integration with Exact Online.
ProductFlow develops software that can be summarised as a combination of a PIM (Product Information Management) with an OMS (Order Management System) that integrates seamlessly with popular marketplaces. ProductFlow ensures the recording of the correct product content, its distribution to marketplaces and the synchronisation of the order with associated statuses.
Exact Online is a very extensive financial and logistics system (and more) in which the entire logistics and financial process is supported. Exact Online has an extensive set of APIs that support full integration.
The desired end result is;
- a number of integration building blocks around ProductFlow with which they can onboard their customers faster, without deploying their own development capacity,
- a standard, predictable integration between ProductFlow and Exact Online Handel as an optimal application set for their target group,
- collaboration with an integration specialist in order to keep their own development capacity focussed on their core activities,
- predictable functionality for the user of the applications, taking the systems, the processes and the people who use them into account
The 2nd layer
Is there an API available?
The 3rd layer
Is there API documentation available?
An API without proper documentation feels like a 3,000 piece puzzle with no picture of the end result. Look in the documentation not only for possibilities, but also for limitations that may exist. Pay extra attention to API limits because they may limit you later if there is large amounts of data or high frequencies that the API does not allow. That documentation is therefore essential!
The 4th layer
Collect the Credentials
You don’t (generally) get access to an API just like that. Sometimes you just need an API key, sometimes more complex steps are requested to access the API (for example Amazon Seller API). To avoid disappointment later in the process, always test whether you can access the APIs of the applications to be integrated with the credentials.
The 5th layer
Which functional areas are integrated?
The combination of these applications is used in this case by companies that are active on one or more marketplaces for B2C sales. To support the process, the following functional areas have been identified;
- Article managementArticles “start” in Exact and are transferred to ProductFlow where further enrichment of product information takes place
- Stock and price updates The stock and the basic sales price in Exact are leading, they are frequently updated in ProductFlow, in ProductFlow prices can be adjusted per marketplace
- Sales orders The marketplace customer order enters ProductFlow from where the status is synced with the marketplace The customer order is loaded into Exact so that the logistics process can start there. Part of the integration is the creation of debtors, contacts and addresses
- Deliveries (Partial) deliveries are administered from Exact as part of the logistics process Deliveries are synchronized with ProductFlow The changed order (line) status is synchronized with ProductFlow
- Track & Trace In Exact, the track & trace code and the carrier are added, these are synchronized with ProductFlow Payment, invoicing and financial booking The invoicing and booking is part of the process in Exact but is not part of the desired end-result
The 6th layer
The functionality of the applications
Application knowledge is essential, on the one hand to understand the process steps from layer 2, on the other hand to understand the invisible business rules (the 7th layer). Going through the entire associated process in the application contributes to a successful integration because results can be better validated and there is a better understanding of what needs to be taken into account. This exercise will also bring other issues to light like technical documentation that is not up to date, data that is visible in the interface but not described in the API, or that the relationship between field names in the interface is difficult or impossible to trace to the characteristics in the result output from the API.
- Go through the process in the application(s) interface and take print screens for later reference
- In preparation for the mapping (the 8th layer), you can provide the fields on the print screen with an identification and make a “paper mapping” of it. Which fields of both applications are needed and have the same meaning
- It is very useful to include the API output as example / reference data in the paper mapping. Make sure that the record that serves as reference is completely filled with different values so that you can trace the value back to an identifier (on screen / in dataset)
- Define which fields can be suitable as unique identifiers, sometimes visible in the interface, sometimes only as API output
When entering an order in Exact Online, Exact knows an order, delivery and invoice debtor. The debtor, contact person and delivery address must exist or are created during the entry.
When submitting that data via API, you must first check whether the debtor from the ProductFlow order already exists, for example by checking the presence of the email address. If it exists, you can retrieve a debtor’s unique identifier (GUID) and save it for later in the process. You should do the same for the contacts. For the addresses, however, it is wise to create the current delivery address stated on the marketplace as a new address and link it in the order.
If certain information is not available when integrating an order, you will have to create it first, the debtor is not yet in Exact Online, for example. Here you can already see a conceptualization of the process in various integrations, in Dovetail we call this a flow. Insight into how information flows within the applications and what information must be available at any time is a precondition for a successful integration.
The 7th layer
Know the business rules
The example above about creating a debtor, contact person and address via API is about business rules associated with the application that you need to know to prevent an old delivery address from being used for the current order.
Another example is price calculations where you have to distinguish between price master data and a price that is determined in the transaction on the basis of discounts, graduated scales or mix/match rules. A price determined in a transaction is only available in a calculation in combination with other items and other variables.
The 8th layer
Data mapping
Not all alligators are crocodiles!
So, each application has its own naming convention for fields, both on the interface, in the database and within the API. History, advancing insight and (non) consistency means that logic is not always logical. So, the chance that the article number in both applications is called ItemID is not that big… That’s why we have to make sure that both sides of the integration speak the same language by means of mapping.
The 9th layer
Configuring the integration flows
With all the preparations done, you can start configuring the integration flow. Within Dovetail you do not need any programming knowledge, you do not need to be able to code.
The Dovetail toolbox is well stocked with everything you need to build the integration.
It is common to build a separate flow for the authentication and to record variables that you can keep “valid” and continue to use later in the process.
You build up the flows step by step until full integration between the systems is achieved.
The 10th layer
Error Handling
Equally important is the handling of errors. Configuring the happy flow is not that difficult, but what do we do if something doesn’t go as expected? And do we actually know what could go wrong? The problem is that we don’t know what we don’t know. And the problem with that is, that we have to make an economic trade-off in the amount of time we spend looking for problems. So we make pragmatic trade-offs and control error handling for everything we can reasonably foresee;
- the API is not reachable or not responding
- authentication fails or is interrupted
- essential data is missing
- the flow does not go through the entire process
Within Dovetail, every flow has a so-called error route. If something goes wrong in the flow, you will fall back on this error route in which you have previously determined what needs to be done.
We also say, integration is iteration, every next version is better!
The 11th Layer
Optimization
It can always be better! Undoubtedly, we now know a number of things that we did not know before. And we have probably fixed a number of errors, which reduces the chance of new errors. As mentioned, integration is iteration.
What is used a lot on a daily basis can always be improved. Realize that a user may not report errors, but “just” fix them themselves. Not realizing that not only he (or she) solves this for himself but that colleagues probably do too. Of course, that was not the purpose of integration! Walk around, ask how things are going and whether the colleagues can go through their process without manual adjustments.
The 12th Layer
Management
Invisible, but every cloud application is maintained in terms of infrastructure, updates and upgrades of operating systems, updates and upgrades of underlying frameworks, hot-fixes and updates of the application(s). All in all, that is work for a team of DevOps, back- and front-end developers, product managers, consultancy and support specialists.
However, it is important to keep an eye on the impact it has on your integration flows. If an effect of an update is that a field name has changed in application A that is part of an integration, that might have effect on all the steps in the flow.
Ensure you keep up with the pace of the updates from the applications. Having exception management in place will support you in maintaining a stable integration.
The 13th layer
Change management
Because there is always room for improvement, according to us or according to the user, because the user’s landscape changes, because new functionality is needed.
Every change requires our people to read and analyse the existing situation.
Requests for change are handled by means of an RfC, a Request for Change in which our consultant assesses the possibilities, the consequences in functionality, time and money.
The moral of the story
Integrating applications is more than developing a technical integration. There are a lot of steps before it, and a few more after it. That is why we say “Integration is complex, but not difficult”.
When peeling the onion, chances are you will have to shed a tear… We try to prevent that by telling the entire story and thereby clarifying expectations.
That is what we call Integration made easy
