“We need to integrate these two apps”. As a digital consultancy with expertise and experience with API development, this is a sentence we hear a lot. When the requirements are straightforward and there’s a native integration available, then the conversation is straightforward. 

When there isn’t a native integration, however, things get a little more complex and we need to start planning the details of how the custom integration should be built. It’s at this point, that our less technical clients need help understanding the components of an integration. So many integration conversations talk about ‘using the API’ that it’s easy to forget that each app’s API is just one small part of the overall integration. 

Behind the scenes, a lot of data processing needs to take place and that involves creating ‘middleware’. This article will help you to understand the core features of an API integration and the different types of middleware that you can use. 

 

The three essential functions of an API integration

Any software integration can be broken down into three parts: 

  • Send a data payload from the source app
  • Process the data into a format accepted by the target app
  • Push the formatted data to the target app

If the integration needs to be bi-directional, then you’ll also want to have the same behaviour in reverse. 

Think of this like a translation service that allows two pen pals to communicate despite speaking different languages. Let’s pretend for a moment that we live in a pre-internet era and an Italian person wants to send and receive letters to/from a French person. In this scenario: 

  • The Italian person writes a letter in Italian and posts it to the translation service
  • The translation service receives the letter from the Italian person. They read the letter and write a translated copy of it in French. 
  • The translation service posts the French copy of the letter to the French recipient. 
  • The French person receives the letter and is able to read it because it’s in their native language. 

A successful API integration operates in exactly the same way. Each app works in its own ‘language’ but depends on the middleware to translate the data into the correct format. 

 

Getting data from the source app

When capturing data from the source app, there are two approaches you can take: 

  • Option one: push the data from the source app when a change occurs
  • Option two: pull the data from the source app on a schedule

Generally speaking, option one (pushing the data) is preferable because this keeps the two systems in sync on a realtime basis. Pulling the data (option two) means that the data is only fully synchronised as frequently as you are scheduling the data pulling process. 

Processing the data

Getting the data from the source app is relatively straightforward. Pushing the data into the target app is also a pretty standard bit of work. These actions are completed using the API for each system. 

The less talked about aspect of an integration is the data processing but this is where the majority of work takes place. 

This data processing takes place using ‘middleware’.

What is middleware? 

Middleware is an application that processes inputs and then outputs it in a format accepted by the target app. This middleware could be an off-the-shelf program (such as Zapier, Integromat or HubSpot Operations Hub) or it could be custom code that you host with a cloud services provider (such as Google Cloud, Amazon Web Services or Microsoft Azure) or on your own server. 

What is middleware?

Regardless of whether you use off-the-shelf middleware or custom code, you’ll need to work with someone who understands how to use that system to receive, process and transmit data. 

Even with the simplest integration projects, you’ll need to perform some data processing. Take, for example, an integration that takes a new contact’s name, email address and phone number from one app and copies that data to another. Here is an outline of some of the data processing that will need to happen: 

Step one: find the required data from the source app’s payload

Your source app will have sent information in a payload formatted using its own schema. As a result, your middleware will need to:

  • identify the payload property representing first name
  • identify the payload property representing last name
  • identify the payload property representing email address 
  • identify the payload property representing phone number
  • Identify the payload property representing record creation date

Step two: process the data

Now that your middleware has identified all of the correct data, it needs to convert it into a format that the target app will accept. To achieve this, it will need to:

  • process the phone number from the received format (e.g. full phone number including international dialling code) into the format accepted by the target app (e.g. one property for the international dialling code and another property for the localised phone number)
  • process the record creation date from received format (e.g. DD/MM/YYYY) into the format accepted by the target app (e.g. UTC date code)
  • trim whitespace from the first name, last name and email address properties
  • validate that the source app has sent an email address for the email data (i.e. that the data includes an ‘@‘ symbol, a domain and a TLD). If not, then trigger an error notification process. 

Step three: send the required data to the target app

With the data processed into the right format, your middleware now needs to send it to the target app. As a result, the middleware will need to:

  • arrange the processed data into a single payload in the format accepted by the target app
  • trigger target system’s API using a webhook that includes the payload
  • listen for the response of the target app 
  • if the target app sends an error code, then trigger either error handling or error reporting processes.  

None of the above is complicated for a capable and experienced API developer, assuming that they have access to reliable API documentation for both the source and target app. It does, however, show that there’s a lot more to an integration than just ‘doing it over the API’. The data processing is the vital part of your integration and it needs to be carefully planned, built and tested. 

How does a bi-directional integration differ from a mono-directional integration? 

A one-way (or mono-directional integration) will work exactly as outlined above. A two-way (or bi-directional integration) will need more planning and development work. 

Revisiting our metaphor of a translation service, the Italian and French recipient can easily communicate with each other because the translation company is able to speak both languages. For a bi-directional integration, it’s a little more complex and our metaphor starts to break down. 

You can’t just ask the same middleware code to ‘work in reverse’. 

Instead, you’ll need to create additional code that treats this as an integration in its own right. In many respects, a two-way integration is really just two one-way integrations. The difference being that you’ll probably add some additional rules that are designed to prevent triggering an infinite loop (i.e. each integration triggering the other one because a change has been identified). 

When estimating the workload for a two-way integration, you’ll probably find that the time required is about 1.5x that required for a one-way integration. You’ll have done most of the heavy lifting while creating the first portion but you’ll need about half the time again to create the reverse journey and any conditional code to prevent unintended loops. 

How do I get started with an API integration? 

Before you even start looking at middleware or API triggers, we strongly recommend that you carefully plan out your use case, user stories and acceptance criteria. One of the most common mistakes we see companies make is that they try to ‘integrate’ two tools but they haven’t really considered what that means.

For example, do you want to have two synchronised databases? Are you looking to enrich data in the target app with data in the source app? Are you looking to trigger a customer-facing action in the target app based on an event in the source app? 

‘Integration’ is a very broad term. Jumping straight into building your integration rather than planning what success looks like can be a very costly mistake. SpotDev’s Diagnostic Engagement is designed to save your company time, effort and money by ensuring that your requirements are carefully planned and documented, leading to an efficient and elegant solution. 

Once you’ve planned your integration, the best way to deliver on that requirement should become more evident. It should be clear if it’s very straightforward and, therefore, suited to a tool like Zapier or Integromat.

Your planning should also help you to identify how many records you’ll be processing on a monthly basis, which is the information you need to accurately price the ongoing costs of your middleware. If you’re processing thousands of records each day, then you’ll save money with a custom-coded integration on Google Cloud even if the technical requirements could be handled by Zapier. 

Either way, if you’ve got questions about a custom integration, then you can book a call with one of our consultants by visiting this page of our website.