API integration with SOA

APIs are something I've discussed a lot recently and this blog continues the theme. In this entry I discuss how API management relates to Service Orientated Architecture (SOA). I discuss how SOA is not going away but it is going to be greatly simplified by the introduction of API management and its concepts. I also discuss how IBM's API Connect product works in both a "traditional" SOA world as well as the more rapid and potentially more agile world, that APIs have grown from.

A brief.... History of Service Orientated Architecture

Service Orientated Architecture (SOA) came to being in the mid 90's and really came to prominence in the Noughties. SOA encompasses many elements and, although synonymous with Web Services, WSDL and application servers, it is important to note that SOA is not tied to those.

When discussing SOA the products most often highlighted will be an Application Server (for running the applications), an ESB (data transformation, routing and logging) with (if you're in the right group of friends) some kind of gateway server (security, throttling and some transformation). Behind those products will almost certainly be a database (probably relational) and, if you're in a banking environment, a transaction server (probably IBM CICS).

As SOA advanced service registries came into being to store the service definitions. There is a lot of confusion around service registries, enough to say that they are usually only used by large enterprises and can be used at both run-time and development time. 

The diagram below shows a simple SOA architecture with secure gateway up-front and an ESB acting as a hub for all the various services. Clearly, this is an enterprise view of the world, but the concepts won't change much in a smaller environment or one that has different back-ends. 

API Management Components

It will come as little surprise to find that the main components required to enable API Management are identical to the ones required to create an SOA architecture. If we look at the architecture of an API run-time environment (below) we can see that we still need the secure gateway to do all that throttling and security of the back-end - just as we did with SOA. 

The API Store may look like a development time addition that the SOA environment didn't have but it has very similar functions to the development registry in SOA, The store allows developers to find and register their usage of the APIs that are exposed in the gateway. This uses a registry underneath it to hold that information - in a very similar fashion to how the SOA developer registry holds details of the services that are being offered by the back-end services. The key difference is the ability for users to register their usage of the API - something that was usually done more manually in an SOA environment .

Basic architecture for an API Environment

The APIs themselves are hosted in the gateway server. Notably however, the gateway usually does not host the actual API service logic - it just acts as a conduit to the actual back-end services it's exposing. So, this picture requires the services that implement the API's to be added in.

API Management gateway fronting SOA environment

It's clear from the above that the SOA architecture still holds true in an API environment. The services are still required to be run somewhere and the SOA architecture is what we know and (cough) love. It's just that now, instead of exposing e.g. a web app, the gateway is now exposing an API. 

Separation of Concerns

The above picture is telling for what it doesn't show - the actual application using the API ! 

Putting that client in doesn't appear to make a lot of difference to the architecture - but, of course it does. The application, in this case a mobile app, has to be served up and deployed from somewhere, so there is a whole load of additional infrastructure which may need to be created for that. 

Although, at first thought, this may appear to be a problem, this actually highlights how the use of APIs can give a much clearer separation of concerns - the organisation who creates and exposes the APIs versus the organisation who creates the app using the APIs. Yes, they may be one and the same organisation, but, even in that case, APIs give us a clear separation of ownership between the API owners and the application owners - something that nearly always will bring benefits.

Building New Services

I've highlighted that when exposing current services the most obvious route is to expose them through the SOA architecture. However, that will probably mean a lot of web service calls and all the complexity which that can bring. This is the antithesis of APIs which are JSON/REST led. 

In order to get an organisation connecting to its back-end systems quickly and decrease time-to-market, IBM's API-Connect product has introduced its micro gateway. This is a full-function API gateway and related tooling but is also designed to host the services directly e.g.in the diagram below access the databases, as well as proxy other services. These are then exposed using the usual API JSON/REST combo.

One key feature of the API-Connect product is that regardless of whether the API is being exposed through the micro gateway or the full-blown gateway, The user of the APIs should get a single experience i.e. All the APIs are registered and made available through a single API store.

Although this is an IBM specific offering, there is a trend in the industry for exposing services in a more dynamic fashion than it is perceived possible by using traditional SOA technologies. Micro-services and technologies like Node.js play in this arena (indeed, IBMs micro gateway is a node.js server under-the-hood).

Conclusions

I've discussed how SOA architecture is more than just web services and actually consists of a number of tried and tested architectural practices e.g. ESBs, Application Servers. This means that although APIs are seen as JSON/REST led they can still, and probably should, use the SOA architecture in the background to expose current, and potentially new, services.

I've also shown that, for new projects, where time is of the essence, IBM and others are introducing new products and gateways that are designed to speed time-to-market. These new gateways are being driven by the API economy which demands simplicity and agility.

I don't believe that these new products will negate SOA architectures for some time to come. 

Inevitably these architectures bring their own problems - for instance, proliferation of micro-services. In addition, all the standard integration problems still abound, regardless of what technology is being used. On that basis, I believe that many of the new technologies we are seeing will have to accommodate all the old problems. The differentiator that they cater for is ever reducing time-to-market and user expectations. I already see many products either being rewritten so they are easier to use or are being usurped by products that essentially do the same thing but in a much more agile fashion.