The Enterprise Service Bus: Ugly Duckling or Beautiful Swan

The debate on whether the ESB is going to be a short-lived fad has recently reignited across a number of blogs.  I say debate but in fact the majority of commentators seem to be on the short lived fad side.  While I am no longer associated with an ESB vendor, I must admit to retaining a certain belief in the term – not because I think ESB is the right TLA but because the products in the category deliver real functionality that solves real problems in SOA deployments.

Lets start on a positive note by saying that I totally agree with David Linthicum’s final conclusion in his recent blog item ‘Why “ESB” will be Dead in a Year’ when he says

Focus on your own requirements, and what the product does specifically to meet them. It matters not if it’s ESB, EAI, B2B, SaaS, or EIEIO Look past the marketing at what the product does, and draw your own conclusions.

However, when you examine his objections most are related to the supposed vagueness of the term and tendency of marketing departments to hype which is rather contrary to the title of the piece.  (I also notice that of the three items he references as knocking the ESB – one was authored by him at another of his blogspots and a second included quotes from him.  These multiple manifestations further reinforces my impression that David is rapidly becoming the SOA blogosphere’s equivalent the Matrix’s Agent Smith). 

A second knock on the ESB term is well expressed by Jim Webber and comes down to the view that ESBs are used to make up for poor service design.

In this item, I will attempt to knock-back these objections…

Objection 1: The ESB is such a vague term it is worthless

It is certainly true that if you were to take all the products which claimed the badge ESB, you would see a lot of diversity of features and functions.  However, this does not correspond to vagueness per se.  So why is the ESB perceived by some as vague?

One factor was that the initial definition from Gartner was heavily influenced by the Sonic product of the time and hence was particularly minimalist, and left space for products to spuriously be given the label ESB.  It really didn’t cover much of the features which would now be regarded as necessarily in any ESB worth its salt (for instance business process orchestration).  The strangest part of this original definition for me was that an ESB required support for only “limited transformation” – a little surprising as a limitation for something that claimed to be enterprise grade.  However, the vendors and more importantly the analysts have moved on considerably since then and the Forrester Wave on the ESB published late 2005 had a very precise and extensive set of functionality requirements.

The second vagueness related accusation against the set of ESB products is that they are so different from each other.  To me at least, this is in fact not so surprising. SOA is being used to solve a broad set of problems across many industries.  It is not surprising therefore that products reflect this diversity.  What will be interesting to watch for is the refinement of the common base of functionality which any ‘decent’ ESB must have.  This is a work in progress as users get to grips with their SOA rollout and ESB-based solutions in parallel to analysts developing eco-systems and frameworks.

However, it is only fair to accept that David is also right to a degree here: vendors are rebadging their old products in an attempt to give them some new life.  But this is always the case with a sexy new product category and to my mind not the main reason for the diversity.  And finally, no I don’t especially like the term itself and in particular the wild use of the letter E for enterprise in this (and so many other software industry terms) – if government agencies use these products do they become ASBs? – however ESB has stuck and so long as we all know what the product is meant to do, we should just put up with it.

The ESB is aspirin for the headache of badly defined services

To quote Jim Webber :

But the solution is not to build services in such a crummy way, not to deploy more middleware to hack round the problem;

The "age-old problem" of putting smarts in your endpoints is exactly what makes the Internet work. Putting smarts in your endpoints is good, putting smarts in your network is not.’

I think Cisco and others may not agree with this view of the internet – the internet works because of the standards and a lot of smarts in the routers in the middle, not to mention the smarts in DNS and the rest of the internet infrastructure. 

However, if I extend the comment to the web which may have been Jim intention, there is a view of the SOA world which objects to the concept of intermediaries believing instead that the service definition if properly defined is sufficient for all SOA-based integration purposes.  The comparison to the web-browser model is interesting because I would argue that it is a very specialised (all be it very popular and wide-spread) integration model.  The clients (i.e. browsers) are dedicated consumers of the services and there is a highly evolved set of standards around this use case.  While the web servers are certainly smart, they are operating in a particular model: a client connects, interacts, disconnects.  The clients are also smart, but similarly limited: typically they interact with a single web server at a time.

This is certainly one style of integration – in fact I would describe it as a substyle of a composite application.  Brenda Michelson of the Patricia Seybold group defined this and the alternative flow style rather well in her “Service Oriented World: Cheat Sheet”.

The two primary styles of SOA used in business solution development are composite application development and flow. In composite applications, the user interaction drives a request for one or many services.  Most of the service invocations are synchronous in nature.  A composite application typically serves one business domain. Composite applications are often delivered in a portal.

In flow, business process and/or events drive the service invocations. The service invocations are a mix of asynchronous and synchronous; however, the overall flow is usually long running and asynchronous. A flow typically crosscuts business domains and often extends outside of the enterprise.

The composite application style works well for the problems it solves (i.e. portal based user driven integration) because the client is built to use the services.  This means it can be written to deal with the tricky part of using a service: inconsistencies between the information model exposed through the service and the consumers information model – if the service definition assumes data submitted in this format, write some code to submit it in that format.

Unfortunately with flow style SOA the world is very different as each business process will span potentially many systems and organizations (and hence technology and data models). For instance, the processing of a purchase of a particular financial instrument by an investor must flow between organizations (the buy side bank to the sell side bank and back), may cross many departmental boundaries (to deal with the components of the derivative such as equity, equity derivative, credit etc) and must integrate into the risk and regulatory compliance units. 

In this environment, there is a clear requirement for potentially multiple entities to control the flow (orchestration or choreography), and to bridge between the information models of the different departments and organizations (mediation) as well as ensuring it is all done in a controlled and auditable fashion (governance).  In fact, the key components of a modern ESB.  If there is no ESB, I would somebody to explain how all of this is done by the smart end-points!  Furthermore, I would to understand how this becomes possible when we reach SOA Nirvana when some believe the ESB requirement melts away.

And so, what does all of this mean

To return to my title for a moment, is the ESB an ugly ducking or a beautiful swan. Unfortunately, it is probably the ugly duckling.  ESBs are there because the real world integration - SOA based or otherwise - isn’t beautiful and clean. It is full of compromises between the way we wished the world was (full of consistent data models and simple business processes) and the way it is (every changing messy complexity).  Unfortunately, transformations into beautiful swans only happen in fairy stories and scary reality television shows.

Comments

[brenda michelson]

Hi Ronan, glad to see you back in the blogosphere!

Thanks for pointing to my cheat sheet.

I'm not particularly enamored with the ESB term (TLA), or the rebranding of old products, but I believe the underlying integration services of the ESB are important, today and in the future. I find it hard to imagine organizations will have a homogeneous platform and never need to interact outside their own walls. And even if that were the case, what are the odds they'd have semantic (dialog) consistency internally?

Let me know (when you can) what you are up to. -brenda

[greg]

I would suggest the Web model satisfies requirements for a radically "uncentralized" information exchange system, which is not what SOA really attempts to solve: the ESB or WSM products are available precisely because users need a model for their data center, a global model over which they can exercise control and enforce governance requirements. That is very different from the Web, which by design precludes that capability.

This is a critical distinction, which should be clear with a little thought experiment. Ex.: what does it mean to do impact analysis on changes in the Web? Not much, if you value your sanity. However, ask a different question: what does it mean to have impact analysis on changes to your business process? In the latter case, its clear that this is a business critical question.

[Loek]

--
I also notice that of the three items he references as knocking the ESB – one was authored by him at another of his blogspots and a second included quotes from him
--

Ronan, my posting was the only reference without quotes from David. Any particular reason why you did not refer to my vision that the ESB is a one-day fly?

[Ronan Bradley]

Greg,

Apologies for not explicity referencing your own contribution to this debate.

However, I think I covered your point when I said that I did not see the ESB as a transitionary requirement which disappears once complete SOA is achieved. To a large degree my argument centers on two points:

- ESB is just a name for a type of product and to my mind some people are getting too hung up on whether they like the name or not or too hung up on some quite narrow definitions of what an ESB is (i.e. reliable messaging) as opposed to what an ESB means today which is quite a broad and goes far beyond where Sonic Software started with the term.

- There is an argument that correctly defined services (or smart end points) remove for anything in the middle and in particular for any mediation or perhaps even orchestration. I simply don't agree for the reasons I highlight in the orginal piece.

I hope this clarifies my position on your blog-item.

Ronan