In future posts we will get into the architecture, use cases, and additional details, but as an intro it will be beneficial for you to understand what we were charged with accomplishing. BOB needed to fulfill the following needs:

• A technology demonstrator
• A reference implementation for a Service Infrastructure Reference Architecture; the engineers view
• A reference implementation for a Service Reference Architecture; the developers view
• A reference implementation for training purposes for both engineers and developers
• A set of use cases for vendor evaluations
• An easy way to flush out opinions and ideas

As one might expect, when a group of smart people get together, ideas and opinions are plenty. It was this simple fact that led us down the path of a contrived sample, rather than to leverage an existing project or use a traditional scenario. If we used an existing project then there would be a very finite set of use cases with little room to explore, not to mention set time-lines and politics.

Next up in this series we will go over our set of guiding principles that we identified.

Directly from their website:

“We started High Scalability to help you build successful scalable websites. This site tries to bring together all the lore, art, science, practice, and experience of building scalable websites into one place so you can learn how to build your system with confidence. Hopefully this site will move you further and faster along the learning curve of success.”

I just discovered the website this morning, but what I have seen so far is certainly useful information. Their “Real Life Architectures” page has a list of MANY company architectures, ranging from Amazon to Google to YouTube. While you will not get any detailed diagrams, you will get a high level look at the technologies and approaches these companies have leveraged. Below is a screenshot from High Scalability, specifically describing Squarespace’s grid architecture.

Screenshot from High Scalability

In my opinion, you cannot wear the architect hat until you check out this site. I’m just saying.

“SOA is not a redefinition of modularization, object orientation, componentization. Yes, SOA builds on all three notions but it extends these constructs and thereby creates additional business and IT value. SOA is about integrating the business world with the information technology world in a way that makes both more effective and flexible using services.”

However, the article seemed to abruptly end, just as it was getting to the real meat.

In our SOA: What’s in it for me? series, we discuss how SOA can be perceived and valued by different roles: business, developer, data, and infrastructure. Which extends several of the author’s points. But if you want an elevator pitch for SOA, certainly start with something along these lines from the article:

“Business pressures are compounding as IT constraints are growing. SOA provides an approach to resolve this conflict, which stems from inefficiencies in IT environments and applications turning brittle within five years after deployment. SOA is not a panacea, true, but it is an architectural (business and IT) approach that has a huge upside business impact.”

Now that Bill and I have established some core conceptual notions around SOA we are working on providing a sample SOA implementation. This will span several posts, but we are currently working on an outline to give you an idea of the topics we will cover. Stay tuned!

“Any thoughts on communicating just exactly what SOA is to mixed audience of technical and non-technical folks, and how it will help them?”

I will assume that you have read part 1, part 2, and part 3 of this series on selling SOA to the business, developer, and data folks respectively. Without further ado, the value proposition of SOA with respect to infrastructure.

Infrastructure folks play a pivotal role in a successful SOA implementation. They generally drive selecting intermediaries in the environment, hopefully from a set of requirements from the business, developers, and data folks. In addition to  these requirements we recommend these design goals:

• Resilient to changes in message format and service implementation. Optimally no intermediary would be impacted if the message format or service implementation changed.
• Ambivalent to message body. An intermediary should provide value (some sort of capability) without requiring complete understanding of the message being passed. That is to say, an intermediary should be able to parse only the data it needs to perform a particular function and no more. The reason behind this design goal is that it should not be mandatory for the intermediary to interrogate an entire message to provide value; if this were the case the performance would be negatively impacted.
• Scale horizontally and vertically to meet changing demand. Ultimately horizontal scaling occurs because applications outgrow the hardware components (vertical scaling). One way to scale out is by merely duplicating services and leveraging intermediaries to load-balance across those services. While some may label this as poor-man’s scaling it actually has some benefits. First and foremost, it is scaling by leveraging the interoperable layer of the SOA environment, the intermediary, and not a proprietary technology.

If these design goals are not adhered to you may quickly find that your services are not easy to manage, consume, and/or discover. Let’s back up a bit and look at a conceptual scenario to prove some points.

Point-to-Point Web Services

It is very common to see many enterprises already leveraging web services today, but in a point-to-point manner. The visual below represents this straightforward design:

Traditional point-to-point Web Services

In this design, a service consumer (whether it be an web-based application, desktop application, or even another service) communicates directly with a Web service. While this may be a step in the right direction (over monolithic applications), there are certainly issues:

• Reuse: in this design certainly progress is being made against the goal of reuse, albeit on a scale short of enterprise class and purely from a technology perspective. Most reuse occurred when a buddy told you about a service they built. Even if the organization had more advanced ways to share services resistance to reuse was still there because they were not built, managed and deployed by the consuming application.  This resistance is a classic example of the “Not Invented Here” syndrome and inhibits enterprise class reuse in favor of department and/or project level reuse.
• Standards: early releases of development tools were inconsistent at best with regard to the interpretation of emerging standards, consequently interoperability was rarely achieved without due diligence. Certainly vendors needed to cater to their development environments first and competing environments second and it was evident in early offerings.
• Shared infrastructure: as more and more services came on-line, the need for shared infrastructure (“Landing Zones”) increased both in terms of shared functional capabilities (for examples: monitoring, logging, caching and storage) as well as operational needs like failover and recovery. The traditional idea that development groups build, deploy and maintain their own and only their own, applications in isolation started to fall apart. Enterprise landing zones became required for a healthy approach to reuse and became what is known today as Service-Oriented Infrastructure (SOI).
• Discoverability: as service offerings exploded so did the need for a means to discover such services both at design-time and at run-time. Often was the case, that enterprises had services that were “invisible” to potential consumers simply due to the lack of a services catalog. Or more problematic, services would, without warning, change physical locations on a network breaking its constituents.
• Message patterns: most early web services implemented a synchronous request/reply messaging pattern, as it was consistent with the traditional component calls. That is, a service consumer would call a web service and wait (sometime in a blocking fashion) for the web service response. This pattern requires that a web service must inherit the service level agreements of the consuming application. This approach became unsustainable in a large enterprise and is today tempered with alternate (but symbiotic) asynchronous messaging patterns like Pub/Sub, Topics and Paired Queues.
• Planning: as enterprise IT shops began delivering consumable services to its application groups, the dependencies among projects routinely became unmanageable. Any one service could potential become a virtual component to hundreds of applications. The logistics of managing priorities, funding, release schedules and deprecation was enormous. As a result a set of disciplines emerged to manage the enterprise as a parent to more independent development groups. Enterprise Architecture (EA) became the vehicle by which an enterprise could develop a vision, produce Business, Data, Application and Technology goals with related architectures; standards and governance and finally an execution plan with funding. This notion made independent development groups accountable to the enterprise and not the other way around. Enterprise Architecture is a very important driver of a healthy approach to SOA. It is recommended that the reader spend some time familiarizing themselves with enterprise architecture concepts, which we will cover more in future posts.

As these issues highlight, point-to-point Web services do not help infrastructure folks, rather cause more headaches.

So how does SOA help infrastructure folks?

Now let’s take a look at a SOA design and the value it adds from an infrastructure perspective.

High Level SOA design

In this conceptual diagram we added more elements to the point-to-point Web services example. The key elements being an intermediary and repository/registry. If the design goals mentioned earlier are adhered to, some significant issues in the point-to-point design are eliminated.

• Brokered service calls: the intermediary behaves as a message broker; intercepting and forwarding messages on behalf of the message originator. Usually, this type of intermediary is implemented as a router, agent, appliance or service bus.
• Provide abstraction of (service) endpoints, technology and capabilities: with this design service consumers are not tightly-coupled to the actual service endpoint (and the technology used to create the service), which not only means more flexibility of where the service lives today (and tomorrow), but also allows for common capabilities to be built into the intermediary offering.
• Shared capabilities = Reuse + Agility: the advantage to the more feature rich intermediaries are the ability to interject capabilities like logging, caching and security. Generally speaking, placing common capabilities like logging or caching in an intermediary and not each service means a consistent implementation and execution. This implies that services need not worry about logging or caching (aggregation, transformation, etc.) as any service behind the intermediary receives the behavior for “free” unless specifically required by business and/or regulatory audit requirements. Hopefully, you can see as your SOA becomes more robust, your services can be simplified and reuse increases, all working towards improved agility.
• Enhanced message pattern support: many intermediaries support protocols other than HTTP, such as JMS. Leveraging protocols such as JMS, you can implement event-driven solutions which opens the door wide open for reliable, high performing designs.
• Metadata driven: the green cylinder icon off to the right-side of the intermediary represents a repository/registry. This is another key element to this SOA design as a repository (design-time) makes it easy for potential service consumers to discover available services and intermediaries can leverage a registry (run-time) to appropriately route messages. Additional metadata is stored as well such as SLAs, policies, additional endpoints, etc.
• Dynamic scaling: as mentioned in our intermediary design goals, one way to scale out is by merely duplicating services and leveraging intermediaries to load-balance across those services. While some may label this as poor-man’s scaling it actually has some benefits. First and foremost, it is scaling by leveraging the interoperable layer of the SOA environment, the intermediary, and not a proprietary technology.

Granted, this is only a brief introduction to the value SOA brings to the infrastructure folks, but it should be enough fire power for the nay-sayers. This type of design makes monitoring and management of services much easier for the infrastructure folks, which reduces concerns that development teams often have with services. And in the end, this makes IT much more agile to support the ever evolving business needs.

Again, this four part series is really just a teaser. Bill and I spend months with clients going into much more detail and working with them to actually implement solutions with a wide scope of capabilities. As some people have already commented, we will work on an upcoming post that will walk you through a sample design and implementation of our SOA work.

“In a federated ESB, there is one “master” ESB to which several “dependent” ESBs are connected, which forms a single federated logical domain. The master ESB acts as a single point of contact for the service consumers (external to the organization). The service consumer can make requests to any business unit in an enterprise via the master ESB. It also hosts the services for governance, security and management. The master ESB can have its own repository, helping in co-ordination of dependent ESBs via routing rules stored in central management.”

Brandon and I have long been proponents of a federated approach to SOA. In that, after all SOA is about disparate systems working together based on interoperability standards not about betting the farm on a single vendor. We do however use the term hierarchical ESBs as opposed to federated and intermediary over ESB, but the premise remains the same.

It may (or may not) come as a surprise that we get quite a bit of pushback when we say we don’t really care how many ESBs (Intermediaries) you have in your enterprise so long as they meet your SOA reference architecture guidelines. In fact, in our engagements we encourage enterprises to select several SOA vendors for their pilot efforts, in addition to any they may already have in house,  to better illuminate how vendors apply industry best practices.

So my question to our loyal readers, would you be interested in a few posts on why SOA is not about a single vendor and/or how we go about conducting vendor evaluations to ferret out bad or potentially troubling SOA implementations?

“Any thoughts on communicating just exactly what SOA is to mixed audience of technical and non-technical folks, and how it will help them?”

Before I get to the problem at hand, I will assume that you have read part 1 and part 2 of this series on selling SOA to the business and developers respectively. That is to say, I am not going to focus on anything other than the value proposition of SOA with respect to data.

First, I have to admit, that I have spent many meetings back in the day as a developer, verbally sparring with Database Administrators, Architects and Analysts. Don’t get me wrong, I actually have friends who are purveyors of data by trade, who I consider well-adjusted, contributing citizens of society. Hopefully they would say the same about me, but I digress.

In my humble opinion, developers and data analysts have competing goals. Purveyors of data, hence forth referred to as Data Folks, are tasked with collecting enormous amounts of data, storing said data in safe places and doling it out as though it was chocolate at fat camp. Meaning in small fragments, that initially look fulfilling, but ultimately leave a camper unsatisfied. As I have grown longer in tooth and moved away from day to day development and all too frequent night sweats brought on by 7th normal form, I have come to realize data folks are not bad people and, in fact, are doing the right thing. As developers and application architects, we need to support them by not undermining their objectives by doing nefarious, unnatural acts with their data. I confess, I have sinned in the care of data and recited the 7 normal forms and have received forgiveness.

I know you must be thinking by now that I have gone off my rocker. Not so, as to learn from the mistakes from others (me, perhaps you too) is to not make those mistakes again. In my exceeding obtuse rant above, I was painting the picture that, as a developer or application architect,  your mission to extol the virtues of SOA to the data folks, doesn’t come with a history of trust. Let me give you a few examples after a couple of definitions.

• Record of Reference (ROR) is simply defined as a data store from which data can be consumed.
• Record of Origin (ROO) is the definitive source of data that backs the transactional system.

Frequently, ancillary systems which require read-only access of a dataset use a secondary data source, such as a data warehouse, in a way that the transaction system (ROR) is not degraded for read-only queries. The chaining of RORs is the result of those pesky developers wrestling with the Chinese finger trap implemented by our friends the data architects. Here is how it all goes wrong. Somewhere along the line a developer who consumes data from the official ROR figures out how to decode data stored in 7th normal form. Let’s say we have a purchase order which is stored in a relational database that requires a 10 table join to reconstitute into something useful for display purposes. The crafty developer then persists this purchase order in his/her personal DB to avoid repeating the reconstitution process. This, in and of itself, is not a crime if the developer imposes a policy on said purchase order that the record is only temporary and is expired in an appropriate amount of time. What can and does happen is the developer doesn’t impose an expires on policy and word gets out that the developer has a handy, flattened version of purchase orders. Through bullying and/or bribes another application developer gets access to this data which results in what is called chaining. That is a second, unauthorized ROR is spawned. The, hopefully obvious, problem here is now the two applications are tightly-coupled by way of unauthorized data sharing. Additionally, should the application that owns the unauthorized data source be EOL’d, the downstream application breaks or is forced to inherit the ROR. Now I know what your thinking. If the data folks that created the official ROR would have implemented the flattening stored procedure these sort of problems wouldn’t exist. Partially true, in that at least the owners of the data are taking responsibility to provide the data to consumers in a useful form. I say it happens, at best 50% of the time, which leads me to the next example of how developers tried to outsmart the data folks.

As developers and the folks who write developer toolsets grew tired of dealing with raw data (datasets), object-orientation gained in popularity. The basic premise was that data could be represented in a form that was understandable to end-users with associated behavior. By behavior I mean the data was encapsulated in an object which only allowed access through methods. Methods are a fancy term for logic, which operated on data as it was accessed. Developer and Data folks fought endlessly about where this logic should exist in the DB or in the objects. Ultimately, the logic ended up in both places reinforcing the ongoing feud.

Once the developer had data in object form, technologies advancements teased us into sharing objects through a variety of mechanisms like CORBA, COM, DCOM, Remoting, etc and more etc. The upside was that the objects represented an abstraction of the database such that developers were spending less time writing SQL and parsing result sets and more time flinging objects around the network. As a result the incidence or chaining RORs (DB) began to diminish but chaining of RORs (Middle Tier) increased. So we gained some abstraction from the DB but continued to tightly couple at a different level within the application stack. However, what we lost was interoperability. While most development toolsets wire quite easily to databases, development toolsets didn’t wire to one another very well. Hence the demise of CORBA.

Please don’t write to me saying object-oriented databases were the answer. I personally lived that hell and barely survived. Great concept but early offerings just didn’t do the job.

Enter SOA…

Up to this point, developers and data folks found ways to continue to violate each others’ delicate sensibilities without remorse myself included. The epiphany I had in 2001 was this, I was mad at data folks for misguided reasons. I expected the data folks to solve my problems when I was better suited to solve them then they were. Consider this, most data analysts, architects and DBAs are not programmers. Scripting sure, store procedures absolutely, but not application developers. So me and one of my developer buddies, Mark (Hey Mark!), put together some research on how the emerging notion of web services could be used to create what we called data services. The premise was that our employer would allocate some developer heads to sit with the data folks and create a service facade over the RORs for the most commonly shared data.  Web services, if you don’t already know, can expose data and behavior on the network using industry standard transports and protocols leveraging XML which got us out of brokering binary objects as we did with early OO. In concept, web services written in .NET could interoperate with JAVA consumers and visa-versa. Back in 2001, standards were emerging from all sorts of sources and interoperability was a challenge. Today it is a no-brainer. So Mark and I made the case that the data folks, if properly supplied with developer heads, could protect their data and data stores and expose their data to any/all consumers in a form that is easily understood and less tempting to instigate the chaining problems of the past. This meant that the data folks owned the creation, care and feeding of these services which was a shock to their system. But it worked, quite well thank you. Since then Brandon and I have proven it time and again, albeit with the latest sexy SOA tools.

Let’s pause a minute to digest the last paragraph. [Pause] If you forget everything else in this post, remember the last paragraph. The value proposition of SOA to data folks is not message buses and registries, but web services. More specifically it’s about data services. The data folks are chartered to protect data and serve data. The underlying ability of SOA to expose interoperable data services is how the data folks finally get to the serve part of protect and serve. Certainly the ability to advertise services in a repository/registry warrant mentioning as a feature of SOA but trust me it is only gravy. The meat on the plate is the notion of data services. That is to say, to get the attention of data folks w/r to SOA is to not sell SOA but to sell the notion of data services. Once again, SOA is “a means to an end“. SOA is an approach to realizing data services.

With that, I will end this post. Next up: Networking and Infrastructure. I think Brandon gets that one.

“Any thoughts on communicating just exactly what SOA is to mixed audience of technical and non-technical folks, and how it will help them?”

In the first post, SOA: What’s in it for me? Part 1 of 4: The Business, Bill discusses how to deal with business folks when it comes to SOA, which boils down to:

“Never sell SOA to the business outside the context of a business problem that needs solving.”

Of course Bill provides many other tasty nuggets so if you missed his post, I recommend heading over there first. In this post, I will be talking about SOA from the developer’s perspective.

We developers, absolutely love to solve problems using our own code. Many business folks, including my wife, think we can be extremely challenging to work with…and I will be the first to admit that they are correct, we often are difficult. After all, we live and breathe code, believing we can create anything and everything under the sun…yes, we developers have created the blue pill (think Matrix, not Bob Dole). Unfortunately this behavior can lead to close-mindedness. We often resist change; especially when it means writing less code and using someone else’s solution (code, tool, etc.). Don’t get me wrong, when we discover some sexy, new way way to solve our daily coding woes we are all over it! In other words, developers embrace new ideas so long as the ideas come from within. So what does this have to do with SOA?

SOA Means A Mindset Change for Developers

My partner-in-crime, Bill Draven, loves to use analogies so let me use one here to demonstrate my point. Let’s say I wanted to build a new house. If I wanted to build on a mountain-side, in the middle of no where, I would have a lot of responsibilities in addition to building a house. I would need to worry about all the infrastructure: roads, water, electricity, etc. This would obviously cost a lot of money and require a lot of additional planning consuming valuable time. This represents a completely custom solution where the developer is completely responsible for building the solution and potentially the supporting infrastructure. It should be obvious, like our analogy, this would require significant time, energy and money.

Going back to my analogy…another option I have, would be to build a custom house on a lot in a planned community. In this scenario, I could leverage existing infrastructure such as city roads, water, electricity and gas and simply concentrate on house and landscaping designs. This not only saves me time and money, but also puts my mind at ease on potential future maintenance issues with the core infrastructure. Say, if at 3:00 AM, bad weather brings down a power line, I don’t have to worry about fixing it myself or covering the cost for someone else to fix it. This analogy scenario represents a developer leveraging services (whether insider their organization or outside, aka software as a service or SaaS) to build their solution. Another way to look at this is, core features (with electricity to a lot or logging for a application) should be considered commodities. Most seasoned developers should not be excited about implementing (either by custom building or reusing their own) a logging solution. Most want to tackle new problems. To state this yet another way: most developers don’t want to deal with the science (repeatable problem patterns like security, logging, caching, etc.), rather the art (business/solution specific problems).

Once again, going back to my analogy…the final option I have would be to simply buy a cookie cutter house. In this scenario almost everything is done and decided for me, all I need to do is make some minor decisions like: what color carpet and paint, what type of appliances, and what kind of wood cabinets do I want. Out of the three scenarios discussed, this is almost always the cheapest, easiest, and fastest option. Can you guess how this relates to the developer? Yep, this analogy scenario represents a developer using an off-the-shelf product (whether commercial or open source). Most of the core features will exist, but a few configuration tweaks and possibly some additional modules/plugins can be created to build a solution.

I am certainly not naive (at least when it comes to architecture and development), there are certainly pros and cons to each of these scenarios, both as a home builder and a developer. Requirements and reason for building a solution in the first place will dictate which option is best for any given situation. However, these nuggets should be comprehended:

1. A developer should strive to spend most of his/her efforts on art not science.
2. Do not think of applications/solutions as 1 technology or 1 executable, rather a part of a distributed system and/or share services.
3. In grander terms: Do not develop solutions in isolation. There are always opportunities for reuse.

SO-A What?

Going back to my analogy, let’s say I’m building a custom home and decide that the standard door height of 6′8″ is not big enough for me, so I put in custom 7′ doors. This of course will make me very happy in the short-term, however, when it comes time to replace those doors I cannot simply go down to the local Home Depot and get a new door. I will have to go through the hassle of finding a place that can build me the custom doors which means extra money and time. Same goes for custom solutions created by developers.

Of course I am not implying that custom code is unnecessary and all developers better find new careers. Rather, the goal should be to leverage standards when possible and let requirements and existing options dictate when custom code is necessary. This is exactly where SOA comes into play. The whole notion around SOA is to leverage and implement industry standard, interoperable, reusable services (and this is not just limited to web services, other messaging patterns, like pub-sub, are included as well). Building a solution should be as easy as going to Home Depot and/or Lowes and/or Sears, picking out paint, doors, cabinets, appliances, etc. (again the science) which are standardized to work together. To make the house meet your personal tastes, pick out some custom artwork or take your own photos and hang on the walls. Likewise, the developer pieces together some reusable services (both infrastructure and code-based) and only develops custom code when necessary to achieve competitive advantage.

True, this change in mindset does not always come easy for developers. If you do come across a developer arguing that their completely custom solution is better, than ask them this:

“Do you think you can do a better job than Google (or some other leading SaaS provider) at their price point?”

To Chris’ comment: “developers rule the world, everyone is just domino effect”, the business (profit-centers) rule the world (and often carries the checkbook), so we developers need to support them in the most efficient, cheapest, and sound manner. This can no longer be achieved in isolation, the business needs change much too rapidly. Instead, the mindset must be to look after the greater good; identifying ways to build and reuse services for the enterprise whilst covertly skimming a fraction of a penny off each transaction into our off-shore accounts.

“Any thoughts on communicating just exactly what SOA is to mixed audience of technical and non-technical folks, and how it will help them?”

Yes Chris, yes we do have quite a bit to say on the subject. In a previous post we talked about SOA being a means to a end which implies that SOA is an architectural response to a bigger need or problem. The worst thing you can do as an architect is to attempt to sell SOA as SOA. The equivalent mistake would be to try to sell Model/View/Controller (MVC) or Event-Driven architecture (EDA) to a business analyst or end-user. Either they will not get it or they will not care or both.

A better approach is to present the problem at hand, describe how the problem can be solved in conceptual terms, and then and only then illustrate how SOA (assuming it does) provides the design to solve the problem.

Having said that, this is part 1 of 4 posts on how to explain SOA to an organization.

The short answer to how to explain SOA to business folks is to NOT explain SOA to business folks. Here are an analogy that hopefully makes the point.  When you go to the dentist to get a painful tooth fixed, do you want to hear how he/she is going to drill a hole in your head? Or how this wonder epoxy is much better than mercury? Well I do, but that is just me. Generally speaking though, you do not want to hear the details, rather you want your problem solved and as a bonus how fixing that tooth allows for more dessert consumption. The same thing is true for the business side of the house. They want you to solve their problems, not tell you how you intend on using a SOAP envelope to carry information in support of content-based routing. We can distill two nuggets here:

• Never sell SOA to the business outside the context of a business problem that needs solving.
• If you have to do some selling of SOA in a business problem slide deck, the slide deck should be 90% business problem and 10% SOA. Less than 10% is better than more.

Rightfully so, business problems articulated by business people are in business speak. Furthermore, the problem is, more often than not, in the form of a solution. If I, as a business analyst, said that my problem is:  I need to receive a phone call from the orders department whenever a left handed blue widget is sold, what would you do as a seasoned IT professional? Hopefully you would not offer a solution to have a direct phone line established between orders and my desk, would you? I am not going to bore you with a primer on analysis and design, but rest assured when probed further the real problem will come out. So ask me, why do I need to know when a left-handed blue widget is sold. Well I need to know because we do not keep inventory of left-handed widgets. Left-handed widgets are only stocked at our main warehouse and have to be shipped to our warehouse, then onto the customer. Consequently, I need to get a call from the Orders department so I can call the main warehouse to ship ASAP.

Are you smelling what I am cooking here? The problem has little to do with phone lines and the biz person in particular. A relatively simple answer would be to publish an event that a left-handed widget was sold such that me (the biz person) and the main warehouse get the event in near real-time. Widget gets shipped the same day and I am out of a job. More nuggets:

• Boil the problem down to its primitive parts, rarely will the problem be presented in a workable form. More likely, it will be presented as a half-baked solution.
• (Bonus Nugget) Never propose a solution that explicitly calls out staff reduction as a feature, let upper management figure that out on their own.

Moving on…Because you are a smart architect you go in search of similar problems that fit this pattern:

1. Event Happen
2. Event detected (Human intervention with latency)
3. Event Received (Human intervention with latency)
4. Re-Raise Event (Human intervention with latency)
5. Event Received (Human intervention with latency)
6. Event Processed (Human intervention with latency)
7. Event Closed

So what is the problem? Fragile system riddled with human intervention and unnecessary latency. Aka “Human Glue”. If you are in a business of any size you will find instances of this pattern under every rock in the building. So what? Well once you have a problematic pattern with many representative instances you have something the business will understand, if presented in non technical terms. Notice I have not said SOA yet? Not even the ESB word! Here come some more nuggets:

• Boil the root problem down into a pattern that can be explained in simple terms.
• Find more examples of this problem pattern in the enterprise.

Without leading the witness too much, let’s say you have done your homework and came  to the conclusion that a relatively primitive event-driven architecture can streamline this pattern through the use of a pub-sub pattern. That is:

1. Event Happens
2. Event detected (Automated)
3. Event Received (Automated and simultaneously for as many receivers as necessary)
4. Event Processed (Human intervention)
5. Event Closed

This is all great, the business will love you because you found a solution pattern that can be reused over and over again for many different business cases. However, there is one problem, the Orders system is Microsoft .NET and the Fullfillment system is Java so they don’t speak to one another. Again you are smarter than the average bear and you propose implementing service facades on top of the middle tiers for the Orders and Fulfillment system such that each system will have an interoperable, standards-based interface from which messages can be sent/received. Finally, you recommend an intermediary of some sort to abstract services from one another and to thwart point to point communication. There you go, slap together a slide deck that extoles the virtues of SOA and you are a hero, right? Wrong!

Here is where propeller-heads like us go wrong. The slide deck should take the business through your thinking process good and bad. The first slide should present the problem as stated. Take the business through your distillation process, the underlying pattern in very simple terms. Show how the pattern repeats itself over and over again in the enterprise. Describe how the current architecture prevents disparate systems from communicating with one another but, for God’s sake, do it in one slide. Show them the pattern that solves their problem, and then and only then, give them a slide or two on SOA and/or message-based architectures.

As you are getting a standing ovation in your presentation, drop the hint that SOA is “A means to a end” for some other patterns that occur in a business such as theirs and you would be glad to develop a slide deck to present to the business and selected IT managers if they would so please.

The moral of the story here is the business doesn’t give a rip about SOA, never have, never will. You, the smart architect that you are, has to walk the business folks down the path that takes them from the problem as stated to the problem pattern, to a solution pattern, to the solution that solves the problem regardless of the technology at hand.

This is in contrast with how you sell SOA to developers, network administrators, DBAs and the like. More on these topics in future posts.

Does that help?

Now this is blasphemy on my part as I have made a nice living over the last 5 years implementing SOA. But let me say this. Every engagement we have had over the last 5 years, started as an SOA implementation and quickly became a justification exercise. What I mean is our engagements are routinely to help an enterprise who has already bought an ESB figure out why they bought it and what they are going to do with it. All to often “SOA” vendors slip under/through the doors/cracks of your favorite business and do a sexy dog-n-pony show, make a deal, drop some installers and run out the door.

Let me say this now ( and I will likely repeat this several times before the end of this blog)…

“SOA is a means to an end.”

The (hopefully obvious) problem with the above scenario is that SOA is an architecture, hence the “A” at the end. An architecture is never the goal, it is (dare I say) a means to a end. All too often vendors sell SOA as though it was the goal. Goals are business-centric and rarely technology focused. A typical business goal that may suggest SOA is the business need to share data across disparate systems in real time. The tidbits to glean from that statement are share, disparate and real-time.  Before you write a flaming response to this blog on how your favorite architecture meets these needs remember it is just an example of a business need that may be solved with some of the features attributed to SOA.

Have you noticed that I haven’t used the term ESB yet. Brandon would be so proud.

Anyway, back to the point of the blog. As I was saying, Brandon and I often find ourselves not walking in the door on day 1 and implementing SOA, rather walking in on day 1 and asking to see their Enterprise Architecture artifacts that supports the decision to buy SOA gadgetry. If, and we rarely do get said documentation, we then ask for the SOA conceptual architecture that clearly shows how the business needs are met by SOA. For the record, we have never received an SOA conceptual architecture. So, as you might expect, we come in on day 1 and begin the process of making the already made decision to “go with SOA” look like it was based on sound requirements that support business goals. Following that, we then move onto developing a conceptual architecture which is used to retroactively evaluate the in-house tools. This is where we tend to alienate a few folks who would prefer to not have egg on their face for purchasing an SOA solution without doing a proper vendor evaluation against a conceptual architecture. In other words, there is no justification for SOA unless your efforts around Enterprise Architecture (EA) justify it. More on EA in future blogs. Hint, hint Brandon!

Stop laughing, this happens all the time.  It is important to note that we have been in situations where the already purchased tool does not meet the business need without what we call “unnatural acts”. We document those facts, show what nefarious things would have to be done to make the toolset work and either do those nefarious things or punt. As a contractor, never violate the laws of nature unless the client fully understands what they are getting into.

Moral of this blog…. “SOA is a means to an end, nothing more”

Let’s say I am talking to a co-worker about some of the beautiful national parks in the United States. When we start talking about visiting them I immediately tell him he needs to rent an RV. I tell him this without knowing if he is married, has kids, pets, or anything about his vacation habits (requirements). I don’t know if he prefers to fly or drive or even if he has a drivers license. The fact is, I don’t know enough about his needs, likes nor constraints to recommend anything to him.

Similarly, it is not a good idea to talk about ESBs (or similar) when talking about SOA designs. ESBs (like RVs) do vary greatly, but they both have stereotypical traits, which could lead to confusion and ultimately a poor vacation implementation. Instead, we recommend using the term intermediaries. ESBs are to intermediaries as vehicles are to transportation. That is, an intermediary represents a class of solutions that assist in realizing an SOA. A vehicle provides transportation to a destination. More specifically, we define intermediaries as such:

Any physical entity that intercepts a message, performs a value-added function(s) and then forwards the message to the intended target.

So unlike the term ESB, an intermediary can refer to not only message/service bus, but workflow engine, router, accelerator, or an appliance. This may not seem like a big deal, but stop and think about the confusions that could occur. True, this may seem like a very subtle point, but as our experience has proven to us, does make a BIG difference.

Why give an answer before you fully understand the question?

Back in 2004 we learned this lesson rather quickly. We were evangelizing our SOA conceptual design to a company. When we started talking about ESBs we could see some feathers being ruffled. After this happened several times we realized that some individuals had preconceived notions on ESBs. Or simply had unwavering alignments with some specific ESB vendors. True, the ESB market has come a long ways in the last four years, but that does not change the fact that there are many use cases that do not require an ESB yet have a very mature SOA design.

Bottom line: we found it much less contentious to speak in generic terms, which really makes sense when talking about conceptual architectures (versus physical or implementation architectures). This has really helped us in our SOA efforts with clients, so feel it is important to share with the community.

Furthermore, SOA  can be over-used, over-hyped and misapplied. And in some cases, SOA should be deferred until other enterprise problems can be solved.  But I will leave that topic for Bill to cover next.