This is Part 3 of 4 posts in response to a question from a former cohort, Chris, from the old days at a giant semiconductor company:
“Any thoughts on communicating just exactly what SOA is to mixed audience of technical and non-technical folks, and how it will help them?”
Before I get to the problem at hand, I will assume that you have read part 1 and part 2 of this series on selling SOA to the business and developers respectively. That is to say, I am not going to focus on anything other than the value proposition of SOA with respect to data.
First, I have to admit, that I have spent many meetings back in the day as a developer, verbally sparring with Database Administrators, Architects and Analysts. Don’t get me wrong, I actually have friends who are purveyors of data by trade, who I consider well-adjusted, contributing citizens of society. Hopefully they would say the same about me, but I digress.
In my humble opinion, developers and data analysts have competing goals. Purveyors of data, hence forth referred to as Data Folks, are tasked with collecting enormous amounts of data, storing said data in safe places and doling it out as though it was chocolate at fat camp. Meaning in small fragments, that initially look fulfilling, but ultimately leave a camper unsatisfied. As I have grown longer in tooth and moved away from day to day development and all too frequent night sweats brought on by 7th normal form, I have come to realize data folks are not bad people and, in fact, are doing the right thing. As developers and application architects, we need to support them by not undermining their objectives by doing nefarious, unnatural acts with their data. I confess, I have sinned in the care of data and recited the 7 normal forms and have received forgiveness.
I know you must be thinking by now that I have gone off my rocker. Not so, as to learn from the mistakes from others (me, perhaps you too) is to not make those mistakes again. In my exceeding obtuse rant above, I was painting the picture that, as a developer or application architect, your mission to extol the virtues of SOA to the data folks, doesn’t come with a history of trust. Let me give you a few examples after a couple of definitions.
- Record of Reference (ROR) is simply defined as a data store from which data can be consumed.
- Record of Origin (ROO) is the definitive source of data that backs the transactional system.
Frequently, ancillary systems which require read-only access of a dataset use a secondary data source, such as a data warehouse, in a way that the transaction system (ROR) is not degraded for read-only queries. The chaining of RORs is the result of those pesky developers wrestling with the Chinese finger trap implemented by our friends the data architects. Here is how it all goes wrong. Somewhere along the line a developer who consumes data from the official ROR figures out how to decode data stored in 7th normal form. Let’s say we have a purchase order which is stored in a relational database that requires a 10 table join to reconstitute into something useful for display purposes. The crafty developer then persists this purchase order in his/her personal DB to avoid repeating the reconstitution process. This, in and of itself, is not a crime if the developer imposes a policy on said purchase order that the record is only temporary and is expired in an appropriate amount of time. What can and does happen is the developer doesn’t impose an expires on policy and word gets out that the developer has a handy, flattened version of purchase orders. Through bullying and/or bribes another application developer gets access to this data which results in what is called chaining. That is a second, unauthorized ROR is spawned. The, hopefully obvious, problem here is now the two applications are tightly-coupled by way of unauthorized data sharing. Additionally, should the application that owns the unauthorized data source be EOL’d, the downstream application breaks or is forced to inherit the ROR. Now I know what your thinking. If the data folks that created the official ROR would have implemented the flattening stored procedure these sort of problems wouldn’t exist. Partially true, in that at least the owners of the data are taking responsibility to provide the data to consumers in a useful form. I say it happens, at best 50% of the time, which leads me to the next example of how developers tried to outsmart the data folks.
As developers and the folks who write developer toolsets grew tired of dealing with raw data (datasets), object-orientation gained in popularity. The basic premise was that data could be represented in a form that was understandable to end-users with associated behavior. By behavior I mean the data was encapsulated in an object which only allowed access through methods. Methods are a fancy term for logic, which operated on data as it was accessed. Developer and Data folks fought endlessly about where this logic should exist in the DB or in the objects. Ultimately, the logic ended up in both places reinforcing the ongoing feud.
Once the developer had data in object form, technologies advancements teased us into sharing objects through a variety of mechanisms like CORBA, COM, DCOM, Remoting, etc and more etc. The upside was that the objects represented an abstraction of the database such that developers were spending less time writing SQL and parsing result sets and more time flinging objects around the network. As a result the incidence or chaining RORs (DB) began to diminish but chaining of RORs (Middle Tier) increased. So we gained some abstraction from the DB but continued to tightly couple at a different level within the application stack. However, what we lost was interoperability. While most development toolsets wire quite easily to databases, development toolsets didn’t wire to one another very well. Hence the demise of CORBA.
Please don’t write to me saying object-oriented databases were the answer. I personally lived that hell and barely survived. Great concept but early offerings just didn’t do the job.
Enter SOA…
Up to this point, developers and data folks found ways to continue to violate each others’ delicate sensibilities without remorse myself included. The epiphany I had in 2001 was this, I was mad at data folks for misguided reasons. I expected the data folks to solve my problems when I was better suited to solve them then they were. Consider this, most data analysts, architects and DBAs are not programmers. Scripting sure, store procedures absolutely, but not application developers. So me and one of my developer buddies, Mark (Hey Mark!), put together some research on how the emerging notion of web services could be used to create what we called data services. The premise was that our employer would allocate some developer heads to sit with the data folks and create a service facade over the RORs for the most commonly shared data. Web services, if you don’t already know, can expose data and behavior on the network using industry standard transports and protocols leveraging XML which got us out of brokering binary objects as we did with early OO. In concept, web services written in .NET could interoperate with JAVA consumers and visa-versa. Back in 2001, standards were emerging from all sorts of sources and interoperability was a challenge. Today it is a no-brainer. So Mark and I made the case that the data folks, if properly supplied with developer heads, could protect their data and data stores and expose their data to any/all consumers in a form that is easily understood and less tempting to instigate the chaining problems of the past. This meant that the data folks owned the creation, care and feeding of these services which was a shock to their system. But it worked, quite well thank you. Since then Brandon and I have proven it time and again, albeit with the latest sexy SOA tools.
Let’s pause a minute to digest the last paragraph. [Pause] If you forget everything else in this post, remember the last paragraph. The value proposition of SOA to data folks is not message buses and registries, but web services. More specifically it’s about data services. The data folks are chartered to protect data and serve data. The underlying ability of SOA to expose interoperable data services is how the data folks finally get to the serve part of protect and serve. Certainly the ability to advertise services in a repository/registry warrant mentioning as a feature of SOA but trust me it is only gravy. The meat on the plate is the notion of data services. That is to say, to get the attention of data folks w/r to SOA is to not sell SOA but to sell the notion of data services. Once again, SOA is “a means to an end“. SOA is an approach to realizing data services.
With that, I will end this post. Next up: Networking and Infrastructure. I think Brandon gets that one.
Print This Post
Tags: Enterprise Architecture, Service Oriented Architecture, Software as a Service