How Are Services, Data And Processes Tied Up in an SOA?

After reading InfoQ's report on Fred Cummins' essay about the data in an SOA, I thought the discussion could get a lot more concrete if we added an overview picture in the mix. Here's my take on how applications, services, data and processes are related.

Please ignore my absent taste for colour and drawing skills :-)

Applications and functionality is typically developed in application silos. Sometimes they come as Commercial Off The Shelf (COTS) products, sometimes they are custom build applications. Fred touches on an interesting topic with

"
Those who focus on data management have, for decades, driven the industry toward consolidation of databases under a philosophy that tighter coupling means greater efficiency and consistency.
"

Since the increased focus on integration technologies and SOA, there is a common misconception that tight coupling is bad. For integrating disparate systems, that is typically true. But inside of application silo's tight coupling is typically a good thing.

An application silo usually manages data and it includes support for the related business processes. These processes can be implemented by an embeddable BPMS or they can be just as well coded in a programming language. These processes can have all the characteristics of processes implemented on top of the ESB in e.g. BPEL. They are long running and transactional, they can include human tasks as well as service invocations to services on the bus and so on. Application silo processes are only distinct from the processes on top of the ESB in the sense that they are an integral part of the application silo and therefore they are hosted in a different environment like e.g. Java instead of XML/WSDL. That is where the value of the jPDL process language comes in. It's really embeddable and it integrates very nice with any Java environment. That's why jPDL is embedded a lot in Java based portals, Enterprise Content Management (ECM) systems and custom Java applications. It's really fit for this tight coupled Java environment.

Each of these application silo's typically has a functional interface that exposes the operations supported by the application. A User Interface (UI) can then expose the functional interface to users and the same functional interface can also be exposed to the Enterprise Service Bus (ESB) via a Service Adapter.

The main purpose of an Enterprise Service Bus is integrating disparate systems. So it's typically XML/WSDL based. If many events and service invocations are published on the ESB related to one business process, then it might make sense to track the overall process with an ESB-level process. BPEL is the ideal process language for this environment.

What not a lot of people realize is that the location of business process implementations can be freely picked. If most of the accessed functionality and data for a given process is related to one application silo, it might make much more sense to implement it inside of that application silo. Still in that scenario, services can be consumed from other systems over the ESB.

On the other hand, if the process must integrate with many disparate systems, most of the events are already published on the ESB and there is no application to which this process clearly belongs, then it's most likely easier to implement it as a BPEL process on top of the ESB.

The main point I have tried to make is that the implementation of Business Processes should not be tied to integration technologies (read: the ESB). They can be just as well located within the application silos.

This also exposes one of the difficulties in mapping the analyst's business processes to implementation level processes. Analysts draw boxes and arrows in a diagram and they are not really concerned with this architectural background that provides the environment for implementing those processes. So an analysis process still has to be mapped to executable process languages in application silo's or on top of the ESB. This is not always a one-to-one mapping.

Task Assignment And Pools Harder Then You Think

Workflows and business processes are often perceived as simple in the sense that non technical people can draw diagrams that can be executed. John Reynolds explain a typical example of task assignment in Getting tasks to the right participants - Part 1:

“When a Loan Application comes in, make the application available to all of our Credit Officers who have sufficient Lending Authority to process the loan, and who haven’t already met their quota of applications for the week. Of course if everybody has already met their quotas for the week, then let everyone have an opportunity to work on the new application.

One more thing, only a few of our Credit Officers can approve loans over $100,000, so don’t route low value loans to them unless they are the only available credit officers.

And don’t let anyone claim more than 3 applications at a time. Our Credit Officers are compensated for the number of loans that they process, and some of them will claim all the applications that they can in hopes of earning more money.”

This really sounds like a realistic task assignment to me. My point here is that it's impossible to capture this kind of logic in the business process diagram. It reinforces the message about process development that I've been giving for a long time. Analysts provide a description of a business process. Typically that is a diagram plus free text documentation like e.g. this kind of task assignment rules. Then it is up to the process developer to come up with an executable process that looks exactly like the business analyst's diagram, but this time it is an executable process. Iterative development can then be realised on the executable process. I don't believe that round tripping between analyst and implementation models is feasible for the majority of BPM use cases, especially if those turn out to be very different.

The second post that I refer to is Bruce Silver's BPMN to Requester: Get Outta My Pool. In that post, Bruce points out that BPMN swimlanes are in fact inspired by BPEL's partner links and that experienced business analyst have a hard time understanding those basics. In fact, I think the business analysts are right. Analyst models should not be constrained with executable details. If we want to create a notation that can serve analysts and process developers, then it should be nothing more but boxes and arrows. All the extra semantics that are added to diagrams only are targetted to either the analysts or to the developer. But those extra semantical decorations and concepts never make (the same) sense to both target audiences.

This really shows the scheziphrenic nature of BPMN. On the one hand BPMN aims to be a free modelling notation for business analysts. And on the other hand, it tries to map one on one to an executable service orchestration language like BPEL.

Brewing The jBPM Community

Our managers were a bit concerned at first. "Are you sure it's a good idea to have an IT event in a beer factory?". Of course it is...

Last Friday, we had the jBPM Community Day. A rememberable day it turned out to be. We assembled the full core of the jBPM contributors from JBoss and other companies at the Guinness storehouse in Dublin. More then 40 people showed up, of which half of them flew in specially for this event.

The three muscateers arrived. The first thing they noticed was this thing that looked like a spaceship mounted on top of the beer factory.

Finding the way to the room took some concentration.
But most of us made it before the Master of Ceremony kicked it off.
We gave most of the platform for talks to core members of the community. That turned out to be a bulls eye.

Joram Barrez started addressed any performance question that people in the audience might wonder about. Also he showed the value of how easy the Business Intelligence information could be extracted from jBPM process executions.

Next, Mark Proctor managed to sneek in a Drools-spy into our jBPM event.
Paul Browne talked about the situations where drools rules and jPDL processes make a perfect match. Both Joram and Paul are great speakers and they know what they talk about. Thanks a lot, guys!

Then it was time to do some sightseeing in the building.
And to taste the Guinness at the gravity bar (the space ship thing).
An awsome, full 360 degrees view!
After that pint of Guinness, my short talk was scheduled. Next time we need to have someone that pulls the plug when I go over my time limit again :-)

Then we splitted the audience in two separate tracks. The core contributors went to discuss the future of the project.
Ronald (aka Kukeltje) showed how to hold a Guinness so that your neighbour doesn't see it and with the other hand answering forum questions at the same time.

On a more serious note: this is where the most constructive feedback was generated. We had a great brainstorm session on how we can improve the migration path from jPDL 3 to the upcoming jPDL 4.

At the same time, the other people users got an introduction to jPDL from Koen Aers, the creator of the jPDL eclipse plugin designer.

The reception dinner closed a fruitful day.
Then we went down to the temple bar area and the Master of Ceremony found this great pub with live music.

But with strickt rules !
The 2BInternational team met with the locals and showed us that they had more moves then only in Business Process Management.

But then things started to get messy.
So I concluded it was time to stop taking pictures :-)

Oh... and there was a memorable close of this memorable evening. When walking to the hotel, I came across this die hard artist still out on the street doing a perfect inpersonation of Mark Proctor. Awsome!
Amen.