Yahoo Pipes, PVM and language-oriented programming

Paul Gallagher's blog "Pipes - Web 2.0 Wake-up Call for BPEL?" gives a very example nicely what we call a graph based execution language. And the best part is that he puts the Yahoo Pipes langauge in perspective with BPEL, paving the way for me to explain why its crucial for future workflow engines like our JBoss jBPM to support multiple process languages.

The only languages often associated with workflow, BPM and orchestration are BPEL, XPDL and BPMN. Of those, only BPEL defines exact execution semantics and hence it is called an executable process language. jPDL is another executable process langauge. Where BPEL is targetted to a web services / ESB environment, jPDL is targetted at a standard Java environment.

Similar to Java being referred to as a general purpose programming language, I refer to BPEL and jPDL as a general purpose process languages. But over the last years, with the more widespread adoption of jPDL, we get a lot of requests to create limited dialects of jPDL for specific purposes. For example approvals in a document management system or tracing of laboratorium experiments.

JBoss jBPM is designed as a platform for process languages. After we had put BPEL, jPDL and SEAM pageflow on one base framework called The Process Virtual Machine, we realized that we could build all these kind of languages on the same base technology. The componentization that we introduced supports this very well already.

As such, Yahoo Pipes are another nice example of a graphical process langauge.

Martin Fowler and Neal Ford recently introduced the notion of Language-oriented programming. The idea is to "build software around a set of domain specific languages”.
The more we realize how big the category of graph based execution languages actually is, the more excited that we get. With our new Process Virtual Machine technology underway, we are not far from building out a complete language workbench focussed only on graph based execution languages.

More concrete, based on the Process Virtual Machine we're building a complete software development kit for developing your own graph based execution language, including runtime, console and designer. You'll be able to code the runtime behaviour of process constructs in plain Java. Also included will be a mechanism to add designer forms, XML serialization and persistence of the process constructs. We see a lot of use cases for easy customization of general purpose process languages like jPDL.

Typically I don't really know how to start explaining the concept of how jBPM is becoming a language workbench for graph based execution languages. Thanks, Paul for providing this example and giving me an opportunity to explain the strategic direction of JBoss jBPM.

jBPM jPDL 3.2.2 released

jPDL is a simple workflow language targetted for the Java platform. We just released version 3.2.2, which fixes a number of bugs and minor (but crucial) enhancements. The detailed list of issues can be found in the jira changelog.

The suite and the runtime can be downloaded from sourceforge. Enjoy!

Great Book About JBoss jBPM

Last weekend I finally came around to reading "Business Process Management with JBoss jBPM". A brandnew book written by Matt Cumberlidge. It fills a gap in the jBPM offerings. Another milestone for the project.

The book targets architects, analysts and developers that want to get started with jBPM. Matt succeeded in bringing great value for all three target groups. It starts with a fresh look at the value of BPM and practical guidelines on how to start a BPM project. Installation of every required piece of software is explained in detail. Even less technical people will have no problems to get the software up and running with these very simple step by step instructions.

The ordering of chapters captures the full jBPM learning curve. After installation, a prototype business process is build in two iterations. Then, the more technical details are explained to transition from prototype to implementation. And finally a chapter on process improvement explains how the processes can be measured and analysed to find optimizations.

jBPM beginners can use this book as an easy to read tutorial to get jumpstarted with jBPM. The available documentation in jBPM itself is sometimes a bit too concise and basics are on the same level as advanced topics. Matt made a good distinction between the basics and advanced topics. The basics are explained in detail and the advanced topics are explained more briefly and referenced. It will lower the treshold for the learning jBPM significantly.

Also, I really like the structure of the book. It adds another dimension. Throughout the book, the link is made with the overall software development methodology and how BPM fits into that picture. It shows the value of BPM from the developer's point of view, which is much more interesting then the typical empty BPM promises on a management level.

Congratulations, Matt! Well done.

You can get the book at Amazon or directly at Packt publishing.

jPDL Webinar With Awsome Demo

Your last chance to register for the jPDL webinar that will include an introduction by me and a long demo by Micah Modell.

It's next Wednesday 11am EDT. Your boss is probably on holiday's while you're working hard. So it's not more then fair if you take the liberty to listen in on this cool webinar. On top fo that, you'll be impressing your boss when he gets back as you'll know the meaning behind the workflow buzzwords.

Don't miss it and register here.

Try out the new jBPM release 3.2.1 of jPDL

Ever tried a simple Java workflow engine ? This is the time to do it! Last week, we released jPDL 3.2.1. jPDL is the process language for writing statemachines in Java. jPDL is build in such a way that it supports Business Process Management (BPM), workflow and orchestration in standard as well as enterprise Java environments.

Download it from sourceforge and and check it out.

If you download and unzip the suite package, you get the full documentation, a graphical designer plugin for eclipse, the runtime engine preconfigured in a JBoss app server (jPDL also runs without an app server or on any other app server), and a web console. All preconfigured so that you don't have a hassle to get started and evaluate the process language.

Is BPMN The Solution For All Process Languages ?

I would like to start this freely available answer with "It depends...", the typical answer of expensive consultants :)

BPMN is clearly a great language for non tech analysts. But the approach that BPMN has to map BPMN to all the executable process languages is sometimes misinterpreted. The Process Virtual Machine describes how multiple process languages can be supported by one single runtime engine.

The text below started off as an explanation of how we can replicate that approach in a graphical process designer. But it also contains a no-so-traditional-but-nonetheless-realistic view on how business analysts and developers can jointly work together on the automation of business processes.

A component model for nodes
On the tooling end, the base framework should be able to display nodes and transitions. Like e.g. in BPMN, but ignore the decorations and the properties for a minute.

Then a process construct should be a plugin. The plugin contributes the following to the base plugin:

- shape: the tool could predefine the 4 BPMN shapes, but a node-plugin should be able to self-define it's shape with its own figure.

- configuration: the plugin should contribute a properties form to enter the configuration information for that node type. to enable this, the internal model of the process graph must have a dynamic set of properties.

- constraints: the plugin should be able to specify constraints like: this type of node can only have 1 outgoing transition. or: outgoing transitions from this node type can only be connected with node type z.

- decorations: which decorations are supported. maybe this could be done with icons so that apart from the BPMN decorations, node implementors can supply a graphical colourful icon.

The whole idea is that you should separate all the node type specifics from the basic process designer container.

Guidance for process languages
BPMN recognizes multiple process languages. But it has suggested a problematic approach to handle that.

BPMN defines a mechanism of bidirectional mappings from BPMN to executable process languages. This suggests that you could model in BPMN and then translate to any executable language. IMO, that is a unidirectional translation.

When an (non-tech) analyst starts to model a process, this has to be done in free modelling language like BPMN, visio or IDS Scheer's ARIS notation. Of course, those models only contain graphical information intended for human-to-human communication and they are not executable.

Executable processes exists of graphical structure and technical details to make a process executable. The graphical picture is the common language between analysts and developers. An executable process is human to system communication in the sense that it specifies to the computer system what it has to do.

The translation from a modelling process (graph only) to an executable process (graph and tech details) is a big one. First of all, the analyst may have modelled steps in the model that are not to be automated by the computer system. Second, the developer makes a selection as to which process language best fits his technical environment. It will most likely not be possible to keep the original process model as-is for the executable process because of the executable language specific constraints.

After the translation to an executable process, analyst and developers have a common language in the graphical part of the executable process. But now, the analyst lost his freedom to change anything he wants since that implies software changes.

This is why I come to the following conclusion: A process designer tool should support each process language individually. BPMN is one of those languages/notation. This is the free modelling tool. Then BPMN diagram can be converted (1 time translation) to executable process languages like BPEL, XPDL and jPDL. This translation should generate a new file.

For modelling an XPDL process for instance, the designer tool should present itself as a straight XPDL editor. All the parts that XPDL specifies should be exposed by the tool with their proper names (properties and node types). But where XPDL is undefined (like in the graphical notation), that is where BPMN can be used to complement. Same story for other executable process languages.

There is a big change going on lately

"There is a big change going on lately. A year ago, all BPM and workflow was centered around BPEL and around web service orchestration. And that was the way to go, that was the single process language that's going to make it. At that time we were already long time working on our multi-language approach."

After noisy attempts at the moscone, inbetween ringing phones of the labs crew, an eager housecleaning lady and an apologising hotel manager we managed to shoot an interview in San Fransisco at JavaOne this year.

New JBoss User Group Starts in Rotterdam

A brandnew JBUG Benelux is launched by Lunatech in Rotterdam, The Netherlands next Friday, June 8th. I'll be speaking about jPDL, but more importantly, there will be pizza's ! It will also be a great way to meet new collegue nerd developers or see how others use JBoss. Check out all the details on Peter Hilton's blog and drop him an email if you intent to come. I hope to meet you there.

Therapy for BPM

To start the therapy for the Business Process Management (BPM) identity crisis, we'll have to go back to its childhood. Why did we start messing with process languages in the first place ? Ah, yes! While all tools can draw process diagrams, the real reason was that we couldn't express process languages in traditional programming languages because they represent a sequence of instructions for the computer system. They don't have support for wait states.

To further expose the pathological state of BPM, I always use the DB metaphore: When I say relational DB, most of this audience thinks tables, columns, primary keys, foreign key references, SQL and so on. Every self respecting developer knows the concepts of this technology. For a technology that is a very sane state to be in. But in BPM, nobody knows the the concepts of state machines that can be used for business processes or what it means to execute those. Every engine has got its own execution model that looks like, but is not completely the same as the next engine. Yet we are creating process language proposals in abundance. It's like proposing query languages without knowing the difference between relational-, object-, hierarchical and object databases.

Can't be that difficult, right ? Afterall, it's just a state machine that we're adding to all programming language capabilities, no ?

Well, not quite so simple. State machines are used in a variety of ways. Different features and functions are added to improve support for those features. Also, there are different environments. E.g. a a service orchestration language like BPEL looks very different from a Java workflow language like jPDL. That creates a whole lot of combinations for which you can build dedicated process languages.

For therapy, the roots are state machines. All of those process languages have some form of state machine in there. That is because traditional programming doesn't have support for wait states (see many of my earlier blog posts).

Now let's reflect on this in real zen style. Close your eyes, but keep reading. Many flavours of process languages... many different features... but all represent some form of state machine.

After one of those sessions, it struck me that I should first learn how to walk before trying to run. Learning about the state machine basics is much more effective then trying to understand all the flavours of the different process languages. And once you get the state machine basics, it will be much easier to learn a new specific flavour of a process language if you need one.

Since I couldn't find good state machine concepts and terminology in the context of executable software, I just wrote it myself. I called it The Process Virtual Machine and a summary of that full article has been published at OnJava.

The Process Virtual Machine was my therapy for BPM. Every developer and BPM-er should read it and have an opinion on it. Feel free to let go and just share it as a comment on this post.

Workflow with jPDL at JavaOne

I'll be at JavaOne to talk about jPDL, the workflow and BPM language for the Java platform. The talk will be on Tuesday from at 3:20 PM in Esplanade 304/306. You don't want to miss it so put it in your agenda.

I'm always interested to meet new people. Ping me at tom dot baeyens at jboss dot com in case you want have a chat while I'm in SFO.

The Schizophrenic Nature Of BPM

Business Process Management (BPM) has always had this disturbing identity crisis. Here's an interesting discussion between industry titans in which several symptoms can be seen:

Keith Swenson: The tipping point for XPDL (see also the comments)
Keith Swenson: Are apples more useful than oranges
Bruce Silver: The real issues with XPDL, BPEL and BPMN
Keith Swenson: The diagram is the meaning
Bruce Silver: Diagrams, models, and Metamodels... Oh My!

I want to put this discussion into a perspective so that you can see why I think they don't speak the same language. This post will try to establish a realistic view on how process modelling and executability can be combined.

On the one hand, BPM is all about letting non-technical business analysts create diagram models in all freedom describing how people and systems work together. On the other hand, the promise of BPM is that you can drop these models into a BPM System and a piece of working software is the magical result. This duality is already indicated by Phil Ayres in the comments: "...and very simply put, both standards live up to their names: XPDL - Process definition language; BPEL - process execution language." According to me, that is just an indication of the fundamental schizofrenic state BPM is in currently.

If you think about it, these two goals are in fact very distinct; Business Process Analysis (BPA) means creating a description of how people and systems work together. It's describing what is observed. On the other hand, creating an executable process means specifying how the BPM engine should behave to support this process. This comes down to specifying what a computer system should do and in that respect, it is no different from traditional programming. It needs the same kind of deterministic accuracy and completeness. The language is different, but the exact behaviour of a software system is defined in an executable business process.

There is only 1 kind of product in the BPM industry that can give complete modelling freedom to the business analyst. Those are the drawing tools that only have a graphical notation, but no runtime engine part. Think products like IDS Scheer's ARIS suite and Visio. But the bulk of the products in the BPM space have a runtime engine part.

Most of these products with a runtime engine component have suffered badly from the schizofrenic nature of BPM. They try to hide the technical part as much as possible, while the remainder of this post will argue that a technical aspect is an intrinsic part of executable process languages and their process engines.

We have been telling it for quite a while now: The graphical model of a process should be separated conceptually from the technical details. An executable process is most often a combination of the two. The following picture shows how we envision the process as a combination of the graphical information and the technical details that make it executable.


Non technical business analysts can look at the graphical part only. Developers will manage the technical details so that the whole business process becomes executable on a BPM engine. In practice, the line between analyst and developer is a bit more blurry. Depending on their technical skills, analysts might already provide some technical details they know. But for the sake of this post, I want to highlight the dual nature of BPM and that's why I put analysts and developers in very distinct roles.

So this results in a situation where the business people can understand a view upon the software artifact that could represent a business process. At the same time, that process actually runs on the BPM engine in production. We also think it's very likely that business analysts come up with the initial diagram for the process to be developed. But we don't believe that in general, non technical people will be able to write both parts of the process. That would imply that non technical people would specify the mission critical parts of your enterprise's IT infrastructure. This thought is as ridiculous as it is scary. Regrettably, it is still preached by many of the BPM vendors today.

This point of view clearly shows that there is a technical aspect to making business processes executable. A good BPM engine should *not* try to make the technical people redundant (which is an agreeable but unreachable goal) , but instead it should facilitate and ease the collaboration between business analysts and technical people.

In software development, Domain Specific Languages (DSL) are described as limited computer languages for a specific purpose. Todays software is full of such languages that have to be combined with a framework. Think any XML language in a typical Java development project: ORM mapping, IoC configuration information, deployment descriptors, IoC bean wiring XML files. Looking at the technical aspect of BPM, an executable process language is just another DSL. Therefor, BPM engines should fit right into todays architectures of software development. And this should not be interpreted as "oh,... it's just a library for developers without any focus on the 'Business' in BPM". On the contrary.

Also in the engines, a technical aspect is often overlooked. A process engine as a monolithic black box is already useful in some scenarios. But a process engine becomes usable in a much broader set use cases when it can be easily embedded into an application. How easy can the workflow engine tables be integrated in my application database ? How easy can test scenarios for processes be written and executed ? How much library dependencies does the engine have ? How easy can the transactions of the workflow engine be embedded in my own application transactions ? All these questions become very relevant when you want to deploy a process engine other then in the black box mode.

The conclusion is that technical aspects of workflow and BPM engines do matter. It's not because business people are involved somewhere in the creation of processes that process languages or process engines cannot have a technical side. When product vendors try to hide those technical aspects, they end up with schizofrenic Frankenstein monsters. So beware. Cause you don't want such a creature to end up in the core of your IT infrastructure, now do you ?

BPEL Compared To jPDL

Michael Neale of the drools team is giving a Talk on JBoss jBPM for the Canberra User Group. He mentioned my <sarcasm>*looooove*</sarcasm> for BPEL and asked me the pitch. In my reply, I explained that I don't think BPEL is bad, but that you have to know when to use it and when other alternatives are more appropriate. I think my reply would be interesting for more people, so I share it here.

BPEL is a process language designed to work on in a web services environment because it is based on WSDL. WSDL could potentially map to java beans as well, but it naturally maps to web services, while WSDL doesn't map naturally to java beans.

Deploying a BPEL process leads to publishing of a web service (a WSDL service to be exact), that is your interface to the process. Internally in a bpel process, the process variables are XML snippets (or xsd basic types). BPEL has some control flow constructs and a construct to invoke a WSDL service. So in essense, BPEL is in fact a language to script a new web service as a function of other web services.

BPEL has a receive construct, which means wait here until this message is received over the published service. This implies that the overall execution behaves as a wait state. Consequently, long running (spanning multiple server side transactions with long periods inbetween) processes can be expressed in BPEL.

Expressing long running processes (== support for wait states) is a requirement to model typical business processes in a language. Business processes like handling an insurance claim take a long time and involve several requests to/transactions on the server. So to support business processes, the runtime engine needs to behave as a kind of state machine. While BPEL has the wait state feature, which is one of the requirements for executing business processes, the assumption of an all-web-services-world makes it hard to integrate BPEL based business processes with every day software development in e.g. Java.

The thing that nobody seems to understand is that you can build state machines in Java as well. And typically much more simple then in web services technologies. That is where jPDL fits in. In jPDL the interface is a Java API and the process variables are POJO's. So this language embeds far better in a Java project.

Conclusion is not that BPEL is bad. The conclusion is that you need to select the process language that fits with the environment in which you are working and with the requirements you have. Use BPEL if you want to script new web services as a function of other web services in a mostly webservices environment. Use jPDL if you need statemachine capabilities in a Java environment. In both technologies it is possible to make the bridge to the other side, but that is of course requires some mappings and translations. So it is all a matter of selecting your process language based on the main development environment.

My Previous Blog

My previous blog can be found at the JBoss Matrix.