Business rule engines have been around for quite a while now, but their utility in common business software solutions has been limited at best. This is about to change.
First, a Story
My first contact with a business rule engine was at Compaq in the winter of 2000. I was given the task of helping to maintain the Compaq Configurator, one of the first web-based server configuration systems. Fresh out of college, I was ecstatic to have a job writing code for a website that had a huge set of real human users.
For the times, Compaq was pretty forward thinking. They were among the first in a set of computer manufacturing companies to field a robust web-based tool that allowed sellers to put together and price complex server orders in more than 50 countries. This was all accomplished using a forward chaining rules engine. By any definition this was transformative for the time. It allowed Compaq’s partner sellers to self-serve instead of calling sales support specialists to get help. It completely removed the human bottleneck in the sales pipeline and it did it with more accuracy at a lower cost than humans ever could have achieved alone. It automated complex decision-making tasks and replaced human experts for pennies on the dollar.
The New Normal
Let’s be clear that this was an uncommon project for its time. We did this at a time when most of IT was concerned with making html versions of paper forms, throwing in a bit of workflow and reporting on it. Fast-forward to almost 2017 and we are at the point where we have more connected devices than people on the planet producing information on a scale that is no longer something humans are capable of making sense of without lots of machine help.
This machine help was expensive in 2000 but AI and machine learning have been packaged and comoditized to the degree that anyone with a small credit card budget can produce predictive results across their data.
This is the new normal. We are creating systems more often that make the decisions humans used to be in charge of. Not only are they drawing insightful conclusions, they are acting on them. What’s remarkable is that we are trusting these systems more and more with ever more important tasks – in some cases our very lives. From self-driving cars to autonomous drilling rigs, this has changed the expectations about what software should do.
Machine Learning is for Squishy Problems
While the internet is buzzing with excitement about big data and machine learning, it is important to note that these tools are best put into practice to solve squishy, non-deterministic problems; problems with more than one potential solution, where good enough answers beat perfect answers any day. As we transform our businesses into digital versions of themselves, we are finding lots of these squishy non-deterministic problems to solve. We are also uncovering more and more business automation problems with deterministic solutions that are not straightforward to implement but also involve high volumes of streaming data.
While solution approaches for the former type of problem rely on tools which evolve algorithms using probabilistic methods, the latter type of problem requires solutions that feature stochastic deterministic algorithms. In the case of the latter, the implementation issue that we face is then how to represent the often large set of rules which form the solution in a way which can be readily organized and reasoned about. Most traditional programming approaches, object oriented, set based languages, and functional approaches make a spaghetti mess out of a complex network of business rules. As these systems grow, more and more effort is required to maintain and organize the logic without introducing duplication and breaking change. While tools are steadily evolving to allow us to examine our gargantuan data sets using SQL-like set based languages, these languages have none of the properties that allow us to reason about large sets of rules and achieve high organization of those rules.
The Classical BRE has warts
Enter the business rule engine. While BRE tools have been around for some time, the fact that we are finding more and more automation problems whose solutions lie in complex sets of deterministic rules may in fact breathe new life into the industry. Some changes will be required though. These are the problems that plague the vast majority of rule engines on the market today.
1) They cater to a user audience that largely does not exist:
In its most classical form, the motivation for employing a BRE was primarily to solve the problem of allowing business users to express rules and put them into production systems without developer intervention. On paper this is great. In practice, this was rarely the case. In fact in most cases, rule authorship required either IT personnel who were well versed in logical programming languages or a special breed of trained business users, ‘knowledge engineers’ that could learn the specific ins and outs of the rule engine in use to produce logically consistent versions of the business reasoning behind a problem set.
The challenge of making a basic forward chaining rule engine isn’t a massively expensive problem. A vast amount of engineering effort (and expense) is required however to make those engines compatible with rule authoring approaches that are business user friendly. In most cases, the solution is to create new pseudo-programming languages which struggle to be logically both complete and simultaneously friendly for non-programmers.
2) The deployment model is heavy:
Many rule engine tools have subscribed to the ‘rule engine as an enterprise service’ pattern where a single scaled enterprise instance of the engine is put into place to be shared across the enterprise. This works in a lot of scenarios but prevents deep integration into the kinds of distributed, embedded processing and streaming problems we are seeing more of.
3) They are expensive:
It is rare that I have been able to compel customers to invest the requisite amount of money to procure an enterprise-class rule engine. It is usually the case that we need the tool for a set of very specific use cases and the client has a hard time justifying the expense for a tool that they see the ability to use once. Several affordable and supported options exist within the open source space. Attitudes towards open source are changing, but especially in the Microsoft platform world, the options have been, and still are limited.
Big Data, Rules, and The Shape of Things to Come
In its modern form, I assert that BREs may be required to prioritize programmer friendliness, rule organization features, and the ability to be applied in the context of big data systems to be relevant. The end result may be lightweight runtimes which can be deployed on each node of a massively scaled data processing cluster and participate in the map reduce pattern of large scale data processing echoing the theme of bringing the algorithms to the data instead of the traditional approach of bringing the data to a centrally hosted algorithm. This flies in the face of the singly installed enterprise ‘rules as a service’ mentality that the rule engine market has been following for years.
aaron murrell 