SOA, Cloud Computing, and Event and Model Driven Architecture

SOA and Cloud Computing, the Predicate to Model and Event Driven Architecture

In a recent post (see Functions Required in the Cloud PaaS Layer to Support SOA), I discussed two SOA patterns and two Cloud Computing patterns and showed how they are, in fact the same patterns applied to the SOA and the Cloud Computing domains.  For SOA, the concepts are Enterprise SOA (ESOA) and Ecosystem or Internet SOA (ISOA), while for Cloud Computing, the concepts are Private and Public Clouds. The correlation of these concepts is shown in Table 1.

	Ownership Domain		Integration Type		App. Layer Comm.
SOA	1 Dom	2+ Dom	Orchest.	Choreo.	ESB	Internet
ESOA	X		X		X
ISOA		X		X		X
Cloud
Private	X		X		X
Public		X		X		X
MAEDA
Model	X		X		X
Event		X		X		X

 Table 1

There are three attributes for both SOA and Cloud Computing shown in Table 1; Where the Services operate (the Ownership Domain variable), whether the service uses orchestration or choreography (Integration Type [see SOA Orchestration and Choreography Comparison]) and where the service components interact (The Application Layer Communications).  As the table demonstrates, on the dimensions shown, an ESOA is a model of the Software as a Service (SaaS) layer of the Private Cloud, while the ISOA is a model of the SaaS of the Public Cloud.  Consequently, SOA and Cloud Computing can be the same at the SaaS layer.  Since the SaaS drives most of the requirements for the Platform as a Service (PaaS [see Functions Required in the Cloud PaaS Layer to Support SOA]) and the Infrastructure as a Service (IaaS) layers, the results should be a nearly identical Technical Enterprise Architecture (or functional design) [Sidebar: but, perhaps with different labels to satisfy the purists, perhaps, "True Believers", in each camp].

Model and Event Driven Architecture

In a Part 3 of a four part paper, The Future of Information Technology: Enterprise Architecture 2006 to 2026, Part 3: Model and Event Driven Architecture (2008 to 2017), The Journal of Enterprise Architecture (February 2007), p. 28-39, I posit Model and Event Driven Architecture as an outgrowth of SOA.  This is an integration of Model Driven Architecture as espoused by the Object Management Group and a widely discussed, but somewhat nebulous Event Driven Architecture.
Model Driven Architecture and SOA
The current concept of Model Driven Architecture (MDA) starts from IT functional requirements, as modeled in UML diagrams and ends up with a complete application to roll into production.  With respect to SOA, using MDA would mean that the software developers create the Service Components using MDA and assemble these components into a Composite Application using orchestration based on MDA.  In fact, modeling the assembly of the composite application is a quick way to spot both coding and system functional design defects.

However, in a broader interpretation of MDA, implementers of new systems, application, and/or services could start with Business Process Modeling to identify business process requirements and functions and use these to derive the IT functions.  This enables the Systems Engineer and System Architect to capture more complete requirements and trace the IT functions to the business or organizational functions they support.  This modeling concept better fulfills the process envisioned for SOA, than merely using MDA to create the composite applications.  Now the services are traceable to the business or organizational processes and functions; which means that when the process changes, the effects on the composite application can be modeled and the composite application is more easily updated to enable and support the new process or function.

Event Driven Architecture and SOA

Event Driven Architecture (EDA) focuses on creating agile software, that is, software that can successfully react to external stimuli in a timely manner.  To perform this mission an EDA must be able to dynamically change both the functions it performs and the ordering of these functions in response to the stimulus.  This function falls along the branching logic dimension, which at one extreme is the simply "IF" statement and at the other is Artificial Intelligence (AI).  Obviously, the concepts from Knowledge Management (KM) can come into play with the functions along this dimension.  The key to EDA is determining what really constitutes an event.  This requires Complex Event Processing (CEP).  In terms of the OODA Loop, CEP constitutes the OO portion ("Observe" and "Orient").

Actually, EDA can support three of the four functions of the OODA based on the concept of types of "event" rules.  There are three type are Knowledge Rules, Event Rules, and Value Rules.  A knowledge rule supports the Observe function of the OODA loop.  A Knowledge Rule

An event rule supports the Orient function of the OODA loop. Several measurements may predict an Event.  An Event Rule is a codification or formalization of this grouping and ordering of the measurements within a confidence interval.  For example, in the 1920s the prediction of a hurricane forming was decidedly hit or miss (a very wide confidence interval).  Today, hurricane and tornado predictions are much more valid (a much smaller confidence interval); in fact, there are possibilities for predicting earthquakes and Tsunamis.

In many cases, it is not a single event that causes a major or catastrophic problem or issue, but an event chain.  This has been found in the vast majority of recent aircraft accidences.  Understanding these event chains is CEP.  Another example is that now, the weather service not only predict that a tropical storm or hurricane will occur, but also identify where it will make landfall, and or if it will make landfall.  These two events, "there is a hurricane", and its "coming our way" are part of the event chain that determine whether, "we'll sit this one out", or "it's time to leave".  To decide requires the decision-maker to understand the risks and rewards of the choice.

A value rule supports the decision-maker in the Decide function of the OODA loop.  A value rule determines the value to the organization of alternatives when an event or event chain has occurred.  In the case of the hurricane, the decision-maker makes the decision-based on the events, and value of staying versus leaving.  One well known example of value rules comes from Isaac Asimov.  The three value rules for robotics are:

1. A robot may not injure a human being or, through inaction, allow a human being to come to harm.
2. A robot must obey the orders given to it by human beings, except where such orders would conflict with the First Law.
3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Laws.

While the knowledge rules are state attributes and change of state attribute, the events create decision points in the process.  With SOA, the orchestration or choreography engines use the processes flows from the assembly process plus the rules, based on the governance, to enable the process flow among the functions (service components) of the composite application.  If the SOA is the Ecosystem SOA (public cloud), then the composite application uses choreography to dynamically interlink the service components [Sidebar: In a web services type of application, these service components are the web services.]  Choreography enables the application to choose which function or service component to use next anywhere on the Internet.  By coupling the choreography repository and engine with a organizational rules repository, and CEP rules repository and engine, the composite application can dynamically link to any appropriate service component that is available that, according to its combined rules will produce the best outcome.  The down side to this is that, first, as the number of alternates increases, the composite application will slow down, and second, the composite application may become much more stochastic than would normally be expected or useful.

Model and Event Driven Architecture and Enterprise Architecture

Model and Event Driven Architecture is an amalgamation of Model and Event Driven Architecture, and in such an alloy increases the speed of the composite application, while reducing the risk that the application may become stochastic.  At the same time, the composite application is much more agile than that built using MDA.

The cost of this amalgamation is in terms of the complexity of the combined orchestration/ choreography/CEP engine (functionality) and the leadership and management discipline needed to ensure that the governance and "event" rules are enabling and supporting the organization's vision and mission, and that they are as internally consistent as possible--that is, a minimum number of "catch 22s".  This will require the a person with the role and responsibilities of an Enterprise Architect, as well as those of Systems Engineer and System Architect.

ROI Versus VOI

Return On Investment

According to Investopedia (http://www.investopedia.com/terms/r/returnoninvestment.asp#ixzz1bcKSKpx9)::

"A performance measure used to evaluate the efficiency of an investment or to compare the efficiency of a number of different investments. To calculate ROI, the benefit (return) of an investment is divided by the cost of the investment; the result is expressed as a percentage or a ratio.

The Return On Investment formula:

In the above formula "gains from investment", refers to the proceeds obtained from selling the investment of interest."

 ROI is simple and straightforward, but does not enable or support an organization's ability to make investment decisions.  It is a lagging indicator; it tells you what were good and bad decisions in the past, but nothing about deciding on investments to increase the long-term effectiveness or agility of an organization, which are vital in today's "bonkers" business environment (to use Tom Peters' term), and disruptive technology environment.

One famous example demonstrates this point.  When Jeff Bezos founded Amazon.com in 1994, he knew that he needed customers to create what he envisioned as "The Walmart of the Web".  This meant that he had to decide on a mission, which was to create a highly visited website.  He then settled on at least four strategies.  First, concentrate on selling books, don't try to sell all the products initially.  Second, create a website with a superb user experience.  Third, ensure that Amazon has access to all books any of its customers could possibly want.  Fourth provide value to the customer, to the point of selling books below wholesale cost, to generate website traffic.

Value On Investment

None of these strategies are designed to produce an ROI, but especially the fourth.  In fact, the mission of creating a highly visited site does not create ROIObviously, if the vision of creating the "Walmart of the web" is realized, there will be a good deal of ROI.  But the mission to start to realize the vision does not support ROI--but it does create Value On Investment (VOI).  A decision-maker creates Value On Investment when his or her decisions increases the organization's Production Capability (after Dr. Stephen Covey), that is, the ability of an organization to be more effective at achieving its vision and mission.

Jeff executed and persevered in the mission to increase Amazon's growth to these disciplined strategies to increase Amazon's VOI, which is the point of the mission to create a highly visited website, that is, a highly visited site is much more valuable than a site that is visited very little.  In executing this mission, it did not turn a profit until 2000, much to the ire of many of its investors, but it took an increasing share of the customer base that itself has been growing at an increasing rate.  Therefore, Jeff met this mission for Amazon.

The enterprise architecture for Amazon was constantly upgraded, including software, hardware, and physical infrastructure.  The net result is that Amazon is now competing with all other major retailers for customers.

Unfortunately, too many CEOs give into the nanosecond market that expects an increasing ROI every quarter or sooner.  It is like expecting George Washington to beat the British in every battle in the Revolutionary War.  Washington lost far more often than he won, but together with the other founding fathers, he did create an enormous VOI that subsequent generations have transformed into ROI.  Likewise, NASA in the Moon Mission created VOI that Jeff Bozos, Bill Gates, Steve Jobs, and the world have profited from.

If the people of the "Occupy Wall St." movement have a point, its that there is too much focus on Wall St. for ROI and too little on VOI.  Wall St. is using the wrong metric for the wrong duration.  And they are "leading" the rest of us into bankruptcy as a result.

Types of Requirements

Definition of a Requirement

A Requirement is a measurable expression of what a customer wants and for which the customer is willing to pay.  Therefore, a requirement has three attributes:

1. It has a description of what the customer wants or desires or some derivation or transformation of what the customer wants or desires.
2. That want or desire is measurable in a way to ensure that the want or desire is fulfilled.
3. The customer is willing to pay for the supplier to fulfill the want or desire.

All types of requirements these attributes.  However, there is another class of documented wants and desires of customers that many customers and Systems Engineers misclassify as requirements.  These wants and desires do not have the second attribute, a metric to determine when the requirement is met.  I call these capabilities and the description, capability statements.  Customers are particularly enamored of the capability statements since, "Customers don't know what they want, but do know what they don't want."  The reason that they are enamored of capability statements is that language is slippery and allows for a great deal of interpretation/interpolation.  This means that if the supplier presents them with a result the supplier feels meets their capability need, the customer can stay NO IT DOESN'T; and the supplier has no recourse but to default on the contract or or ask for "the real requirements" to implement the product against.  This is very expensive for the supplier and very enjoyable for the customer, especially if they really don't what they want (measurably) and want the supplier to pay for the research; which is essentially what the supplier is doing in this mode of operation.  I have been on too many "opportunities" where I have seen this in action, despite my best attempts to stay the effort until at lease some of the "the real" requirements are known.  When the effort has been stayed by the program or project manager and the customer begins to understand the importance of requirements, as opposed to capability statements, then the customer tends to pay much more attention to "getting the requirements document right".

Types of Requirements and Roles

I will define and delineate the types of requirements by the Roles the identifies or defines and documents them.  There are three roles within the major discipline of Systems Engineering that identify or define requirements, as I discussed in my posts The Definition of the Disciplines of Systems Engineering and Enterprise Architecture and System Architecture and a presentation of the same at 2009 SEI SATURN conference.  The roles are: the Systems Engineer, the System Architect, and the Enterprise Architect.

Systems Engineer

The Systems Engineer identifies and manages the Customer's Requirements.  The Systems Engineer never should define the customer's requirements, but work with the customer to identify the requirements.  Once identified, the Systems Engineer must manage the customer's requirements to validate that these requirements are met.  There are two types of Customer Requirements, Programmatic Requirements and Customer System Requirements.

Customer Programmatic Requirements

There are two types of programmatic requirements:

• Cost--What the customer is willing to pay for a product that meets the System Requirements
• Schedule--How long the customer is willing to wait for the product that meets the System Requirements

Program Management considers the programmatic requirements within their exclusive purview.  They are wrong!  To start with, the Systems Engineer must analyze the known requirements to determine whether or not the effort is even feasible within budget and time constraints. 

Sidebar
To often, business development (or marketing), or for internal efforts, "management" will promise more (in very general terms) than any supplier can deliver.  I call this, "Listening with your mouth!"  Good Systems Engineering practice requires ears.  The Systems Engineer must work with the customer to clearly (measurably) identify the customer's real requirements, in part, by listening to the customer and documenting the requirements the customer states.

End of Sidebar

Once the Systems Engineer, together with any Subject Matter Experts (SMEs), have determined that the effort is feasible, the Systems Engineer must use Requirements Analysis (which I will discuss in a future post) to determine any gaps or conflicts among the requirements.  These should be documented, and if possible, discussed with the customer.  Then the Systems Engineer, SMEs, and Contracts expert should put together a proposal, based on the known and inferred requirements, including a notional Work Breakdown Structure (WBS) with a Bases Of Estimate (BOE), and a notional project plan.  If the proposal is accepted, the Systems Engineer/System Architect and SMEs should refine the WBS and Project Plan before turning it over to the Program Manager (unless the Program Manager is also an SME or Systems Engineer, which, in my experience, they ain't).  In this scenario, the people with the knowledge and the skills to execute the plan are the ones that create the plan; not those only interested in managing the programmatic requirements.

Customer System Requirements

Of more interest to the Systems Engineer, because these are the requirements from which the product will be constructed, while meeting the programmatic requirements (which are a form of Design Constraint), are the Customer's System Requirements.  In the US DoD Mil-Spec nomenclature from the 1970s and 1980s, the customer's System Requirements are the A-Spec.  The reason I bring this up is to show that these requirements concepts are not new.  As I describe in an early post, Types of Customer Requirements and the System Architecture Process, there are two types of System Requirements, those that the product must perform and those the product must meet.

Sidebar
The metrics associated with the Customer's System Requirements are the System Validation criteria.  While some Systems Engineers would argue that they can only Verify that the product meets the customer's requirements.  That misses the distinction between verification and validation.  By the best definitions I've seen, verification indicates that the components or functions meet the component specifications or the functional requirements.  Validation means that the product meets the customer's system requirements, that is, it does what the customer requires.  Many times all the parts of a product work as specified by the System Architect, but the System Architect did not specify a product, in total, that would meet the customer's system requirements.  That is the key difference and the rest is semantics.
End of Sidebar

Customer Functional Requirements

The Customer's Functional Requirements are those requirements that the product, system, or service "must perform".  The customer's functional requirements are based on how the customer envisions using the product.  For example, in IT, it would be "one way one type of user would use the application, system, or service".  This is nearly the exact definition of a Use Case (see the Glossary for the exact definition).  I've found that Use Cases are the simplest, clearest way for the Systems Engineer to communicate the customer's functional requirements to the development/transformation team.  Again, a Functional Requirement is always an action.

Design Constraints

Design Constraints are those requirements that the product, system, or service "must meet".  For example, the customer might want a display in the new Service or a report from the new Service to be identical with that of the current application.   Or the customer might want given level of dependability, the term which the Reliability, Maintainability Serviceability organization uses as the umbrella for all the "-ilities" (e.g., reliability, maintainability, serviceability, scalability, recover-ability, response time, up-time, etc.).  These are all attributes of the product, system, service.

There are many types of Design Constraints.   For example, internal and external policies and standards all constrain a design of the product and there can be sever consequences if a supplier ignores one or more of theses Design Constraints, even if the Systems Engineer does not know of them.  The Systems Engineer has to verify that all of these Design Constraint, type requirements, are met before the product is put into operation.

System Architect

The System Architect transforms the customer's System Requirements in the product's Functional Requirements, then orders and structure these Functional Requirements into a System Architecture or Functional Design, allocates tightly coupled groups of these requirements, together with the Design Constraints, to candidate Components Requirements or Specifications, and finally to perform Trade-off Studies to determine the initial Bill Of Materials (BOM).

System Functional Requirements

The System Functional Requirements (or Functional Requirements) are a set of functions the product, system or Service must perform to meet the Customer's System Requirements.  In the US DoD Mil-Spec terms, this would be the B-Spec.  The Functional Requirements are the result of a transformation of the Customer's System Requirements by the System Architect.  This transformation is the result of two fairly complex procedures, Decomposition (a form of Requirements Analysis) and Derivation (which can use a fairly formal process including UML diagrams, but that requires significant creativity if done properly.)

System Component Requirements (or Specifications)

System Component Requirements (or Specifications) are the requirements for the actual components needed to create the product, system, or Service.  In the US DoD Mil-Spec terms, this would be the C-Spec.  Again, the System Architect transforms requirements, this time it is the Functional Requirements into the Component Requirements.  Again the transformation is a result of two fairly complex procedures, the creation of a System Architecture, also called a Functional Design through ordering and structuring the requirements.  Then the System Architect allocates both the grouped System Functional Requirements and the Design Constraints to potential components.  The System Architect can use metrics associated with the System Component Requirements (specifications) within a trade-off study function to determine the actual components that the product will use.

Enterprise Architect
The Enterprise Architect supports the organization's leadership in optimizing their investment decisions, their policies and standards, to enable the organization to achieve its Vision, and Mission, as I've discussed in A Model of an Organization's Control Function using IDEF0 Model, The OODA Loop, and Enterprise Architecture and Governance, and Policy Management Processes: the Linkage with SOA. Optimizing is not increased cost efficiency of the investment, but the effectiveness of the processes used to achieve the mission or goal, and the cost efficiency of the tooling in supporting the effective processes. This is exceptionally important to any organization, since it can mean the difference between thriving and not surviving.

To adequately perform his or her role, the Enterprise Architect must work with three types of requirements, the Organizational performance requirements, Organizational (business) process requirements, and service component requirements.

Organizational Performance Requirements

There are three layers of performance requirements for an organization they all derive from the organization's Vision Statement.  For example, the Vision Statement for the United States--one of the best I've seen--is its Preamble,

We the People of the United States, in Order to form a more perfect Union, establish Justice, insure domestic Tranquility, provide for the common defence, promote the general Welfare, and secure the Blessings of Liberty to ourselves and our Posterity, do ordain and establish this Constitution for the United States of America.

As you can see, there are a series of statements "establish Justice", "provide for the common defence", and so on, that both provide the Vision and give a broad hint at both a series of Mission Statements and the method for measuring the achievement of the Vision through the various Missions.  These are the Vision Requirements.  These Vision Requirements enable the measurement of the success of the organization's various Missions.

To realize the Vision, organizations create various Missions.  Military organizations are best known for Missions, while many business organizations have a goal with multiple objectives fulfilling the role of Vision and Mission.  To perform a Mission, it has Mission Requirements, that is, needs that must be fulfilled to perform the Mission.  These include requirements for the Strategies, Processes, and enabling and supporting tooling.  The Enterprise Architect can use the metrics from these requirements to determine the success of the Strategies, Processes, and tooling in supporting the Mission, and through the Mission's requirements linkage to the Vision, their success in achieving the organization's Vision.

The final class of objects Strategy Requirements are needs of a strategy (guidance for achieving the Mission), in terms of the processes and tooling to enable and supporting it.  Products to enable and support these are supplied by the Processes and tooling.  Therefore, they are the key "business requirements" imposed on both the processes and tooling.

Organizational (Business) Process Requirements

The Organizational (Business) Process Requirements come from two sources.  The first set is the requirements of the Performance Architecture (Mission and Strategy Requirements) and the second set is derived from other Organizational Process Requirements.  This set of requirements falls into two categories.

Mission Alignment Requirements

Mission Alignment Requirements are needs of the Mission and Strategies that the processes and tooling must perform must perform to achieve the organization's vision.  Additional Mission Alignment requirements may be imposed on one process by another process.  These tie measurably back to the Mission and Strategies.  The Enterprise Architect uses them to determine which processes and tooling to retire, which to transform, update, or upgrade, and what new processes and tooling the organization requires move toward achieving it Mission and Vision.  This is the objective of the Mission Alignment process.

Policy and Standard Requirements
Policy and Standard requirements are needs the processes and tooling must meet to reduce the intra-organizational process friction, thereby making the processes and tooling both more process effective, and also more cost efficient.  The Enterprise Architect uses these requirements to identify candidate policies, standards, and business rules to enact, revise, or delete.  The Enterprise Architect will use this information and knowledge to support the organization's Governance process.

Service Component Requirements
As a result of the Mission Alignment process, the organization's leadership will fund individual development and transformation efforts.  Mission Alignment identifies "customer requirements" from the Enterprise level.  These are "Customer" Functional Requirements and are used by the effort's Systems Engineer.  The products of the Governance and Policy Management process are Design Constraints also used by the effort's Systems Engineer.

So, as you can see, all of these types of requirement form a requirements web or mesh.  And all of them, if use consistently and in concert can substantially boost the velocity of the organization toward its Mission and Vision.

Governance, and Policy Management Processes: the Linkage with SOA

Business Rules and Process Flow

In a recent post, A Model of an Organization's Control Function using IDEF0 Model, The OODA Loop, and Enterprise Architecture, I briefly described the Governance and the Policy Management processes within the context of the IDEF0 Model of the organization and the OODA Loop process pattern.  One function of the Policy Management process is to create "business" (organizational) rules and measurably instantiate the policies and standards.  Once created, these rules can be automated since they turn out to be the "if-then-else" statements in the process logic (for more on this see Types of Business Rules).  As discussed in the post, there are at least three types of Business Rules, knowledge rules, event rules, and value rules.  Each of these rules is an attributed metric, that is, it measures what is going on or changing and compares that with some limit or standard.  This makes the rules relatively easy to automate.

SOA and Process Flow

Many individuals and organizations still confuse Service Oriented Architecture (SOA) with Web Services on either the Internet (the Public Cloud) or an intranet (a Private Cloud).  SOA is a much larger change in IT architecture than that (i.e., "A gaggle of Web Services does not a SOA make").  As I discussed in SOA, The (Business) Process Layer, and The System Architecture , one key difference is that SOA formally separates the process flow from the functions of the process.  This difference makes it feasible to enforce a continuously changing set of rules while the system is operating rather than updating or developing new applications then rolling them out.  Consequently, this is one of the reasons that SOA-based systems are very agile and one of the reasons that good Governance and Policy Management processes are so important to the success of SOA-based systems as Gartner Group and other studies of SOA implementations have found. 

Additionally, it is the reason that both the Orchestration and Choreography engines of the SOA-supporting infrastructure must be directly linked to a Rules engine, which uses a Rules Repository (within the AEAR) since both control the process flow of the Composite Application.  In this configuration any changes in a rule will immediately affect all processes using the rule (so rules modeling, verification, and validation is an imperative as functions of the Policy Management Process or the Services will end up in a complete).

Where Rules Apply
There is yet another complication in this rules management discussion, relative to SOA, System, and Enterprise Architecture, that is some policies and standards apply during the development or transformation process (Design Time) and other apply during Service operation (Operational).  All Design Time and Operational Policies and Standards are Design Constraints (see Why Separate Design Constraints from the Customer's System Requirements? for a discussion of Design Constraints).  Both the Systems Engineer and the System Architect must ensure that these requirements are identified and met.

Design Time

Design Time policies and standards divide into two types, those that effect how the designer and developers implement the Composite Application and those that effect the resulting Composite Application.  For example, in a SEI CMMI Level 5 organization, developing a Composite Application following the dictates of the CMMI Level 3 process is a must, but, whether or not the Composite Application has two-factor authentication is not a concern.  However, if the organization has an organizational standard that all applications will have two-factor authentication will directly affect the resulting Composite Application.  So the first example standard effects "how the Composite Application is built" while the second dictates "what is built".  Good process engineering and management, systems engineering, and system architecture will ensure that all of these requirements are met.

Operational

Operational Policies and Standards are not IT-based, but "business" based for whatever the "business" of the organization is.  They are the attributes of the "If-then-else" branching clauses in the "business" process flow, which is instantiated in the code.  While the branching statements themselves will, generally (See the sidebar for the caveat), remain the same, the attributes, as instantiated by the rules can and will change.  This makes Verification and Validation of the Service (Composite Application) process flows during the development or transformation process difficult, at best.  This is the reason for the modeling, verification, and validation of the Services affected by the rules change before it is roll into the operational environment.

Sidebar
In a four part paper in the Journal of Enterprise Architecture, I envisioned the Self Adapting Service Architecture in which both the form and attributes of the branching statements adapt to changes as part of self-rules setting (or information abstraction) process (see my post The Hierarchy of Knowledge and Wisdom for the definitions).  A prototype of such a system is the IBM's Watson computer that won at Jeopardy over several Jeopardy champions.  As I suggested in the Fourth part of the paper, I would not expect such systems to be operational before 2021.
End of Sidebar

The Linkage
From this brief discussion, I think I've made clear that there is a close linkage between SOA and the Policies and Standards set by the Governance and Policy Management processes.  Getting this linkage in place will require the teaming of the leadership, Enterprise Architect, management, Systems Engineers, System Architect, and Subject Matter Experts (SMEs); this is a significant cultural shift throughout the organization, but organizations that achieve it will have a major competitive advantage over those that don't.

SOA Architects: "Don't Bury the Business Rules in the Database"

Why Rules Get Buried in Databases

Even though the Millennium Bug was first noted in 1982 (or perhaps earlier), it was not widely recognized until the mid-1990s.  Fundamentally, the "millennium bug"was caused by the need to minimize the amount of data storage during the 1960s and 1970s and the lack of understanding of System Architecture earlier.

Fragmented Architecture

Initially, the business logic was hard coded into the programming.  Partially, this was due to the fact that in the 1950s and early 1960s, computers only assisted in supporting the mathematical (e.g., accounting) functions of an organization.  So, it was rather straightforward to code in the Business Rules (the if-then-else, as I discuss in Types of Business Rules).  However, as the System Developers attempted to integrate these functions and as the organization's leaders attempted to respond to a changing organizational environment and to changing governmental regulations, they found that changing the enabling and supporting IT systems difficult and expensive (And many large organization's still do, since they are still dependent on millions of lines of code written in the 1960s and early 1970s).  A great part of the reason for this lack of agility was and is the hard-coded business rules in the program's logic--some of which is documented only in that logic.

Monolithic Architecture

In the early 1970s, as disk storage became somewhat less expensive, Relational Data Bases (RDBs) and Relational Data Base Management Systems (RDBMS) started to replace earlier data storage systems.  The RDBMS enabled developer to map data elements into tables with relationship among all the data.  This mapping essentially converts data into information (see the Blog's glossary page).

The advent of the RDBMS was one of the technical enablers of the MRP, MRPII, and ERP systems, built using Monolithic Architecture.  For example the SAP R3 system could not exist without an associated Oracle or other RDBMS system.  The reason is that the Monolithic Architecture is superior to a Fragmented Architecture because it reduces the number of interfaces that have to be maintained among programs to nearly zero--and as discussed in SOA, The (Business) Process Layer, and The System Architecture I found that the cost per interface was $100 per month.  When you multiple that by the 1000s or 10,000s of interfaces required in a large organization with a Fragmented Architecture, the apparent cost efficiency is enough to delight any finance engineer, be they the CFO or any of his or her minions.

Once the RDBMS systems took hold, the Database Designers (DBD) and coders found that the easiest point to validate the data input was just before data insertion into the tables.  The RDBMS suppliers responded with triggers, (if-then-else logic within the RDBMS).  These triggers rejected data that was out of conformance with some standard; which is the essence of a Business Rule.  However, once the DBD was given this tool, he or she started creating ever more complex triggers; triggers that embody Business Rules that may have little to do with data validation.  For example, many times when data is inserted a trigger will change a flag or other indicator within a set of metadata (data about the data).  The flag may then change the course of the function or business process.  This type of trigger buries the Business Rules in the database.

The problem with an application built on a Monolithic Architecture is the inflexibility of the entire system, as I've discussed in The (Business) Process Layer, and The System Architecture and several other posts.  One key technical element of this inflexibility is the System Architect and SME's inability to change or tailor the data structures because the application is dependent on a single data structure (as well as a standard process, etc.).  When the triggers in the RDBMS include a significant number of Business Rules, this increases the inflexibility because the detailed designer/coder/SME may have great difficulty in even locating the Business Rule.  Therefore, he or she must write an add-on or bolt-in to modify the Business Rule, with all of the inherent configuration management problems that creates.  Further, even if the detailed designer can find the proper trigger to modify, ERP systems do not usually identify all of the functions that use the data or the trigger, so there is a significant risk of induced defects, defects in a secondary process caused by the rules change to support the primary process or function of the transformation effort.

Service Oriented and Other Agile Architectures

As I discussed in  The (Business) Process Layer, and The System Architecture, and several other posts, Service Oriented Architecture (SOA) and other agile processes require separation of the processes and Business Rules from the functions of the process. From the discussion, above, the reasons should be obvious.  Both good Governance and Policy Management (see A Model of an Organization's Control Function using IDEF0 Model, The OODA Loop, and Enterprise Architecture, and Enterprise Architecture, Chaos Theory, Governance, Policy Management, and Mission Alignment) and Agile Mission Alignment require this separation.  Without a Rules Repository, the Enteprise Architect and the System Architect are much less effective at optimizing the SOA or other agile architecture to support the organization in a constantly changing operational and technical environment.

The Prescription: an Agile Architectural Process Pattern

As I see it, the following is a prescription for transforming an organization from function-based to process-based, and the IT architecture of the enabling and supporting systems from Fragmented and Monolithic to agile and Service Oriented is to first understand and target agility, in addition to process effectiveness and cost efficiency.  Second, define or ensure that there are formal Enterprise Architecture (Mission Alignment, Governance and Policy Management) and Systems Engineering/System Architecture processes that coordination with one another.  These processes should have a three month cycle, not a year or more, to ensure requirements agility.  Third, as I described in my posts, Initially implementing an Asset and Enterprise Architecture Process and an AEAR, and Asset and Enterprise Architecture Repository, start with the output of current projects to create the AEAR and exercise the processes.  Fourth, expect that the processes will be less than optimal at the start (the learning curve, see the Superiority Principle).  The risk reduction on this is that since the process cycle is three months, instead of a year or more, learning takes place at a much greater rate.

Addendum: When to Use Triggers When Building to An Agile Architecture
I had a futher thought about triggers.  The Gartner Group recommends segmenting the complete data structure (supporting a monolithic architecture) into databases supporting the individual Service Components (e.g., Web Service).  Each of these Service Components would support either an entire or a logical portion of an organizational function.  I would suggest that triggers can be used within the databases supporting individual functions (Service Components), especially for ensuring data completeness and correctness.