Product Architecture Thinking Versus System Architecture Thinking

Cultural Thinking about Architecture

Until the early 1960s, the discipline of architecture (or functional design) focused on the creation/design/ development/implementation of products like buildings, cars, ships, aircraft, and so on.  Actually, other than buildings, most of the Architects were called "functional" designers, or some such term, to differentiate them from detailed designers and engineers/design analysts.  This is part of the reason that most people associate architecture and an architect with the design of homes, skyscrapers, and other buildings, but not with products, systems, or services.  In fact Architects themselves are having a hard time identifying their role.

In the late 1990s, the US Congress mandated that all Federal Departments must have an Enterprise Architecture to purchase new IT equipment and software.  The thrust of the reasoning was that a Department should have an overall plan, which makes a good deal of sense.  I suspect the term "Enterprise Architecture" to denote the unification of the supporting tooling, though they could have used "Enterprise IT Engineering" in the manner of Manufacturing Engineering, which unifies the processes, procedures, functions, and methods of the assembly line.  And yet, Enterprise Architecture means something more, as embodied the the Federal Enterprise Architecture Framework (FEAF).  The architecture team that created this framework to recognize that processes, systems, and other tooling must support the organization's Vision and Mission.  However, its up to the organization and Enterprise Architect to implement processes that can populate and use the data in the framework effectively.  And that's the rub.

Functions vs Processes and Products vs Systems
In the late 1990s and early 2000s the DoD referred to armed drones as Unmanned Combat Air Vehicles (UCAVs), then in the later 2000s, they changed the name of the concept to Unmanned Combat Air Systems (UCAS).  Why?

There are three reasons having to do with a change in western culture, the most difficult changes for any organization.  These are: 1) a change from linear process understanding to linear and cyclic, 2) a change from thinking about a set of functions to understanding a function as part of a process, and a change in thinking from product to system.

Linear vs Cyclic Temporal Thinking
Product thinking is creating something in a temporally linear fashion, that is, creating a product has a start and an end.  D. Boorstin in the first section of his book, The Discovers, discusses the evolution of the concept of time, from its cyclic origins through the creation of a calendar to the numbering of years, to the concept of history as a sequence of events.  To paraphrase Boorstin, for millennia all human thinking and human society was ruled by the yearly and monthly cycles of nature.  Gradually, likely starting with the advent of clans and villages a vague concept of a linear series of events formed.  Still, the cycles of life are still at the core of most societies (e.g., in the east, the Hindu cycles, and the Chinese year, and in the West, Christmas and New Years, and various national holidays). 

The concept of history change cultural thinking from cycles to a progression through a series of linear temporal events (events in time that don't repeat and cause other events to occur).  In several centuries this concept of history permeated Western Culture.  The concept of history broke and flattened the temporal cycles into a flat line of events.  With this concept and with data, information, and knowledge, in the form of books, meant that Western culture now had the ability to fully understand the concept of progress.  Adam Smith applied this concept to manufacturing, in the form of a process, which divided the process into functions (events), and which ended up producing many more products from the same inputs of raw materials, labor, and tooling.

Function vs Process

In the Chapter 1 of Book 1 of An Inquiry into the Nature and Causes of the Wealth of Nations (commonly called The Wealth of Nations), Adam Smith discussed the concept of  the"Division of Labour".  This chapter is the most important chapter of his book and the concept of the Division of Labor is the most important concept; far more important than "the invisible hand" concept or any of the others.  It is because this concept of a process made from discrete functions is the basis for all of the manufacturing transformation of the Industrial Revolution.  Prior to this, the division of labor was an immature and informal concept; after, many cottage industrialists adopted the concept or were put out of business by those that did.

Adam Smith did this by using a very simple example, the making of straight pins.  In this example he demonstrated that eight men each serving in a specialized function could make more than 10 times the number of pins in a day when compared with each of the men performing all the functions.  He called it the division of labor; we call it "functional specialization".

Functional specialization of skills and tooling permeates Western Culture and has led to greater wealth production than any prior concept that has been created.  Consequently, as Western Civilization accreted knowledge, the researchers, engineering, and skilled workers became more expert in their specialized function and increasingly less aware of the rest of the process.

Currently, most organizations are structured by function, HR, accounting, contracts, finance, marketing or business development, and so on.  In manufacturing there are designers (detailed design engineers), engineers (analysts of the design), manufacturing engineers and other Subject Matter Experts (SMEs).  Each of these functions vie with one another for funding to better optimize their particular function.  And most organizations allocate funding to these functions (or sometimes groups of functions) for the type of optimization.

Unfortunately, allocating funds by function is a very poor way to allocate funds.  There is a principle in Systems Engineering that, "Optimizing the sub-systems, sub-optimizes the system".   J.B. Quinn, in “Managing Innovation: Controlled Chaos”, (Harvard Business Review, May-June 1985), demonstrated this principle, as shown in Figure 1.

Figure 1--Function vs Process Funding

As shown in Figure 1, at the bottom where you cannot really see it, for every unit of money invested in a function, the organization will get, at best, one unit of money improvement in the total process.  However, if the investment effects more than one function would yield 2(N-1)-1 in total improvement in the process.  So focusing on investing in the process will yield much better results and focusing on the function.  This is the role of the Enterprise Architect, and the organization's process and systems engineer using the Mission Alignment process.  While this point was intuitively understood by manufacturing (e.g., assembly line manufacturing engineering) for well over 150 years, and was demonstrated in 1985, somehow Functional Management is not willing to give up their investment decision perquisite.

Product vs System

Influenced by the Wealth of Nations, from about 1800 on, industries, first in Britain, then across the Western world, and finally globally, used Adam Smith's concept of a process as an assembly line of functions to create more real value than humankind had ever produced before.  But this value was in the form of products--things.  Developing new "things" is a linear process.  It starts with an idea, an invention, or an innovation.  Continues with product development to initial production and marketing.  Finally, if successful, there is a ramp up of production, which continues until superseded by a new product.  This is the Waterfall Process Model. 

The organization that manufactured the product had only the obligation to ensure that the product would meet the specifications the organization advertised at the time the customer purchased the product, and in a very few cases, early in the product's life cycle.  Generally, these specifications were so general, so non-specific, and so opaque that the manufacturing company could not be held responsible.  In fact, a good many companies that are over 100 years old, exist only because they actually supported their product and its specifications.  Their customers turned into their advertising agency.

This model is good for development (what some call product realization) and transformation projects, but the model has two fatal flaws, long term.  The first (as I discuss in my post Systems Engineering, Product/System/Service Implementing, and Program Management) is that the waterfall process is based on the assumption that "All of the requirements have been identified up front"; a heroic assumption to say the least (and generally completely invalid).  The second has equal impact and was caused by the transportation and communications systems of the 1700s to the 1950s.  This flaw is that "Once the product leaves of the factory it is no longer the concern of the manufacturer."

This second flaw in historical/straight line/waterfall thinking effects both the customer and the supplier.  The customer had and has a hard time keeping the product maintained.  For example, most automobile companies in the 1890s did not have dealerships with service departments; in fact they did not have dealerships, as such.  Instead, most automobiles were purchased by going to the factory or ordering by mail.  And even today, most automobile manufacturers don't fully consider the implications of disposal when design a vehicle.  So they are thinking of an automobile as a product not a system or system of systems (which would include the road system and the fuel production and distribution systems.  The flavor of this for the United States is in its disposable economic thinking; in everything from diapers to houses (yes, houses...many times people are purchasing houses in the US housing slump, knocking them down, to build larger much more expensive housing...at least in some major metropolitan areas).  Consequently, nothing is built to last, but is a consumable product.

Systems Thinking and The Wheel of Progress
Since the 1960s, there has been a very slow, but growing trend toward cyclic thinking with organizations.  Some of this is due to the impact of the environmental movement, and ecosystems models.  More of this change in thinking is due to the realization that there really is a "wheel of progress".  Like a wheel on a cart, the wheel of progress goes through cycles to move forward.
 
The "cycle" of the "wheel of progress" is the OODA Loop Process, that is, Observe, Orient, Decide, Act (OODA) loop.  The actual development or transformation of a system occurs during the "Act" function.  This can be either a straight-line, "waterfall-like" process or a short-cycle "RAD-like" process.  However, only when the customer observes the of the transformed system in operation, orients the results of the observation of the system in operation to the organization's Vision and Mission to determine if it is being effective and cost efficient, then deciding to act or not during the rest of the cycle.  The key difference between product and systems thinking is that each "Act" function is followed by an "Observe" function.  In other words, there is a feedback loop to ensure that the output from the process creates the benefits required and that any defects in the final product are caught and rectified in the next cycle before the defect causes harm.  For example, Ford treated is Bronco SUV as a product rather than a system.  "Suddenly", tire blowouts on the SUV contributed to accidents, in some of which the passengers were killed.  If Ford had treated the Bronco as a system, rather than a product, and kept metrics on problems that the dealers found, then they might have caught the problem much earlier.  Again, last year, Toyota, also treating their cars as products rather than systems, found a whole series of problems.

OODA Loop velocity
USAF Col. John Boyd, creator of the OODA Loop felt that the key to success in both aerial duels and on the battlefield is that the velocity through the OODA Loop cycle was faster than your opponent's.  Others have found that this works with businesses and other organizations as well.  This is the seminal reason to go to short cycle development and transformation.  Short cycle in this case would be 1 to 3 months, rather than the "yearly planning cycle" of most organizations.  Consequently, all observations, orientation and deciding should be good enough, not develop for the optimal, there isn't one. [this follows the military axiom that Grant,  Lee, Jackson, and even Patton followed "Doing something now is always better than doing the right thing later".]  Expect change because not all of the requirements are known, and even if they are known, the technological and organizational (business) environment will change within one to three months.  But remember the organization's Mission, and especially its Vision, change little over time; therefore the performance the metrics, the metrics that measure how optimal the current systems and proposed changes are, will change little.  So these metrics are the guides in this environment of continuous change.  Plan and implement for upgrade and change, not stability--this is the essence of an agile systems. 

This is true of hardware systems as well as software.  For example, in 1954, Haworth Office Furniture started building movable wall partitions to create offices.  Steel Case and Herman Miller followed suit in the early 1960s.  At that point, businesses and other organizations could lease all or part of a floor of an office building.  As the needs of the organization changed these partitions could be reconfigured.  This made for agile office space, or office systems (and the bane of most office workers, the cubicle), but allows the organization to make most effective and cost efficient use of the space it has available.

The Role of the Systems Engineering Disciplines
There are significant consequences for the structure of an organization that is attempting to be highly responsive to the challenges and opportunities presented to it, while in its process for achieving its Mission and Vision in a continuously changing operational and technical environment.  It has to operate and transform itself in an environment that is much more like basketball (continuous play) than American football (discrete plays from the scrimmage line with its downs)--apologies to any international readers for this analogy.  This requires continuous cyclic transformation (system transformation) as opposed to straight line transformation (product development). 

Treating Process in Product Thinking Terms
Starting in the 1980s, after the publication of Quality is Free, by Phil Crosby in 1979, the quality movement and quality circles, the concept of Integrated Product Teams (IPTs, which some changed to Integrated Product and Process Teams, IPPTs) organizations have been attempts to move from a focus on product thinking toward a focus on system thinking).  Part of this was in response to the Japanese lean process methods, stemming in part from the work of Edward Deming and others.  First international attempt to is ISO 9000 quality Product Thinking (starting in 2002), though in transition to Systems thinking, since it is a one time straight-through (Six Sigma) methodology, starting with identifying a process or functional problem and ending with a change in the process, function, or supporting system.

Other attempts at systems thinking were an outgrowth of this emphasis on producing quality products (product thinking).  For example, the Balanced Scorecard (BSC) approach, conceptualized in 1987. The BSC was attempting to look at all dimensions of an organization by measuring multiple dimensions.  It uses four dimensions to measure the performance of an organization and its management instead of measure the performance of an organization on more than the financial dimension.  The Software Engineering Institute (SEI) built layer four, measurement, into the Capability Maturity Model for the same purpose.

In 1990, Michael Hammer began to create the discipline of Business Process Reengineering (BPR), followed by others like Tom Peters and Peter Drucker.  This discipline treats the process as a process rather than as a series of functions.  It is more like the Manufacturing Engineering discipline that seeks to optimize the processes with respect to cost efficiency per unit produced.  For example, Michael Hammer would say that no matter size of an organization, it's books can closed at the end of each day, not by spending two weeks at the end of the business or fiscal year "closing the books".  Or in another example, you can tell if an organization is focused on functions or processes by its budgeting model; either a process budgeting model or a functional budgeting model.

Like the Lean concept, and to some degree, ISO 9000, ITIL,and other standards, BPR does little to link to the organization's Vision and Mission, as Jim Collins discusses in Built to Last (2002); or as he puts the BHAG, BIG HARRY AUDACIOUS GOALS.  Instead, it focuses on cost efficiency (cost reduction through reducing both waste and organizational friction, one type of waste) within the business processes.

System Architecture Thinking and the Enterprise Architect
In 1999, work started on the Federal Enterprise Architecture Framework (FEAF) with a very traditional four layer architecture, business process, application, data, and technology.  In 2001, a new version was released that included a fifth layer, the Performance Reference Model.  For the first time the FEAF links all of the organization's processes and enabling and supporting technology to its Vision and Mission.  Further, if properly implemented, it can do this in a measurable manner (see my post Transformation Benefits Measurement, the Political and Technical Hard Part of Mission Alignment and Enterprise Architecture).  This enables the Enterprise Architect to perform in the role that I have discussed in several of my posts and in comments in some of the groups in the LinkedIn site.  These are decision support for investment decision-making processes and support for the governance and policy management processes (additionally, I see the Enterprise Architect as responsible for the Technology Change Management process for reasons that I discuss in Technology Change Management: An Activity of the Enterprise Architect).   Further, successful organizations will use a Short Cycle investment decision-making (Mission Alignment) and implementing (Mission Implementation) process, for reasons discussed above. [Sidebar: there may be a limited number of successful project that need multiple years to complete.  For example, large buildings, new designs for an airframe of aircraft, large ships--all very large construction effort, while some like construction or reconstruction of highways can be short cycle efforts--much to the joy of the motoring public.]   The Enterprise Architect (EA), using the OODA Loop pattern, has continuous measured feedback as the change operates.  Given that there will be a learning curve for all changes in operation; still, the Enterprise Architect is in the best position to provide guidance as to what worked and what other changes are needed to further optimize the organization's processes and tooling to support its Mission and Vision.  Additionally, because the EA is accountable for the Enterprise Architecture, he or she has the perspective of entire organization's processes and tooling, rather than just a portion and is in the position to make recommendations on investments and governance.

System Architecture Thinking and the Systems Engineer and System Architect
One consequence of the short-cycle processes is that all short-cycle efforts are "level of effort" based.  Level of Effort is a development or transformation effort is executed using a given a set level of resources over the entire period of the effort.  Whereas in a waterfall-like "Big Bang" process scheduling the resources to support the effort is a key responsibility of the effort (and the PM), with the short-cycle the work must fit into the cycles. With the waterfall, the PM could schedule all of the work by adding resources or lengthened the time required to design, develop, implement and verify; now the work must fit into a given time and level of resource.  Now, the PM can't do either because they are held constant.
 If, in order to make an agile process, we use axiom that "Not all of the requirements are known at the start of the effort", rather than the other way around, then any scheduling of work beyond the current cycle is an exercise in futility because as the number of known requirements increases, some of the previously unknown requirements will be of higher priority for the customer than any of the known requirements.  Since a Mission of a supplier is to satisfy the needs of the customer, each cycle will work on the highest priority requirements, which means that some or many of the known requirements will be "below the line" on each cycle.  The final consequence of this is that some of the originally known requirements will not be met by the final product.  Instead, the customer will get the organization's highest priority requirements fulfilled.  I have found that when this is the case, the customer is more delighted with the product, takes greater ownership of the product, and finds resources to continue with the lower priority requirements.

On the other hand, not fulfilling all on the initially known requirements (some of which were not real requirements, some of which contradicted other requirements) gives PMs, the contracts department, accountants, lawyers, and other finance engineers the pip!  Culturally,generally  they are incapable of dealing in this manner; their functions are not built to handle it when the process is introduced.  Fundamentally making the assumption that "Not all the requirements are known up front" makes the short-cycle development process Systems Requirements-based instead of Programmatic Requirements-based.  This is the major stumbling block to the introduction of this type of process because it emphasizes the roles of the Systems Engineer and System Architect and de-emphasizes the role of the PM.

The customer too, must become accustomed to the concept, though in my experience on many efforts, the once the customer unders the customer's role in this process, the customer becomes delighted.  I had one very high-level customer that said after the second iteration through one project, "I would never do any IT effort again that does not use this process."

SOA in a Rapid Implementation Environment

"The farther backward you can look, the farther forward you can see. "

Winston Churchill

Assertion

SOA is particularly well suited to Rapid Implementation and Rapid Transformation, which is an outgrowth of RAD Processes. 

I've divided this post into two sections.  First, a brief history of the trends  in software coding and development, I've experienced; then, a discussion of how these trends will be amplified using Rapid Application Implementation processes, coupled with SOA.

Background

In my post The Generalized Agile Development and Implementation Process, I indicate that in my experience (that is over 40 years of experience (boy, time goes fast)), RAD software implementation process produces more effective and cost efficient software than any other method.  There have been many changes in that time;  binary to machine language, from machine languages to assemblers, from assemblers to procedural languages, from procedural languages to structured code, from structured code to object oriented languages, from object oriented to web services, which have been extended to service oriented. 

Up to Object Oriented, all of these changes were developed to enhance the cost efficiency of creating and maintaining code.  For example, going from coding in an assembler to coding in FORTRAN 1 enabled coders to construct an application much faster by using a single statement to insert a standard set of assembler code into the object deck of the application.  For example, no longer did the coder have to rewrite all of the assembler instructions for saving data to given location on a disk--the instruction together with disk utilities took care of ensuring there was adequate space, writing the data, and the associated "housekeeping" activities.  The downside to this change was that the object code was much larger, requiring more CPU cycles and memory.  Consequently, there were (and are) applications that still require coding in an assembler (e.g., real time flight controls).

The second evolutionary transformation of coding was from procedural to structured.  Due, simply, to the limitations in CPU cycles and memory in early computers and the programming culture of assembler where code was reused by branching into a section and then to another to minimize the number of separate instructions.  With increasing computing power and the implementation of procedural languages, like FORTRAN and COBOL, the branching of code continued; to the point that maintenance of the code became infeasible in many cases, depending on the programmer.  One way that many programmers, including me, used to avoid the issue way to use a modular IT system architecture--that is using callable subroutines as part of the code.  In some cases the majority of the code was in subroutines with only the application work flow as part of the main--Sound familiar?

Anyway, this concept was formalized with the third evolutionary transformation, from procedural to structured (also known initially as Goto-less programming).  This was a direct assault on programmers that used unconditional branching statements with great abandon.  The thought here was to reduce the ball of tangle yarn logic found in many programs to increase the code readability and therefore maintainability.  That's when I first heard the saying, "Intel giveth, structured programming taketh away", meaning that structured code use a great deal more computing power.  At this point, compiler suppliers started to supply standardized sub-routine libraries as utilities (a current example of this type of library is Microsoft's Dynamic Link Library (DLLs)).  Additionally, at this time Computer Aided Software Engineering (CASE) tools began to come on the market.  These tools aided the code developer and attempted to manage the software configuration of the application.

The fourth transformation was from structure programming to object oriented development.  Object Oriented Development (OOD) is not a natural evolution from structured programming.  Instead, it's a paradigm shift in the programming concept--notice the shift in terminology from "programming" to "development".  OOD owes as much to early simulation languages as coding.  These languages, like Simscript, GPSS, GASP, all created models of "real-world" things (i.e., things with names or objects).  Each of these model things was programmed (or parametrized) to mimic the salient characteristics of a "real-world" thing.  OOD applied this concepts of objects with behaviors interfacing with each other to "real-world" software.  So that, for example, a bank might have a class of objects called customers.  Each object in the class represents one customer.  The parametrize attributes of each customer object would include the customer's name, account number, address, and so on.  Each of these customer objects has "behaviors" (methods) that mimic, enable and support what the customer wants and is allowed to do.  For example, the customer object will have deposit, withdrawal, and inquiry methods.  The interaction of the input and output methods, together with the objects and attributes, create the application.

A key problem in creating Object Oriented applications is with the interfaces between and among methods supporting the interaction among objects.  OOD did not have standards for these interfaces so that sharing objects among applications was hit or miss.  Much time and effort was spent in the 1990s on the integration among object orient application objects.

To resolve this interface, application developers applied the concepts of HTML so successfully use by web developers.  HTML is based on the Standard Generalized Markup Language (SGML), developed, in part, by the aerospace industry and the Computer Aided Logistics Supporting initiative to support the inter-operation of the publishing of technical documentation.  And this industry had a lot of it.  At the time, (the late 1980s) a Grumman Corporation technical document specialist reported that when the documentation for all aircraft on an aircraft carrier was initially loaded on, the carrier sank 6".  The SGML technical committee came up with standard tagging and standardized templates for document interchange and use.  Web development teams latched on to the SGML concepts to enable transfer of content from the web server to the web browser.

Likewise, those seeking to easily interlink objects were looking for standardizing the interface and its description.  The consequence of this process research was XML.  The application of this standard to objects creates Web Service style Service Components.  These components have a minimum of three characteristics that enable them to work as Web Service style Service Components:

1. They persistent parametrized data mimicking the characteristics of some "real-world" object
2. They have methods that create the object's behavior and
3. They have interfaces adhering to the XML standard.

Having a great many Web Services is goodness, but it could turn the Internet into the electronic equivalent of a large library with no card catalog.  The W3C and OASIS international standards organizations recognized this issue early and created to standards to help address the problem, Web Services Description Language (WSDL) and the Universal Description Discovery and Integration (UDDI).  These standards, with many additional "minor" standards are converging on a solution to the "Electronic Card Catalog for Web Services" and/or the "Apps Store for Web Services".

In summary, initially programming was about creating code in as little space as possible.  As hardware increased in power (according to Moore's Law), programming and its supporting tools focused on creating more cost effiecent code, that is, code that is easily to develop and maintain.  This led to the paradigm shift of Object Oriented Development, which began to create code that enabled and supported particular process functions and activities.  In total, this code was much more complex, requiring more CPU cycles, more memory, and more storage.  Since, by the 1990s, the computing power had doubled and redoubled many times, this was not a problem, but linking the objects through non-standard interfaces was.  The standards of Web Services provided the answer to this interfacing issue and created the requirements for another paradigm shift.

Now that Service Components (Web Services and Objects meeting other interface standards) can enable and support functions or activities of a process, organizations had a requirements for these functions to be optimally ordered and structured to support the most effective process possible, and the most cost efficient manner (as technology changed) and to be agile enough to enable the organization to successfully respond to unexpected challenges and opportunties.

Rapid, Agile, Assembly

Even a superb research library with a wonderful automated card catalog is of no value to anyone that has no idea what they are looking for.  To use another analogy; my great uncle was an "inventor".  But since he didn't what type of mechanical invention he might want to work on, he would go to auctions of mechanical parts--which meant that he had a shed sized room filled with boxes of mechanical part.  He didn't know if he would ever use any of them, but he had them just in case. 

Many organization's initial purchase of Service Components (Web Services) has been much like my uncle's acquisition of mechanical parts.  Many times, the purchase of Service Components and supporting infrastructure components from multiple suppliers creates as many issues as it resolves because later versions and releases of "the standards" are incompatible with earlier versions of the same standard, and many software suppliers interpreting the standard to better match their existing products.  The result is "the Wild West of Web Services" with supporters of one version of a standard or one product suite "gunning for" anyone that doesn't agree with them.  However, the organization IT is cluttered with the equivalent of a Service Component bone yard (which is like an auto junkyard--a place where you can go to potentially find, otherwise difficult to locate car parts.

To convert these (Service Component) parts into Composite Applications supporting the organization's processes requires a blueprint in the form of a functional design (System Architecture), an identification of the design constraints on the process and tooling (including all applicable policies and standards).  The key is the functional design and its implementation as a Composite Application using Assembly methods; see my posts Assembling Services: A Paradigm Shift in Creating and Maintaining Applications and SOA Orchestration and Choreography Comparison.

In addition to the paradigm shift from software development (coding) to Service Component assembly, there are two major changes in programming when moving from creating Web Services to creating SOA-based Composite Applications.  If the processes for these are not well thought out, or have poor enabling and supporting training and tooling, or are poorly executed, then assembling Service Components will be difficult, time and resource consuming.  This means the creating Composite Applications would be Slow Application Development (SAD???) instead of RAD.  This happens frequently when initially operating the assembly process and seems to follow the axiom that "the first instance of a superior principle is always inferior to a mature example of an inferior principle".  However, many books on the development of SOA-based applications describe processes that either are inherently heavy weight and slow and resources consuming, or that will process applications with a great many defects; and rectifying a defect in a product that is being used costs much more than rectifying it in design.  Therefore, spending the time and effort to fully develop the process and will continue to provide a genuine ROI with each use of the process--though the ROI is difficult to measure unless you've already have a poor process that measure first, or if you can measure the ROI in a simulation of alternatives.

Here is an outline of the RAD-like implementation process I would recommend.  It fits very nicely within The Generalized Agile Development and Implementation Process I discussed.  Implementing and operating/ maintaining a SOA-based Composite Application is much more than simply coding into a work flow Service Component and then executing it in production.  There are several activities required to implement or transform a Composite Application in an effective and cost efficient manner using a RAD-like implementation process.

This implementation process is predicated on at least 5 items in the organization's supporting infrastructure.

1. That the implementation team has established or is establishing an Asset and Enterprise Architecture Repository (AEAR).  This should include a Requirements Identification and Management system and repository (where all of the verification and validation (tests) are stored).
2. That the implementation team is using a CMMI Level 3 conformant process (with all associated documentation
3. That there are business process analysis (including requirements identification and modeling), and a RAD-like Systems Engineering and System Architecture process preceding the implementation process.
4. That the implementation team has access to a reasonable set of Composite Applications modeling tools that link to data in the AEAR.
5. That the organization's policies and standards are mapped into automated business rules that the process flow can use through a rules engine linked to both the Orchestration engine and the Choreography engine.

Model and Verify the Composite Application's flow Before Assembly

First, create and model the process or work flow of the Composite Application before assembling the Service Components.  The key technical differentiator of an SOA-based application from a web service or other type of application is that there are three distinct elements of the SOA application; the work flow, the Service Components (that support the activities and functions of the process), and the business rules that are derived from the organization's policies and standards.  This first activity creates the process flow and ensures that all applicable business rules are linked to the flow in the correct manner.

For those Composite Applications using Orchestration, the entire Composite Application should be statically and dynamically modeled to verify the functions using Service Components models to ensure that they are in proper order, that their interfaces are compatible, that they meet all design constraints, that the Composite Application will meet all organization's business rules (that automate the organization's internal and external policies and standards, and that the Composite Application's performance and dependability meet their requirements.

I found that with RAD, performing this type of verification activity can take from 1 hour for simple applications to two days for a complex application with complete regression testing.  Additionally, I found that both software coders/programmers and program management wanted to skip all of the verification activities that I recommended, but for somewhat different reasons. 

The reason for developers not wanting to do it, was, according to them, "Boring"; actually, it is more "programmer friendly", that is, to ignore verifying that the requirements have been met and that there are no induced defects (defects introduced in the process of updating the Composite Application's functionality), which required in a RAD process to ensure reliability.  Additionally, it's more fun to program than it is to ensure that the version verified is the version that goes into production.

The reason why program managers don't pay attention to the customer's requirements and the requirements verification is that the earned value procedures--the supposed measurement of a program's progress--is not based on measuring how many of the customer's system requirements have been met through verification and validation (V&V), but on the resource usage (cost and schedule) versus the implementer's best guess at how much of which task in the "waterfall" schedule have been completed.  Since producing the earned value is the base for payment by the customer and the PM's and management's bonuses, that is what they focus on--naturally.  Therefore, neither the implementers or management have any inducement to pay a great deal of attention to ensuring that the customer's requirements are met.

However, with a good RAD process like the Generalized Application Development and Implementation Process, the customer's system requirements are paramount (additionally, my experience has been focusing on the programmatic requirements (cost and schedule) pushes the schedule to the right (it takes longer), while focusing on ensuring the customer's requirements (functions, and design constraints) are met, pushes the schedule to the left).

Assemble the Composite Application and Verify

Second, once the work flow has been verified, then it is time to link the Service Components to it in the development environment.  The verification, in this case, is to assure that the Service Component behaviors are as expected (the components meet the component requirements), and that the components have the expected interfaces.

Verify the Sub-systems

Third, once components have been assembled into a portion of the system, or a sub-system, these too, must be verified to ensure that they enable and support the functions allocated to them from the System Architecture (functional design) and meet all applicable design constraints.  Again, this can be done in the development environment, and must be under configuration control.

System Validation

Fourth, System Validation means that the current version is evaluated against the customer's system requirements, that is, that it performs as the customer expects, with the dependability that the customer expects, and meets all of the policies and standards of the organization (as embodied in the design constraints).  Typically, this step is performed in a staging environment.  The are several reasons for doing this, as most configuration management training manuals discuss.  Among these are:

• Separating the candidate application from the developers, since the developers have a tendency to "play" with the code, wanting to tweak it for many different reasons.  These "tweaks" can introduce defects into code that has already been through the validation process.  Migrating the code from the development environment to the staging environment and not allowing developers access to the staging environment, can effectively lock out any tweaking.
• The migrating the application to the staging environment, itself, can help to ensure the migration documentation is accurate and complete.  For example, the order in which code patches are applied to their COTS product can change, significantly, the behavior of the product.
• The staging environment is normally configured in a manner identical with the production environment.  It uses the same hardware and software configurations, though from large installation, the size of the systems is generally smaller than production.  Since the code or product implementers have a tendency to "diddle with" the development environment.  Very often I've been part of efforts where the code runs in the development/assembly environment, but not production because the configuration of the two environments is very different with different releases of operating system and data base COTS software.  Having the staging environment configuration control to match the production environment obviates this problem.

The result is an application that rolls out quickly and cleanly into production.

Roll-out

Fifth, Roll out consists of releasing the first or Initial Operating Capability (IOC) or the next version of the Composite Application.  In the RAD or Rapid Application Implementation process, this may occur every month to 3 months.  If all of the preceding activities have been successfully execute, post roll out stablization will be very boring.

Advantages of SOA in a RAD/I Process
There are several advantages to using SOA in a Rapid Application Development/Implementation process including:

1. The implementer can use Service Component or assemble a small group of Service Components to perform the function of a single Use Case.  This simplifies the implementation process and lends itself to short-cycle roll-outs.
2. Since the Service Components are relatively small chunks of code (and generally perform a single function or a group of closely coupled functions) it allows the implementers at least the following actions:
1. Replace one Service Component with another technologically updated component without touching the rest of the Composite Application as long as the updated component has the same interfaces as the original.  For example, if a new developmental authentication policy is set
2. Refactoring and combining the Service Components is simplified, since only small portions of the code are touched.
3. Integration of COTS components with components that are create for/by the organization (for competitive advantage) is simplified.  While my expectation is that 90 percent or more of the time, assembling COTS components and configuring their parameters will create process enabling and supports Composite Applications, there are times when the organization's competitive advantage rests on a unique process, procedure, function, or method.  In these cases unique Service Components must be developed--these are high-value knowledge-value creating components.  It's nice to be able to assemble these with COTS components in a simple manner.
3. The Structure of an SOA-based Composite Application has three categories of components: Service Components, Process or Workflow Components, and the Business Rules.  Change any of these and the Composite Application changes.  Two examples:

1. Reordering the functions is simplified because it requires a change in the process/workflow component rather than a re-coding of the entire application.
2. If properly created in the AEAR, when an organization's policy or standard changes, the business rules will reflect those changes quickly.  Since in well architected SOA-based systems will have a rules engine tied to the rules repository in the AEAR and to the orchestration and choreography engines executing the process or work flows, any change in the rules will be immediately reflected in any Composite Application using the rule.

These two example demonstrate that a Composite Application can be changed by changing the ordering or rules imposed on the application, without ever write or revising any code.  This is the reason that SOA is so agile, and another reason it works so well with short-cycle Rapid Application Development and Implementation processes.

Tooling

In addition to good training on a well thought out Rapid Application Development and Implementation process, the implementation of the process for creating SOA-based Composite Applications requires good tools and a well architected AEAR.  While implementing the process and acquiring the tooling and instantiating the AEAR may seem like it's a major cost risk, it really doesn't need to be, (see my post, Initially Implementing the Asset and Enterprise Architecture Process and an AEAR).  Currently, IBM, Microsoft and others have tooling that can be used "out of the box" or with some tailoring to enable and support the development and implementation environment that is envisioned in this post.  Additionally, I have used DOORS, SLATE, CORE Requisite-Pro, and other requirements management tools on various efforts and can see that they can be tailored to support this process Rapid Appliction Development and Implementation process.

However, as indicated by the Superiority Principle, it will take time, effort, and resources to create the training and tooling, but the pay back should be enormuous.  I base this prediction on my experience with the RAD process that I developed in 2000, which by 2005 had been used on well over 200 software development efforts and which was significantly increasing the process effectiveness of the IT tools supporting the process, and the cost efficiency of the implementations. 

(PS--the Superiority Principle is "The first instance of a superior system is always inferior to a mature example of an inferior system").

The Generalize Agile Development and Implementation Process for Software and Hardware

Background

In 2000, I created a Rapid Application Development (RAD) process, based on eXtreme Programming (XP) that was CMMI Level 3 conformat. Conformance means an outside audit was performed and the process conforms to the requirements of the CMMI level 3 key practices (note that CMMI levels 4 and 5 are organizational practices rather than process practices, so the Level 3 is as high as a process can get).

Advantages of the RAD Process

This RAD process, as I created it, has several advantages over traditional software development efforts, like "waterfall process".  The following advantages derive from XP.

1. Works in monthly to 6 week cycles, enabling the customer to start using an Initial Operating Capability (IOC) version of the system, while the system in under development.  This meant that the customer was gaining some value even while the application was being implemented.  In the case of one small but unusual asset management, accounting, and accounts payable system, the customer, in the process of loading data into the IOC version found enough errors in the accounts to recover more than twice the cost of the effort to implement the application, that is, $100K+ errors during the implementation of a $50K system.
2. Enables the customer to get their highest priority functions.  The RAD process does this by allowing the customer to add requirements during each development cycle and reprioritize the requirements between cycles.  This produces a much more satisfied customers, than "Big Bang" processes like "the waterfall" process.
3. The development process is agile because it allows the customer to reprioritize their requirements.  In a book by Dr. Ralph Young on requirements, he cites a statistic that only 49% of the real customer requirements are known at the start of the development effort.  This RAD process allows the 51% to be incorporated, while keeping in budget using this repriortization function.  Since the product always meets the customer's highest priority requirements, they are much more satisfied with the result.
4. Focuses the developer on developing, gathering than documenting--this is what the software developer is good at, while documenting isn't.  Consequently, the developer's morale improved (which meant their effectiveness and cost efficiency improved) so both the quantity and quality of the products improved.
5. Minimizes the number of intermediate artifacts, that is, it eliminates the need to create documents like the Program Management Review (PMR) reports, status reports, design documents, Engineering Change Orders and so on.  This greatly reduces the need for developers and the program manager to create much of the documentation of other formal development processes.  In fact, as I created it, this RAD process eliminated the need for PMRs, because the customer was involved with the development team on a weekly basis.  And since the process assumed "that the program/project doesn't have all of the requirements at the start and enables the customer to add and reprioritize the requirements throughout the process, the resource requirements are kept level.  Therefore, from a programmatic perspective, this is a Level of Effort.  This means the same amount of money is spent each period; which reduces management of cost to nearly zero.  And since the implementation schedule is based on an unknown set of requirements, there is no way for a program manager to really create a schedule at the start of the effort and keep to it.  Instead, there is a monthly agreed set of requirements that must be fulfilled in development and rolled out--which by the way works, but which makes transaction managers and finance engineers paranoid--because they have so little CONTROL.  However, this process produced products of both a higher quality, and much greater quantity, and much more satisfied customers than the previous CMMI Level 3 processes.
6. Like XP, I used Use Cases to gather requirements.  A Use Case is One way for one type of actor (either a person in a particular role, or another application) to use the application under development.  In my experience and in the experience of many of the 50+ systems engineers that I coached or mentored, or otherwise trained that Use Cases were the best way to gather requirements because they could be documented in ways the meant something to the customer (they can be documented to use the customer's own words); most other ways, like "shall statements" communicated much less because they are a transliteration of the customer's requirements.  Additionally, in general, they communicate as much, if not more to the developer.  Since the goal of requirements identification and management is to "communicate the customer's requirements to the developer and to ensure that these requirements are met", using the Use Cases, as recommended by XP proved both highly effective and cost efficient.

The net of all this is more satisfied customers, happier developers, and much more effective and cost efficient processes and products--in this case a real live "Win-Win" situation.  One consequence was that by 2005, literally, hundreds of small and medium (and even a few large) effort adopted this RAD process--and it proved highly successful within its domain (software development).

Limitations of Rapid Application Development (RAD) Processes

The key limitation of the RAD category of formal processes (processes that meet CMMI Level 3 practices) is that they are strictly for software development.  Formally, it is not possible to use this process for software or hardware integration, and for linkage to systems external to the organization, those activities that are most likely to occur in today's SOA and cloud-based environments.  Since most organizations purchase software and integrate it into systems rather than develop their own, a short cycle, agile implementation and transformation process is needed for such efforts.  This process should be the equivalent  for integration of the RAD process  that I designed and documented for software development.  In fact, it should have the same advantages as the RAD process, but applied to integration and tailoring of systems.

The Generalized Agile Development and  Implementation Process

Starting in about 2005, I gave thought to this Generalized Agile Implementation and Development (GAID) process would work.   This is a notional design and the challenges in designing such a process.  The notional design consists of four loops.

1. The Requirements Identification and Validation Loop (Approximately 3 Months).  The purpose of this loop is to work with the customer to identify their requirements (the Systems Engineering Role), to decompose and derive the functions, to structure the functions, to allocate functions to components, to determine the acquisition method for components through trade off studies (make/buy/rent) (all in the System Architect role), and to validation that the product meets the customer's system requirements for this loop and all prior loops.  Additionally, each three month cycle allows for the acquisition and installation of any needed software, hardware, and networking components.
2. The System Assembly/Implementation and Verification Loop (Approximately 1 Month).  The purpose of this loop is to assemble the various sub-assemblies (either those developed or those acquired from a supplier) and to verify that they meet their allocated functional and design constraint requirements.  Additionally, this loop enables the customer to see the overall progress of the product and suggest minor changes that will enhance the utility of the product for the customer. For example, this might include the placement of various data windows on a particular display.
3. The Sub-system Design and/Assembly and Verification Loop (Approximately 1 Week).  The purpose of this loop is to ensure that components perform the functions specified in the component requirements to the level of dependability required and that the interfaces among components match for integration.  Additionally, this loop give the customer a first loop at the components, as they are being implemented, which means they can communicate changes they to like to see to the designers before the product start full up assembly and testing.  This greatly reduces the cost of the changes.
4. The Component Construction and Verification Loop (Daily).  This is the loop where all of the detailed design, development, and COTS product configuration takes place.  While this loop might strike the reader as silly, all loops including this consist of engineering functions with a single program management meeting at the start/completion of a loop.  In this loop, it is a 15 minute long "stand up" meeting.  The reason for the meeting is to identify risks and issues, and to have the implementers set aside time to reduce the risk or close the issue before the next stand up meeting.

If this works like the RAD process I developed worked (and it should), after the initial period of learning the process, both the customer and the implementers will find it much more straightforward, simpler, and easy to use, while producing much more effective and cost efficient products.