Changing the Congressional Budget Office to the Congressional Enterprise Architecture Office

The Current Mission of the CBO
Currently, the mission of the Congressional Budget Office (CBO):

"is to provide Congress with objective, timely, nonpartisan analyses needed for economic and budget decisions and the information and estimates required for the Congressional budget process" [from CBO TESTIMONY Statement of Robert D. Reischauer Director Congressional Budget Office before the Joint Committee on the Organization of Congress,[the] Congress of the United States]

The director broke that into three operating strategies:

1. Helping the Congress formulate a budget plan;

2. Helping the Congress stay within that plan; and,

3. Helping the Congress consider policy issues related to the budget and the economy.

The Problem with the Current Mission

This Mission and Strategies is part of the economic, political, and social problems currently facing the United States, and, potentially, a source for the solution of those problems. The reason that the CBO is part of the problem is that finance engineering has influenced Congress to emphasize the "financial" part of an overall Enterprise Architecture. That is, Congress proposes functional and component changes to the US Federal Government and the CBO responds with an analysis, which Congress can choose to spin-doctor to its political purposes. Consequently, Congress can choose to support any industry; examples include agriculture (subsidies) and housing (mortgage deductions, etc.), gambling (gambling deductions), and so on.

A Solution the Congressional Enterprise Architecture Office

As I demonstrate in my book, Organizational Economics: The Formation of Wealth, the body performing the controlling (see IDEF0 post) and governing functions of any organization has three Missions, Security, Standards, and Infrastructure.  This is particularly true of any organization that has a spatial domain.  These missions appear in the Preamble of the US Constitution and throughout that document.

Given these three high-level missions, and my discussion of the role and responsibilities of the Enterprise Architect (as a sub-discipline of Systems Engineering), what the US Federal Government needs is a real implementation of the FEA Framework and a formal Enterprise Architecture process.  This process aligns the departments’, agencies’, and other organizations’ of the Federal Government with the three high-level missions of government. 

Additionally, Enterprise Architecture proposes where develop, transform, reform, end or otherwise change the organizations’ missions, strategies, processes, and tooling.  For the US Federal Government, (or any other organization of this scope and size), the EA process must be recursive, but traceable and integratable.  The CBO is in the position with some of the responsibilities for doing this. 

Why not have Congress empower them as the Congressional Enterprise Architecture Offiice?

The Cost of Rockets Built by NASA: Waterfall Process vs Short-cycle and Agile Processes

Shortcomings

This post is really not about the shortcomings of NASA, it's more about the inevitability poor, high cost deliverables when a) an organization looses its focus because of a constantly changing Vision and Mission; b) a development process for a complex system is stilted by formality, documentation, and transactional management; c) the development process assumption is that all of the requirements and risks (unknowns in the design) must be known before the next step in the effort starts.  Unfortunately, NASA is a good example.

NASA and the Waterfall Development Process

NASA does not have a clearly defined mission.  Instead, it has three poorly defined missions.  The first is to improve aircraft.  Currently, I and many others would expect that the Mission statement might be, "Research methods to improve aircraft fuel efficiency"; but apparently not.  From the sales to American Airlines, the Eurobus aircraft has much better efficiency than the Boeing aircraft. 

The second mission is to perform space science, in particular astronomy.  The consequence is that the astronomers have taken over NASA as their personal toy-maker.  While they are attempting to answer some big questions, it is not clear that these questions need to be answered by this generation of astronomers.  In actuality, they are racing each other to get into the "history of science" books.  The value of that work is not readily apparent except in the very long term.  Further, much of it could be better accomplished from space stations.  And while it is interesting to explore Mars by robot, it would be much more interesting to have humans explore and colonize the moon. 

The third mission of NASA is to make it possible to colonize space.  NASA has fallen down completely with this mission.  Over the past twenty years, NASA has been the poster-child for the highly formal waterfall process for development of new systems.  It's process demonstrates what happens when transactional and bureaucratic management takes on development efforts; it's programs fail to deliver.  It seems to me, there are two interconnected reasons for this.  First, they've lost their Big Hairy Audacious Goal (their BHAG, see Jim Colins' book Built to Last for a discussion).  Second, they've tied up their innovative thinking, designing, and developing in a web of red-taped processes and procedures. 

BHAG and NASA

Elton Musk, the creator of PayPal and the Tesla Motors, has also created SpaceX.  The reason that Mr. Musk invested in SpaceX is that he

"...believes the high prices of other space-launch services are driven in part by unnecessary bureaucracy. He has stated that one of his goals is to improve the cost and reliability of access to space, ultimately by a factor of ten." [see SpaceX]

First, this quote demonstrates that SpaceX has a BHAG, "...to improve the cost and reliability of access to space...by a factor of ten."  In fact this is a very ambitious mission for a small organization, yet it seems to be one at which they are succeeding.

NASA had such a Mission, "...to put a man on the moon before the end of the decade."  That's a Mission is short, clear, and measurable, the three criteria of a good mission statement.  NASA performed admirably.  NASA performed the Mission using a strategy of short cycles, each building on the previous.

However, once, it had achieved its Mission, the NASA leadership assumed that colonizing the Moon was its next Mission, but LBJ had bigger plans, the Entitlement Programs, which have saddled the United States with unimagined debt--predictably; at least I could see it coming in 1965.  One of the first results of the increased debt was cutting the Moon landing program was shut off completely.  In substitute, because NASA has a political constituency, the Mission was to "build" the Skylab on the cheap.  They partially achieved this mission by using components from previous manned space programs.  Then NASA was charged by the US Government with creating a Space Transportation System (STS) that would significantly lower the cost to boost a ton of materials or people into low earth orbit.  Initially, this was couple with constructing a US space station in low earth orbit, similar to the one built of the USSR.  This was a fairly ambitious effort, but no where near as ambitious and audacious as exploring and colonizing the Moon.  The result was, in 1971, a Mission to construct the space shuttle and, in 1984, a Mission to construct the United States space station, Freedom.  As NASA moved from the mission to the Moon to subsequent missions, they were hobbled with the "politically correct" requirements.  These includes ensuring that states of politically powerful Senators and Representatives got contracts to firms in their districts, ensuring the minority owned, female owned, veteran owned firms got a certain percentage of the contracts--as mandated by Federal Law.  From personal experience on the Freedom effort, I can pretty much guarantee these mandates added a minimum of 10 percent to the cost of the shuttle and space station efforts--probably much more when added to the second reason for NASA's failure.  This was a big part of the reason that President Clinton cancelled the US space station program.  Since, NASA has had several manned Missions, but the US Congress has not been on board, so these have not been adequately funded--using the NASA processes (which are very financially wasteful).

Short Cycle Versus Big Bang in Aerospace Development and Transformation

The second reason NASA is failing is their choice of processes.  Since the Apollo Program, NASA has become continuously more bureaucratic, with transactional management and processes rather than transformational management and processes.  Their formal processes are based more and more on the waterfall process with all of the Program Management overhead, in terms of intermediate artifacts--again, increasing intra- and inter-organizational friction and pushing up the cost and out the schedule of the program.

Part of the problem is that the waterfall process assumes that "all requirements are known at the start of the effort" and that "risks (unknowns) can be turned into known through schedule-based invention and innovation", that is, invention and innovation can be performed on a timeline.  Both of these are heroic assumptions and were proven false time and again in the NASA Programs.  However, transactional management and finance engineering militated against it because those assumptions allow them to "control" the effort.  The consequence has been, since the Shuttle Program, there have been no successful development of systems to deliver materials to space; there have been no lessons learned about how to build better shuttles, or even how create better heat shield for the shuttle.  Instead, to "protect" the personnel, these transactional managers have added more audits and check lists and redundant check lists.  And one administration after another cut off one of the key technical centers for real innovation, which from work in the 1960s, led in the 1990s, to advances in computing, the Internet, medicine and so on; instead, choosing to expend the funds on entitlement handouts. ["Feed a man a fish and he satisfied for a day, teach a man to fish...", do research on fish to ensure their survival and you feed generations.]

The results have been similar in character to what Boeing found.  In selling the 777 to Japan, they guaranteed that the Japanese aerospace industry would have a work share.  However, before they let the contract for a tail surface, Boeing asked the Japanese firm for a "test article".  When they received the tail surface, it met all of the specifications and met them more closely than the Boeing manufactured tail surfaces.  However, they had a mystery, the Japanese tail surface weighed ~150 lbs less than the one Boeing built internally.  On taking the Japanese test article and an "identical" Boeing test article apart, the Boeing engineers solved the mystery, the Boeing test article had ~150 lbs of shims to bring into conformance, while the Japanese built it to specs, rather than shimming it to specs.  Since reducing aircraft weight is one key to reducing fuel costs per passenger mile, the shims all over the aircraft were a big issue.

I've found that many program managers of governmental contracts require program management shims.  When a program gets into trouble the first thing they require is more status, more PMRs, and more detailed schedules.  All of this requires formal replan documents, which takes a significant chunk of the program's budget.  The is what has happened to the NASA man-in-space programs and is happening in most other federal, state, and local programs.  And again, it is exacerbated by the great additional friction of federal "fairness" policies, which, as noted earlier, direct funding to organizations with certain types of ownership, regardless of competence.  These are in fact, blatant attempts at quick cultural change.  They may be somewhat successful in meeting their mission, but they have wrecked programs like those of NASA.  This is noted in the quote from Mr. Musk, "the high prices of other space-launch services are driven in part by unnecessary bureaucracy." 

Since the aerospace industry started just over 100 years ago with a flight of 120 feet and in slightly over sixty years, was flying at supersonic speed (in fact Clarence "Kelly" Johnson was one designer who 1933 helped develop the Lockheed Model 10 Electra and finished his career by designing and developing the SR-71 Blackbird, flying at Mach3+) and had reached the Moon.  How did these inventors, innovators, and designers do it?

The Wright Brothers versus Langley

Invention and innovation in the aerospace industry is rife with examples of both short cycle and Big Bang development and transformation, but especially development, starting with the Wright Brothers.  Starting in 1899, the Wright brothers started active work on developing a "controllable heavier than air craft" (their BHAG).  Prior, they had identified that building a "stable" aircraft, an aircraft that would fly straight in still air, as all previous aircraft pioneers were doing, and then adding control functions would not achieve the goal of heavier that air flight.  Instead, they decided that they needed an unstable aircraft, one that the pilot would actually have to fly.

[Sidebar: This concept is found in sailboats as well.  Cruising sailboats have very long keels and small rudders.  The long keel provides stability for going in a straight line.  In fact, frequently, the crew can leave the helm unattended for 5 to 10 minutes without have the boat change direction by 5 degrees. on the other hand, racing sailboat have to be maneuvered before and during the race.  Therefore, they have short, deep keels and large rudders.  This makes then inherently unstable for holding their course.  In fact, my boat, which is a combination racer/cruiser will wonder 30 degrees or more off course in a matter of seconds; but it will turn around in practically its own length.  This shows the difference between stability in instability.]

Consequently, in 1899 the Wright Brothers started to build a series of controllable and maneuverable kites.  By 1901, they felt they had a flyable design, but it behaved poorly when compared with the predictions of the research current at the time.  They felt that there was a risk that the research was wrong.  Since there was no way to avoid, transfer, or accept the risk and be successful, they instituted a mitigation plan.  Initially this consisted of attaching small model airfoils to a bicycle and pedaling as fast as possible.  While the data from this "exercise" showed that the data, current at the time was not reliable, they had to invent the wind tunnel to get accurate data.  Again, they went through short-cycle experiments on very small wind shapes and developed the first set of highly accurate data on airfoils. 

With this data, in 1902, they were able to build a guilder that was fully controllable.  But after a series of test flights, the 1901 manned kite also showed a need for a vertical tail, and again, after a series of short cycles, they implemented the vertical tail surface and allowed it to move.  This too, was incorporated into the 1902 guilder.  The 1902 guilder proved that they were ready to add an engine to create the first fully controllable aircraft.

In 1903, the Wright Brothers found two more unknowns.  First, they needed a light-weight engine and second, they needed to determine what shape of propeller would produce the most thrust.  With respect to the light-weight engine, since they couldn't buy one, they built one with the aid of an employee/team member.  With respect to the prop, the conventional wisdom of the time said that the shape should be much like a ship's propeller blade.  However, the Wright Brothers found that thinking of a propeller as an airfoil that spins produced much more thrust.  Thus by the end of 1903 they were ready and flew the Flyer 120 feet on their second attempt (their first, the previous day had just gotten of the ground when they crashed it).  As the gained experience and confidence they flew farther, to 852 feet.

In 1904, they built a new Flyer based on their experience.  They flew in Ohio, close to home.  There they gained more piloting experience as well as refinements to the design, again in short-cycles, mindful or new requirements and risks as they came along.  They considered the 1904 craft merely a design step and at the end of the 1904 flying season, salvaged it, and burned the leftovers.  By 1905, they finally designed a Flyer that was truly usable, at least for the time and it was from these aircraft design that all useful controllable heavier than aircraft can be traced.

I left out two parts of the story.  First, they spent ~$1000 to create their aircraft (excluding their own time, as an investment) and Second, they were in a competition with Samuel Langley.  Samuel Langley was a well connected researcher that, having shown was funded by the US Government through the Smithsonian Institution (Museum) to over $50,000.  The reason that Langley was funded was that he had created a model unmanned aircraft that flew 3/4 of a mile under ideal conditions; it was stable in its design. 

[Sidebar: Again, as noted earlier, stability allows an aircraft to continue in a straight line.  It does enable the course of the aircraft (or boat) to turn easily.  This is a good analog to lean versus agile processes and waterfall versus short-cycle processes.  To create a lean process, the process engineer looks for waste.  While there are many forms of waste, one is "unnecessary" activities and procedures; and there are many of those, as well.  reducing these enables to process to flow more quickly to "the solution" or "the deliverable".  Therefore, this is a very stable process.

However, if external conditions effecting the lean process change, the process has no ability to effectively or successfully respond.  The same thing happens with the waterfall process.  The process is based on the assumption that "all of the requirements are known up front".  Then the Program Manager can plan out the effort (sometimes to the bathroom breaks) and manage to the schedule.  Unfortunately, this assumption is entirely false.  Consequently, projects using the waterfall process have to undergo much replanning, Engineering Change Orders (ECOs) and so on (that keep Program Managers employed).  This too means that waterfall process-based programs are stable, but brittle in that changing direction, however, slight, requires major effort.]

Langley scaled up this design, changed the powerplant from steam to gasoline, and built a bigger barge; but apparently not big enough since his craft hit something on the barge and immediately crashed.  As several writers has pointed out, even if the aircraft had flown, Langley did not have a good plan for his pilot to land his craft.  Since the pilot wasn't hurt, this attempt ended much better than it might have even if it were "successful", that is, flown successfully only to have the pilot killed at the end of the flight.  So Langley rebuilt his craft again and again it crashed.  While Langley was trying to assess the results and the damage, the Wright Brothers flew their aircraft.

However, because of the secrecy in which they flew, it wasn't until 1908 that they demonstrated their accomplishments to the world.  They flew both in the US and Europe, literally and figuratively flying circles around their competitors; they could control their agile aircraft while their competitors could fly their stable aircraft in straight lines.  The Wright Brothers succeeded because they understood their Vision as flying (controlling) a heavier than air craft where they wanted.  They spent from 1899 to 1903 getting the basics right in a series of short cycles, then refined the design in a series of short cycles.  On the other hand, Langley and a fair number of European competitors had a Vision of a stable heavier than aircraft, which they then hoped to find means to control.  Consequently they built small stable models, then scaled up the results in a waterfall like process--and failed.

Robert Goddard and Wernher Von Braun and Short Cycle Development

Rocket science too, used short cycle development, with little formal program management.  The father of modern rocket science, Dr. Robert H. Goddard worked to a single Vision (his BHAG), manned space travel virtually his entire life.  To realize this Vision, Dr. Goddard started by creating strategies for getting into space.  This led to two landmark patents in 1914 of 214 that he was granted.

By 1915, Dr. Goddard was working on meeting one of the requirements derived from his strategies, creating an engine with sufficient thrust to get into space.  He found that the rocket engines of the time actually converted only 2 percent of the energy they produced into thrust.  When he applied steam turbine nozzle technology to the nozzle of a powder rocket, he could the conversion rate to above 40 percent.  However, the total thrust produced by powder was not great enough to achieve his Vision.

Therefore, he started to investigate other fuels.  At the time, liquid fuels had the greatest chance of producing the needed thrust to weight ratio (and had an added advantage of being controllable, that is, reducing the flow of fuel to the engine reduced the thrust, while increasing the fuel to the engine increased the thrust).  In 1926, Dr. Goddard is credited with the first liquid fuel rocket to lift off, after working a number of years, with many tests on this engine.  Like the Wright Brothers, the first flight Dr. Goddard's flying machine was very short in both time and distance, but it proved that liquid fuel could be used.  And like the Wright Brothers, he then both continued to refine the engine design, and went to a new challenge.  In the case of the Wright Brothers, they learned to control their aircraft first, then to power it, while Dr. Goddard first learned to power the craft and then spent a significant amount of time learning to control it; even while refining the propulsion system.  Additionally, Dr. Goddard spent a good deal of time on raising funds for his research and development efforts.

By 1937, Dr. Goddard had flown a number of his rockets.  While none of them were particularly successful, at least to the general public, they did draw the attention of the rocket science community around the world,  and while Dr. Goddard, like the Wright Brothers, tended to be secretive, he did share technical information with others in the community.

In Germany, one who paid attention was Dr. Wernher Von Braun.  Like Dr. Goddard, Dr. Von Braun had had a Vision (again, his BHAG) of human space travel since he was a child.  By 1930, Dr. Von Braun had joined the "Space Flight Society" in Germany and started working on liquid fuel rockets; this is what brought Dr. Goddard's work to Dr. Von Braun's attention.  By 1934, the work of Dr. Von Braun came to the attention of the Nazis.  When in 1934 he was ready to publish his doctoral thesis, the Nazis classified it.  They then offered to support his work.  Obviously, the Nazi Vision and the Mission for rocket technology differed wildly from Dr. Von Braun's.  Still, both Dr. Braun and the Nazis wanted to develop rocket technology (and there was an implied, but very real, threat that non-cooperation would have dire consequences).

The net result was that Dr. Von Braun built on Dr. Goddard's work during the 1930s and until 1939, asked technical questions of Dr. Goddard from time to time.  During this time he first created the A-1, then A-2 and A-3 series of rockets in a series of short cycles.  Built on these prototypes, the A-4 series first flew in 1941, but due to reliability problems and interference from the Allies, it was not put into production until 1943--the A-4 was then redesignated as the V-2.  This was the first rocket to demonstrate the potential of space travel as well as the first intermediate ranged rocket.  Both Dr. Goddard and Dr. Von Braun confirmed that the V-2 design was actually a refinement of Dr. Goddard's original work.

While it did not affect the course of WWII as Hitler hoped, it did get the Allies attention. At the end, both the USSR and the US captured some V-2 missiles and while the USSR captured some of the German rocket scientists, the majority followed Dr. Von Braun into surrendering to the US.  In the period of the late 1940s to 1957, the USSR worked in secret on derivatives and upgrades of the V-2, while the US made minimal use of the technology and expertise they had available.  Only when the USSR launched Sputnik in 1957 and several Vanguard rockets failed, spectacularly, was the Von Braun team asked to put a satellite into space.  Basicly, this team rolled out the Redstone rockets they had prepared for launching a satellite three years before (They had not been allowed to launch it because it would have been politically incorrect to do so).  This was Explorer 1.

Dr. Von Braun went to NASA and continued his work.  NASA itself, in the early days created a short cycle plan to get a manned landing on the Moon by 1969.  It started with the Mercury Program, which "simply" got men into space, then graduated to Gemini, which verified that vehicles could meet and mate in space (necessary for the method the US chose to go to the Moon).  And finally, using Dr. Von Braun's Saturn 1 and Saturn 5 rockets, move in relatively short cycles to the moon landing.

Notice that from 1915 to 1969 and actually to 1972, the space/Moon landing program was not a single Big Bang process, but a series of small steps, each building on the previous.  This is the only type of process that would work for a Moon landing.  However, since, NASA has abandoned this process in favor or working the way other government departments work, with much red tape, little flexibility, few successes or deliverables...as described so succinctly by Musk.

Final Thoughts

Short cycles allow for experimentation, the waterfall process doesn't.  As noted by Dr. Goddard,

"It is not a simple matter to differentiate unsuccessful from successful experiments. . . .(Most) work that is finally successful is the result of a series  of unsuccessful [short-cycle] tests in which difficulties [risks and issues] are gradually eliminated." (Written to a correspondent, early 1940s, See Lehman, Milton, This High Man: The Life of Robert H. Goddard [N.Y., N.Y.: Farrar, Strauss, and Co., 1963], p. 274.)

 Experimentation is not economic in the calculations of finance engineering, since most experiments are failures, and cannot be preplanned.  The first generation of NASA personnel and management understood this, but later generations of NASA management have not.  Since the Moon Mission, NASA has lost its way.  First, NASA's Mission keeps changing with the political winds, both within government and within the "scientific" community.  There has been no clarion clear Mission since the Moon landing Mission.  In the last 10 years, NASA was given a Mission to set up a Moon colony, a Mars colony, then a Moon colony (again), then a manned Mars mission.  Each of these had different strategies for achieving their mission.  From what I've read, none of these include short-cycle processes...that's too expensive.  Instead, they have been Program Management and Finance Engineering controlled waterfall-like programs.
[Sidebar: If it were me, I would choose a Mission to Mars by way of the Moon.  The reason being to learn more about space travel and colonization of a planet in an environment where emergency and other short-cycle risk reduction flights would have a chance, rather than one that the "short duration" is seven months.  But, that's only my opinion.]

Unfortunately, as events over the past 10 to 15 years have shown, it's not only NASA, but the entire Federal Government that needs to overhaul its Missions, Strategies, laws, regulations, policies, and standards.  Given the two current diameterically opposed views of the roll of government (see my post on "The Purpose of Government" and linking posts, for my thoughts on the roll of government) the US Federal Government remains uncontrolled for all practical purposes.

Short Cycle, Agile, Level of Effort efforts, and Changes in Roles and Responsibilities

In a recent post I briefly discussed the changes in roles and emphasis when a development or transformation effort changes from a waterfall (Big Bang) effort to a short cycle-agile effort.  This post will discuss the topic in more detail in terms of a Short-Cycle, Agile, Level of Effort projects and programs.

Short Cycle

A development and transformation effort is a program or project that is changing some process, component, procedure, or tooling that supports an organization's Vision, Mission, and Strategies.  There are two types of efforts, Big Bang, and Short Cycle.  Big Bang development and transformation processes are straight line processes (see my post Product Architecture Thinking Versus System Architecture Thinking) in which there series of steps from requirements identification, through design, implementation, validation, and roll out.  It delivers the product in a single delivery--one Big Bang.

Alternatively, the Short Cycle (1 to 3 months) development and transformation process is a process that delivers the functional of the total product or system in small increments through a series of short duration development or transformation cycles.  Typically, the deliverable from the first cycle (a one month cycle) is "usable", after a fashion, and is called the Initial Operating Capability (IOC) of the system. 

I've found that the IOC will consist of the initial version of the system's access control together with a minimal set of input screens with associated data stores.  The reason for this IOC functional architecture is that security of an operational system is normally a "business" requirement; it's not wise to operate an open system.  Second, you need to "put garbage in" to "get garbage out."  If the team builds a report during the first cycle with no way to insert data to support the report, then the system is not operational in any sense.  However, if you build input screens with associated data store during the first cycle, then the customer can start inserting data during the second, while the team will likely be adding to the number of input screens and developing at least one report or information display.  My experience has been that it will take the customer about a month to get enough data in to have usable reports and output displays.  This means that by the end of the second cycle the system will be producing at least a minimum ROI for the customer.  This ROI will increase with each subsequent short cycle, which delights most customers.

Agility

The Agility of an organization  is defined as "its ability to successfully respond to unexpected challenges and opportunities".  The "successfully respond" phrase has two dimensions: making the right decision and making a timely decision.  Various components of the US military have a saying  "improvise, adapt, and overcome" that embodies the concept of agility.  This has long been a tradition of the US military.  For example, the Allies won D-Day in part, because they were much more agile.  That is, the NCOs, junior officers, and field grade officers improvised new tactics, based the manpower and equipment available, to achieve the initial mission of creating a beach head, when all of the plans for the invasion were breaking down--the capture of Omaha Beach is the seminal example.  On the other hand, the German forces followed their defense plans as closely as possible.  Consequently, senior officers were asking permission to move units to defend the coast.  By the time Hitler, hundreds of miles away, made the decision to release the panzer units, the Allies had enough units ashore to defend the beach head. 

The same definition of Agility can be used for a development or transformation process.  If a process is founded on the assumption that "All of the customer's System Requirements are known up front" then I would suggest that the process is rigid and inflexible--a totally brittle process.  That is, if new customer System Requirements are discovered as the development or transformation team is designing and implementing the system, then a formal Engineering Change Order (ECO) process (with contractual changes) is required.  All of this "administriva" costs time and resources that the team could otherwise use to create more of the product or system the customer wants.  Consequently, the customer will identify only those new requirements that will make the product or system completely unusable if not met.  This leads to poor systems and unhappy customers.  This is in contrast, agile processes based on "Not all the customer's System Requirements are known up front" so as to support the inclusion of new requirements as the system develops.

Level of Effort

Short cycle, agile development and transformation efforts require using the Level of Effort (LOE) pattern of contractual management rather than any other type.

What is a Level of Effort Project or Program?

A Level of Effort (LOE) program or project is one where the budget for the effort is spent uniformly across the duration of the effort.  Therefore, the amount of development or transformation work is uniform across the effort's duration.  This differs from the Big Bang style of development in that, normally, the Big Bang starts with a small team performing the initiation tasks, builds up the effort during detailed design and component and assembly verification and reduces the effort during post-roll out product or service support.  If the product has too many defects the PM as the ability to add personnel and use up the budget faster.

With a LOE project or program, the PM does not have that ability.  Instead, if the design team has under estimated the complexity or risk in meeting a requirement, they have the ability to refactor the requirement into two or more requirements.  They can then complete the first of these during the current segment and the others during later segments.  The fact that I'm using the term segment implies that I think that an LOE effort can only be successfully used with short cycle efforts; each segment being a cycle.  So, in short cycle, agile, LOE efforts, the System Requirements are the variable, while the budget and schedule are held as constants.

Why is it important?

Currently most efforts are still in the Big Bang style because most formal processes are of that type.  These processes include all of the required intermediate artifacts like a project plan, schedule, PDR, CDR, EWBS, IWBS, documentation for PMRs, ECOs, weekly status reports, and so on.  The reason I call these intermediate artifacts (or work products) is that they have no lasting value for the customer.

For example, since LOE projects and programs produces a uniform amount of output for each equal segment of these development or transformation efforts, there is little need for metrics like "Earned Value".  This is particularly true in short cycle development and transformation efforts where the customer can see and use the work products.  If properly performed at the end of each cycle, be it monthly or every 3 months, either the IOC version or an upgrade is rolled out.  It can be rolled out into the operational environment, or in some cases into a preproduction environment (an environment where the customer can use the new system in parallel with the old system).  Since the customer can use the IOC or upgraded system, what is the purpose of metrics like the "Earned Value" metrics (since the goal of the EV metrics is to show progress)?

Changes in Roles and Responsibilities of the Team

If, as I've experienced, there is little need for most intermediate artifacts supporting the PM procedures and methods because there is little need for those procedures and methods, then there must be changes in the roles and responsibilities of the program or project's team using a short-cycle, agile, LOE development or transformation process.

First, the process is short-cycle (one to three months) and agile ("not all the requirements are known up front"), so the process must have these two characteristics.  Having these two characteristics, immediately reduces the role of the PM, as discussed above.  No longer are all of the PM procedures  for a PDR, CDRs, EWBS, IWBS, PMRs, ECOs, weekly status reports, earned value reports, and so on necessary; in fact, they are counterproductive in that they reduce the effectiveness and cost efficiency of the effort.

Second, the emphasis is on creating or transforming a product or system to meet the customer's highest priority System Requirements, which may or may not be known at the start of the effort.  The last clause in the prior sentence is the high-level capability statement for agility.  This has two consequences.

• There is a requirement for capturing new requirements in every cycle of the effort.  This is within the role of the Systems Engineer.
• There is a requirement for the customer to prioritize the complete set of requirements at the start of each cycle.

These requirements indicates that the Systems Engineer's responsibility in identifying and managing requirements has increasing importance to any development or transformation effort.

[Sidebar:  There are studies to indicate that in software development efforts 10 to 20 percent of the software developer's effort is spent on functions not called for in the requirements.  Frequently, the customer asked for these to be removed, which costs more time and resources; and none of which produces value for the customer.  But this isn't only a problem for software developers today. A famous example is that at the start of WWII, the M-3 tank came rolling out of the factory with a police siren. No one could explain why since it wasn't in the requirements.  So, having the entire effort focus on the requirements will frequently greatly reduce costs of design and development as well as program management.]

Third, the development of functions or services are tied directly to the Functional Requirements, which link through a traceability matrix to the customer's System Requirements (see my post Types of Requirements for definitions) [though, in some smaller software development efforts, this level of decomposition may not be necessary].  Since, by the definition of a requirement, it must have a metric for when it is met, all verification and validation procedures and methods must be traceable to these metrics.  Additionally, with short-cycles the risk that the designers/developers/implementers will induce defects greatly increases.  An Induced Defect is one that is created in the process of a roll out or update of a system.  The key procedure to reduce the number of induced defects is regression testing--performing all of the same verification and validation procedures and methods used on prior releases.  Within short-cycle and agile processes, linking the V&V procedures and methods to the requirements and regression V&V emphasize both the importance of the requirements--therefore the requirements management system--and the role and responsibilities of the Systems Engineer.

Fourth, because, as discussed above, short-cycle, agile processes require LOE program management, the role of the PM is diminished further.  No longer can the PM "control" the effort by adding to reducing resources to a particular task.  Instead, the Systems Engineer must decide in each cycle how many of the highest priority requirements the team can meet during the next cycle.  Consequently, the PM is really only responsible for working with the suppliers and consultant (both external and internal to the organization), ensuring that the processes, procedures, and methods of the short-cycle, agile process are properly executed, and reporting results to higher management.

These changes in roles and responsibilities are a dramatic shift in the cultural of development and transformation.

SOA, The (Business) Process Layer, and The System Architecture

As I discussed in my posts The Paradigm Shift of Service Oriented Architecture and SOA in a Rapid Implementation Environment, there is a significant cultural/business/technical shift in thinking when an organization starts to migrate from a fragmented, monolithic IT architecture, or Object Oriented/Web Service using code development using COTS or custom developed applications to Composite Applications within an SOA.  This key shift is formal linking of the IT application with the business process instead of the functions that make up the process.  I discuss several of the advantages in Assembling Services: A Paradigm Shift in Creating and Maintaining Applications.

Process and The Assembly Line

While it may seem like much of a change to link the application to the process instead of the function, consider this concept in the context of an assembly line.  An assembly line is the tooling enabling and supporting the process of assembly.  In turn, a process is "a set of activities ordered to achieve a goal", the goal or operational mission for the assembly line is to assemble the product with no defects.  I suspect that all manufacturing engineers would agree with that goal, but understand that Murphy's law works on all assembly lines, so they attempt to minimize the number of defects, still... On an assembly line, as much as is feasible, all of the activities and their enabling and supporting tooling works in concert with one another, orchestrated by the manufacturing engineering team.  This team spends significant time and effort measuring the flows through the assembly line to ensure, a) that the line produces products with as few defects as possible, while b) producing those products as cost efficiently as possible (e.g., minimizing the work in process [WIP]), and c) meeting the constantly changing demand with as little inventory as possible (which constitutes operational agility for the assembly line).  Additionally, the manufacturing engineers want the tooling to be agile, so that as the components of the product change, with changes in customer's requirements (and wants), the cost of retooling is minimized.

Process and IT

Contrast that with the way organizations engineer IT.  From its inception as supporting single functions, like accounting and payroll, with card-based semi-mechanical "electronic computers" in the late 1950s, IT has focused on supporting individual functions.  By the early 1980s the result was apparent to anyone that developed or used applications; they were information silos, that is, the tooling supporting each function had associated data, but there were no interfaces between functions to enable activity 2 in the process to share its data with activity 3, for example.  During this time, I worked at a major university.  In one Dean's office, that I supported, the Dean's administrative assistant had five terminals on her desk.  Why?  For two reasons; she needed data from five applications to perform her job of supporting the Dean, and she needed output from one application to serve as input to another--she was, in fact, the interface between the functions.

Since this situation was intolerable and expensive for operating the functional applications, the developers/coders created interfaces as the customers identified the need (and had funding to provide for it).  Consequently, the application layer functional design went from a series of virtually unconnected silos of data and code to something approaching a picture of a bowl of spaghetti.  In one study I did measuring the cost of maintenance of each link in this "fragmented" architecture, I found that for one small but business critical process for one of the business units, the cost was approximately $100/month/link.  Considering that even in mid-sized organizations, if half the maintenance cost of what I found is representative (i.e. $50 per link per month), then even a mid-sized organization would be spending a significant portion of its IT budget on link maintenance.  This cost created great pressure by finance engineering on IT management to become more cost efficient. 

Again, manufacturing engineering provided the concept, Materials Resource Planning (MRP).  The MRP systems interlinked all of the data systems and IT functions supporting the manufacturing floor by integrating them into a single application, thereby eliminating all of the linkages among functions.  This concept was expanded into MRPII, and then into the Enterprise Resource Planning (ERP) systems, like the SAP and Oracle tools.  The architecture is call Monolithic because all of the functions are tightly coupled in a single application.

Within a couple of years system users found several issues with ERP systems.

• The ERP systems are difficult to tailor to support the organization's processes.  Typically, the suppliers of ERP systems build the functions to a group of standards that are used by most industries or most organizations within an industry.  If for whatever reason, a) the enterprise that is implementing the ERP does not have a process that uses those standards, then either the enterprise must change its process--unlikely--b) it must attempt to tailor the ERP system to match its process, or c) some combination of the two.  Generally, none of the alternatives are particular effective or popular in large organizations with a diverse product mix.  The result is that either the initiative fails completely or that the personnel of all of the business lines work much harder and longer (and possibly create "shadow systems") to overcome the limitations imposed by the ERP system.
• The Systems are difficult to update.  Frequently, the software supplier implements the next version of a package  the equivalent of installing, configuring, and tailoring the original system.  The reason is that they have updated the technology, perhaps the functional architecture (design) sufficiently, that any bolt-ons, add-ins, or other tailoring will not work with the new version.
• The Systems are operationally inflexible.  That is, they are not agile; able to respond successfully to unexpected challenges and opportunities.  Because the tailoring required for an ERP system can be major, the costs and time involved becomes a significant item in the organization's budget.  Consequently, the time to change becomes long and the cost of change becomes high.  This leads to the problem that IT systems have become an impediment to change rather than an enabler, as studies by Gartner and Forester have shown.

The advent of Object Oriented Programming and its successor, Web Services, enabled organizations to partially address the issues above.  As discussed in my post SOA and a Rapid Implementation Environment, OOD design enables a developer to treat an application as a set of objects.  If these objects use Web Services Standards (either WS or JSR) for their interfaces, then they are Web Services.  In effect, this transformed an application into a set of components (classes, or Web Service components) that could be lashed together to create the application.  The advantage of the OO software architecture is that each of the objects is a small code module that could be updated or replaced with new technology and as long as the objects external interfaces and the resulting functions remained the same, the application would operate in the same manner.  This change met two of the three issues noted above; updating of the application's technology is possible without a complete re-installation and configuration of the application and initial tailoring of the application is much more feasible.

While this tailoring of an application made up of objects or Web Service components does address the agility issue, it is only in a very minor sense.  Since Object Oriented Architecture only enables an implied workflow, rather than making it explicit and independent of the Service Components, it would require reprogramming to change in response to unexpected challenges and opportunities.

Process, SOA, and the System Architect

 As discussed in my post SOA and a Rapid Implementation Environment, SOA formally separates the process flow from the functions.  This formal separation means that the organization can have its transformation teams restructure and order the functions without reprogramming of the functions themselves.  Instead, they can simply be reassembled (see my post Assembling Services: A Paradigm Shift in Creating and Maintaining Applications).

Formally and explicitly assembling Composite Applications using orchestration and choreography enables the System Architect to model the organization's (business) process for both effectiveness (in achieving the organization's mission) and cost efficiency and then to model the Composite Application linked with the process to determine its ability to optimally enable and support that process in both an effective and cost efficent manner.  Consequently when the organization's processes require change in response to changes in the organization's mission, strategies, the organization's environment, or technology, the System Architect, together with the rest of the transformation team may only need to change the process flow Component,  may need to add or update certain Service Components supporting functions, or both.  Still, this will make much more agile support for the processes.  This links the SOA-based Composite Application directly to the organization's process.

There is an additional benefit to separating the process flows from the Service Components.  When the organization's automated business rules change in response to changes in policies and standards (which in turn should be in response to changes in Governance), the System Architect can very quickly evaluate the change by modeling the process and Composite Application to understand the consequences of the change.  In the best situation, the System Architect would model all affected applications before any proposed change in policies, standards, or business rules is put into place because this would enable both the Governance and Policy Management teams to understand the consequences and results of the proposed change before those changes become unmanageable negative externalities.

Obviously, this is a significant shift in the organizational culture.

SOA, the User Interface Service Components, Apps, and Validation before Registering

System Architecture and the User Interface
In an article I wrote for The Northrop Grumman Technical Review Journal, entitled Service-oriented Architecture and User Interface Services: the Challenge of Building a User Interfaces in Services, I described five functional categories of user interfaces that a system architect might use to enable a user to employ  for an SOA-based Composite Application.  These included: Thin Client, Portal, Rich Client, Smart Client, and Rich/Smart Client.  I describe the characteristics sub-functions of each such that a system architect be able to determine which would serve the Composite Application's user best. 

In the article I linked the user interface categories to three categories of users (not my categories, but I could not track down the source), informational, transactional, and authoring.  An informational user is one who uses data and information in an ad hoc (through search and discovery methods) or regularly to retrieve information from one or more sources.  The data flow is primarily to the user.  Many commercial cable companies set up their networks for this type of user; with data rates twice as fast to the customer of the data and to the supplier of the data.  A transactional user is one who repetitiously inserts and/or retrieves the same type of data from an application.  For example, the office administrator in a doctor's office makes appointments, that is, he or she looks up on a calendar for dates and times that are open and then inserts an appointment.  For organizations, including businesses, the vast majority for the uses of data and user interfaces are of this type.  Finally, there is the "authoring" use.  This customer function is the most creative and would include: programming, document, video, and audio file creation and editing, engineering, verification and validation, and enumerable other content creation.  Creating a user interface for the authoring category is the most complex because this service links to the most complex functions of an application.

There were two reasons for writing this article.  First, for SOA-based Composite Applications, to demonstrate that the user interface could be built in Service Components, as there was a good deal of discussion between myself and my colleagues on this point.  Second, to demonstrate the patterns that a System Architect can use in the activity of decomposing the customer's system requirements and deriving the functional requirements, which is a key step in the system architecture process.

An Example

While a functional design for the user interface Service Component may seem like a semi-useful abstraction, Apple seems to have intuitively understood the concepts very well.  For those versed in SOA, an Apple "App" is simply a Service Component.  It performs a single function or a small group of closely coupled functions (in a system architecture sense), for the user.  Apple, and subsequently many other suppliers of wireless user interface devices, are now selling these User Interface Service Components.  Most of them are for the informational user, if you can call streaming a movie informational--the functional concept is the same.  Likewise stock quotes, news feeds, the weather radar real time image, the restaurants closest to the user (with customer critiques), and earth images of the user's current location (i.e. navigational maps) are all informational, that is, they are likely used on an as needed basis, or semi-regularly, to gain data and information about what is going on in the environment that affects them.

Even Apple's iTunes and Apps store are informational in that they link to discovery and download functions (services) on the server-side of the system.  There are Apps, like playing music, games, and currently, the calendar, that are standalone applications (functions), but even the camera function (service) is loosely couple with the e-mail user interface Service Component.  Consequently, the system architecture of the new mobile devices is, in fact, SOA-based, they just don't want to call it that.

As further proof, Apple is treating their Apps the manner a Service Component supplier should be treated.  Apple has significant verification process to ensure that Apple's policies and standards are met, before the App is offered to customers.   That is, Apple is verifying the Apps, the way all Service Component supplier should verify Service Components, especially, a user interface Service Component.  These include meeting security and dependability standards, as well as having a Service Component Description.  The Apps Store serves as the Service Component registry and discovery function in the SOA-based architecture.

Consequently, this is an excellent example of an SOA-based architecture (though it really doesn't use Web Services).

For more on SOA see SOA in a Rapid Implementation Environment, Enterprise SOA vs Ecosystem SOA, Private Cloud Computing vs Public Cloud Computing, Choreography vs Orchestration: Much Violent Agreement, and SOA Orchestration and Choreography Comparison.  For more on System Architecture see SOA in a Rapid Implementation Environment, Enterprise Architecture and System Architecture, and Lean or Agile Enterprises and Architecture.

Enterprise SOA vs Ecosystem SOA, Private Cloud Computing vs Public Cloud Computing, Choreography vs Orchestration: Much Violent Agreement

Ownership Domain Confusion Reigns

There seems to be a great deal of confusion and much violent agreement regarding terminology within SOA and Cloud Computing.  In particular there is confusion within the SOA community regarding the difference between Choreography and Orchestration (see my post SOA Choreography and Orchestration Comparison), and much confusion about Enterprise SOA versus Ecosystem SOA and how they relate to Private versus Public Cloud Computing.  Actually, all of these relate in a very straightforward manner, once the definitions of each is understood, and once the proper architectural framework is applied.  That framework is based on the Ownership Domain of the Application.  The Ownership Domain is defined as the person or persons, or organization or organizations that either own and maintain the application code.  If there is one owner/steward for the installation, configuration, and maintenance of the code, then the application resides in a single ownership domain.  If  the code runs on an IT infrastructure with a single owner/steward, then it is running in a "Private Cloud", if the code runs within two or more IT infrastructure ownership domains, it is running on in a "Public Cloud".

Let's take a look these concepts using this Ownership domain framework.

Single Ownership Domain Composite Applications

An Enterprise SOA-based composite application is one where all of the application's Service Components are within a single organizational ownership domain.  Characteristics and attributes of the Enterprise SOA-based application include:

• It supports a high-value producing process.  Organization's have two types of processes, those that have some competitive advantage, which produce value for the organization and those supporting processes that are required by the organization to "stay in business".  These are processes like accounting, HR, and finance.
• Enterprise SOA uses:
• Orchestration to assemble the Composite Application because all of the linkages among the various Service Components are controlled by the personnel within the ownership domain.
• An ESB to interconnect the Service Components during Composite Application operation.  In fact, the key characteristic of an Enterprise SOA and an Enterprise Composite Application is that it uses the ESB within a single ownership domain.  (while most ESB suppliers add many functions to their ESB product offerings, functionally, the ESB is simply an application layer communications function.)
• An internal UDDI as the repository of Service Component descriptions (for those using Web Services, this would be the WSDL repository.
• A Private Cloud enables and supports Enterprise SOA.  The caveat is that only the top two layers of the Cloud architecture (the SAAS and PAAS) are required to be within the single ownership domain.  However, by definition, an Enterprise SOA-based application can never operate within the Public Cloud.

Multiple Ownership Domain Composite Applications

An Ecosystem SOA-based composite application is one where the application's Service Components operation across multiple organizational ownership domains.  Characteristics and attributes of the Ecosystem SOA-based application include:

• Typically, it supports, organizational supporting and capacity value producing processes.  These are all processes that are not part of the core competence of the organization, the ones that produce the organization's competitive advantage.  Personnel within an organization may also implement an Ecosystem SOA-based application for applications support ephemeral processes, like a research effort.
• Ecosystem SOA uses:
• Choreography to assemble the application since choreography is much more dynamic during the execution of a Composite Application.  For example, if a Composite Application in one ownership domain uses a Service Component in another, and the owner of the second decides to change the location of the Sevice Component, without the use of Choreography, the Composite Application is very likely to fail from not finding the Service Component.
• the UDDI to a much greater degree than for a Composite Application in an Enterprise SOA. Choreography is much more dependent on UDDI repositories for the reasons just described.  However, the process of discovery during execution is likely to significantly reduce the performance and throughput of the Composite Application; so this is a tradeoff between process effectiveness and cost efficiency.
• Internet instead of the ESB.  The fundamental function of the ESB is to enable and support communications among the Service Components of a Composite Application, that is, it supports application layer communications within an Enterprise (or organization).  However, with Ecosystem SOA, interfacing with Service Components across the Internet is the norm.  While there is much discussion about cross-domain ESBs, the issue is difficult given the security requirement involved in application layer communications across the boundaries.
• A Public Cloud enables and supports Ecosystem SOA.  The Public Cloud is by definition, a multi-ownership domain entity.  Therefore, the Platform as a Service (PaaS) layer and the Infrastructure as a Service (IaaS) layer perfectly support Ecosystem SOA.

The Generalize Agile Development and Implementation Process for Software and Hardware

Background

In 2000, I created a Rapid Application Development (RAD) process, based on eXtreme Programming (XP) that was CMMI Level 3 conformat. Conformance means an outside audit was performed and the process conforms to the requirements of the CMMI level 3 key practices (note that CMMI levels 4 and 5 are organizational practices rather than process practices, so the Level 3 is as high as a process can get).

Advantages of the RAD Process

This RAD process, as I created it, has several advantages over traditional software development efforts, like "waterfall process".  The following advantages derive from XP.

1. Works in monthly to 6 week cycles, enabling the customer to start using an Initial Operating Capability (IOC) version of the system, while the system in under development.  This meant that the customer was gaining some value even while the application was being implemented.  In the case of one small but unusual asset management, accounting, and accounts payable system, the customer, in the process of loading data into the IOC version found enough errors in the accounts to recover more than twice the cost of the effort to implement the application, that is, $100K+ errors during the implementation of a $50K system.
2. Enables the customer to get their highest priority functions.  The RAD process does this by allowing the customer to add requirements during each development cycle and reprioritize the requirements between cycles.  This produces a much more satisfied customers, than "Big Bang" processes like "the waterfall" process.
3. The development process is agile because it allows the customer to reprioritize their requirements.  In a book by Dr. Ralph Young on requirements, he cites a statistic that only 49% of the real customer requirements are known at the start of the development effort.  This RAD process allows the 51% to be incorporated, while keeping in budget using this repriortization function.  Since the product always meets the customer's highest priority requirements, they are much more satisfied with the result.
4. Focuses the developer on developing, gathering than documenting--this is what the software developer is good at, while documenting isn't.  Consequently, the developer's morale improved (which meant their effectiveness and cost efficiency improved) so both the quantity and quality of the products improved.
5. Minimizes the number of intermediate artifacts, that is, it eliminates the need to create documents like the Program Management Review (PMR) reports, status reports, design documents, Engineering Change Orders and so on.  This greatly reduces the need for developers and the program manager to create much of the documentation of other formal development processes.  In fact, as I created it, this RAD process eliminated the need for PMRs, because the customer was involved with the development team on a weekly basis.  And since the process assumed "that the program/project doesn't have all of the requirements at the start and enables the customer to add and reprioritize the requirements throughout the process, the resource requirements are kept level.  Therefore, from a programmatic perspective, this is a Level of Effort.  This means the same amount of money is spent each period; which reduces management of cost to nearly zero.  And since the implementation schedule is based on an unknown set of requirements, there is no way for a program manager to really create a schedule at the start of the effort and keep to it.  Instead, there is a monthly agreed set of requirements that must be fulfilled in development and rolled out--which by the way works, but which makes transaction managers and finance engineers paranoid--because they have so little CONTROL.  However, this process produced products of both a higher quality, and much greater quantity, and much more satisfied customers than the previous CMMI Level 3 processes.
6. Like XP, I used Use Cases to gather requirements.  A Use Case is One way for one type of actor (either a person in a particular role, or another application) to use the application under development.  In my experience and in the experience of many of the 50+ systems engineers that I coached or mentored, or otherwise trained that Use Cases were the best way to gather requirements because they could be documented in ways the meant something to the customer (they can be documented to use the customer's own words); most other ways, like "shall statements" communicated much less because they are a transliteration of the customer's requirements.  Additionally, in general, they communicate as much, if not more to the developer.  Since the goal of requirements identification and management is to "communicate the customer's requirements to the developer and to ensure that these requirements are met", using the Use Cases, as recommended by XP proved both highly effective and cost efficient.

The net of all this is more satisfied customers, happier developers, and much more effective and cost efficient processes and products--in this case a real live "Win-Win" situation.  One consequence was that by 2005, literally, hundreds of small and medium (and even a few large) effort adopted this RAD process--and it proved highly successful within its domain (software development).

Limitations of Rapid Application Development (RAD) Processes

The key limitation of the RAD category of formal processes (processes that meet CMMI Level 3 practices) is that they are strictly for software development.  Formally, it is not possible to use this process for software or hardware integration, and for linkage to systems external to the organization, those activities that are most likely to occur in today's SOA and cloud-based environments.  Since most organizations purchase software and integrate it into systems rather than develop their own, a short cycle, agile implementation and transformation process is needed for such efforts.  This process should be the equivalent  for integration of the RAD process  that I designed and documented for software development.  In fact, it should have the same advantages as the RAD process, but applied to integration and tailoring of systems.

The Generalized Agile Development and  Implementation Process

Starting in about 2005, I gave thought to this Generalized Agile Implementation and Development (GAID) process would work.   This is a notional design and the challenges in designing such a process.  The notional design consists of four loops.

1. The Requirements Identification and Validation Loop (Approximately 3 Months).  The purpose of this loop is to work with the customer to identify their requirements (the Systems Engineering Role), to decompose and derive the functions, to structure the functions, to allocate functions to components, to determine the acquisition method for components through trade off studies (make/buy/rent) (all in the System Architect role), and to validation that the product meets the customer's system requirements for this loop and all prior loops.  Additionally, each three month cycle allows for the acquisition and installation of any needed software, hardware, and networking components.
2. The System Assembly/Implementation and Verification Loop (Approximately 1 Month).  The purpose of this loop is to assemble the various sub-assemblies (either those developed or those acquired from a supplier) and to verify that they meet their allocated functional and design constraint requirements.  Additionally, this loop enables the customer to see the overall progress of the product and suggest minor changes that will enhance the utility of the product for the customer. For example, this might include the placement of various data windows on a particular display.
3. The Sub-system Design and/Assembly and Verification Loop (Approximately 1 Week).  The purpose of this loop is to ensure that components perform the functions specified in the component requirements to the level of dependability required and that the interfaces among components match for integration.  Additionally, this loop give the customer a first loop at the components, as they are being implemented, which means they can communicate changes they to like to see to the designers before the product start full up assembly and testing.  This greatly reduces the cost of the changes.
4. The Component Construction and Verification Loop (Daily).  This is the loop where all of the detailed design, development, and COTS product configuration takes place.  While this loop might strike the reader as silly, all loops including this consist of engineering functions with a single program management meeting at the start/completion of a loop.  In this loop, it is a 15 minute long "stand up" meeting.  The reason for the meeting is to identify risks and issues, and to have the implementers set aside time to reduce the risk or close the issue before the next stand up meeting.

If this works like the RAD process I developed worked (and it should), after the initial period of learning the process, both the customer and the implementers will find it much more straightforward, simpler, and easy to use, while producing much more effective and cost efficient products.