Blog Series for a New Software Engineering Approach

The purpose of this blog series is to introduce a new software engineering approach. The blogs in this series discuss the differences between the current traditional SDLC approach and new software engineering approach. It starts with the fundamental differences in desired results. The traditional SDLC approach seeks to meet customer requirements on time and within budget. The new Organic Software Engineering Approach seeks the customer and business alignments that will propel the business through the realization of innovative, high quality systems that achieve business purpose and satisfy the customers. 
 

In order to achieve business alignment, the project system must be designed for high levels of interactions with customers and users. Instead of the single objective of meeting scheduling and budget commitments, projects can opt for quality objectives like business purpose, system performance, satisfaction, and of course project commitments. For example, business purpose means the achievement of mission, purpose, values, vision, goals and objectives.

The onus of the requirements and objectives are on both the customer and the project. The challenges are inventing the right system requirements; using an effective evolutionary development system that provides the feedback that supports organizational learning, and joint optimization of effectiveness, quality, and efficiency. From a human resource perspective, there is the need for a professional model. Extensive collaboration with customer, managers, and users require both generalists and specialists with human, social, conceptual, business and technical skills. Employee loyalty, managing careers, training, and career development are the main challenges for these socio-technical organizations. 

People versus Process



This blog deals with the people versus process issues. Business alignment requires a complete rethinking of the behavioral aspects of software engineering. A rethinking is required because business alignment requires both a focus on people and business environment interface processes. It also requires software generalists with knowledge of the application domain. Many intuitively know that improving software engineering is about people but a body of knowledge is lacking. Nothing has changed Since Gerald Weinberg wrote “The Psychology of Computer Programming” in 197. This blogs details why and how people are so important to software engineering.




A New Software Engineering Philosophy

The first step in improving software engineering is to change the software engineering philosophy including the identification of the right desired results. With the traditional software development practices, the implicit desired result is to implement requirements on time and within budget. However, business customers want the developers to focus on customer value, business purpose, innovation, creativity, quality and effectiveness. The desired result should be the maximization of customer value through the joint optimization of effectiveness, efficiency, and quality. This requires a new philosophical and strategic approach that alters our perception, belief systems, paradigm, mental model, behaviors and practices.   

Economic Revitalization thru New Software Engineering Approach

The changes in a project's desired results will increase quality, innovation, business value, and productivity and massively stimulate the economy. Although American companies cannot compete from low labor costs, we can compete on the basis of better quality, time to market, localization, flexibility, adaptability, innovation, agility, and quality. The global winners will be the businesses and governments that replace the underlying mechanistic paradigm with a system paradigm that will facilitate the support the reorganization, redesign, reinvention, and reengineering of their institutions.

The world has become asymmetrical and the new global reality often no longer fit the current causal, linear, operational, and symmetrical realities associated with linear thinking. The new realities of complex adaptive systems include not only software engineering, but our education, healthcare, and many other systems. In order to survive these organizations must evolve from reductionist, machine thinking to relational, organic, thinking. 

Decrease Software Costs by an Order of Magnitude

This blog presents how organic software engineering technology can save government and business organizations billions of dollars in software development costs. This report is about reducing software development costs by a factor of ten using the new software engineering approach. Imagine the impact of major increases in customer value derived from software systems and at 10% of the current costs. In order to achieve our cost reduction goals, the blog addresses four high cost areas: software rework, excessive functionality, software complexity, and low staff productivity. Other major contributors to high software costs such as inaccurate software estimates, excessive schedule slippage, and cost overruns are addressed in separate blogs. A hypothetical $20 million project is used to explain how reductions in rework and excessive functionality and increases in project efficiency and staff productivity will decrease costs to around $2 million

The End of Offshore Development    

This blog presents software engineering solutions that reduce the incentives for offshore development. Alignment of the systems with the business requires a new organic software engineering approach that is designed to improve business innovation, quality, business value, productivity, and satisfaction. At the same time, project must achieve efficiency and commitments goal like significantly reducing software costs and gaining the ability to virtually always meet schedules and budgets. The achievement of business effectiveness, project efficiency and software quality goals require a “face to face and eye to eye” level of coordination, collaboration, communication, and cooperation with the business customer. This is something that offshore companies cannot feasibly provide. With such efficiency, quality and effectiveness benefits, American companies will have no incentive to send American jobs offshore. Millions of jobs will stay home!  

Advanced Software Estimating with Near Zero Variance

This blog is about software project estimating using a history-based software estimating technology called project specific modeling. In project specific modeling, each project can develop its own particular estimating equation, relational models, and profiles. The difference between project specific modeling and generic costs modeling is the accuracy of the estimates. Project specific modeling use internal history data for estimating while the generic cost models use empirical data. In project specific modeling estimators use a history database that allows estimators to “listening” to their own data” and create an estimating equation for that particular project instead of using a single equation that is supposed to accurately estimate all software projects, past, present and future.

The End of Scheduling and Software Project Problems 

This blog presents an organic software project management. Project managers need to constantly assess what is out of balance, what is functioning effectively, and then make decisions regarding where and how to deploy the time, attention, resources, and energy needed to restore the balance. Trend analysis is essential. The purpose of trend analysis is to quickly identify potential problems and make the mid-course corrections before project system objectives are put at risk. There is a need for an early warning system that identifies when the project is becoming problematic so that the project team can take actions to restore the balance. The solution is to create a set of role model projects that become the basis for the quantitative models that guide the project team towards project success.

People versus Process

People versus Process 

This book has only one major purpose—to trigger the beginning of a new field of study: computer programming as a human activity, or in short, the psychology of computer programming. There are, by various estimates, hundreds of thousands of programmers working today. Each of them could be functioning more efficiently, with greater satisfaction, if he and his manager would only learn to look upon the programmer as a human being, rather than another one of the machines. At the moment, programming—sophisticated as it may be from an engineering or mathematic point of view—is so crude psychologically that even the tiniest insights should help immeasurably. – Gerald Weinberg, The Psychology of Computer Programming, Preface

This blog series presents revolutionary advancements in systems engineering (software development, project management, software improvements). The objective is to solve the software problem and end the software crisis. The strategy is to start at the beginning and create a new software engineering approach that incorporates the behavioral aspects of software engineering. In other words, to adopt a new software approach that focuses more on people, design, and decision-making rather than process, technology, and best practices. A software approach represents a particular way of developing software systems. It integrates the belief systems, worldviews, perception, practices and behaviors that software engineers use to realize software systems. It is like the software engineering “collective consciousness” that conveys how software practitioners think, judge, decide, act, behave and practice.

An emphasis on people is needed because this country and its software engineers cannot compete using the current software engineering approach that is based upon early 20^th century manufacturing model that emphasizes efficiency. There is a need for a 21st-century model that focuses on business alignment as the means of increasing customer values, satisfaction, business purpose, innovation, high quality, and high performance. Business alignment is the means by which software engineering will evolve to this higher level. This is where software systems will move from supporting the business to improving the business.   

The current software approach produces a generic solution to generic problems instead of a particular solution to a particular business problem. The former emphasizes a process driven approach while the latter emphasizes a design driven approach. A process driven approach favors hordes of different specialists, process maturity, detailed methodologies and common best practices while the design driven approach favors people both domain generalists and technical specialists, decision making, and design as the key differentiator for successful projects. The design driven approach is not new. It is called concurrent engineering and is highly successful. In other words, like we design the software system; we must also design the project system that will realize the software system based upon the project’s situational context and project objectives.

Since Gerald Weinberg wrote “The Psychology of Computer Programming” in 1971, nothing has changed; the behavioral aspects of software programming have been ignored and software engineering has remained trapped in the dogma of mechanistic paradigm. Simply stated, the mechanistic paradigm is based upon Newtonian concepts that the universe and nature are closed systems machines. Responses to organizational life must be impersonal and machine like. This is why software engineering is stuck in the Dark Ages as I characterize this period of non software engineering enlightenment. Although there have been major advances in hardware technology, software remain entombed in mechanistic dogma. The mechanistic paradigm is the reason why there has been no major technological breakthrough in software engineering. Systems evolve, machines do not. Even though billions have been spent on software technology and process improvements, the ability to achieve the most marginal objectives like meeting requirements, on time and within budget remain alarmingly elusive. 

Obviously technology process-centrism is not the answer because it has not worked. It produces software system that provide marginal business value and create the productivity paradox where billions are spent on software without the commiserate increases in productivity. Since inception, software organizations have searched for the technology or process “silver bullet” as the solution.  In fact, one can conclude that the history of software improvement represents a long line of “silver bullets” that include Structured Programming, Information-Engineering, Object-Oriented, CASE, ISO, CMMI, Agile, and many others. Now the new savior is Cloud.

Dr Randall Jensen and Les Dupaix( “Commandments for a Productive Development Environment,” Jan. 2006 Issue Crosstalk) stated that after all forty years of technology and software advancement, the average productivity for DOD software  increased from 60 line of code per person month in 1960 to 100 lines of code per person month in 2000. The authors proposed better management as a possible answer and cited Gerald Weinberg who argued that management represented the major source of productivity improvement and Barry Boehme who proposed that poor management more than any other factor was responsible for increases in software costs. One can easily conclude that low developer productivity is a function of the behavioral factors. According to Watts Humphrey, the hierarchical management style associated with traditional software development practices does not work because people resent being treated like grunge. Applying a factory management style to highly intelligent professionals that are specifically educated for their job and who love the challenges software development presents is problematic. Software engineers should be considered as part of the Creative Class. According to Richard Florida, “To such people, work offers spiritual as well as economic gratification. They may come to the office dressed in jeans and sneakers, but they happily work 12-hour days, view their co-workers as close friends, and look to their jobs for a sense of personal fulfillment, growth, and even identity”. However, the centralized, hierarchy approach to project management treat people like grunge and cause the morale and motivation problems that decrease productivity.

Technology and process centrism also seems contradictory to quality, productivity, customer value, satisfaction and commitments. In 2003, a US Government Accounting Office (GAO) study indicated that the DOD software budget was 21 billion dollars of which eight billion (40%) was wasted on software rework because of schedule and quality problems. Another report, “The Challenges of Complex IT Projects” from the UK’s prestigious Royal Academy of Software Engineering and the British Computer Society estimated that the European Union wasted 140 Billion and the US 150 Billion Dollars per year in software rework costs. According to Barry Boehm, a 20% improvement in software productivity will be worth $45 billion in the U.S and $90 billion worldwide.

People

“Now one of the very first requirements for a man who is fit to handle pig iron as a regular occupation is that he shall be so stupid and so phlegmatic that he more nearly resembles in his mental make-up the ox than any other type. The man who is mentally alert and intelligent is for this very reason entirely unsuited to what would, for him, be the grinding monotony of work of this character. Therefore the workman who is best suited to handling pig iron is unable to understand the real science of doing this class of work.” Frederick Taylor Principles of Scientific Management

In the same book as the above excerpt, Frederick Winslow Taylor wrote that people are like machines solely motivated by money. Although Taylor wrote this most highly influential book in 1911, it pervades current hardware and software thought. In 1911, the principles were applied to factory laborers worldwide, now such principles are applied to technology workers in China, Mexico, and other countries. Technology companies have become very rich, successful, and powerful using the same inhuman principles that in the early 20^th century gave rise to the union movement. Unions were formed because management ignored human issues and needs. American software engineers have not been immune from Taylor’s principles. Although he explicitly stated that his principles of Scientific Management should not be used on intelligent people as mental workers. It has. These same principles that gave the world “Sweat Shops”, created the union movement, and exploits people world-wide gives to software engineering the “Death Marches”. This is why the SDLC, the incarnation of Taylor principles to software engineering, is a failure.

Any software development approach that ignores human behaviors in software projects is doomed to failure. There is no way a machine, assembly line; approach can develop the complex, adaptive software systems including ultra-large systems, systems of systems, complex adaptive application systems, computational social science systems, and cognitive computing systems. Cognitive computing is the development of computer techniques to emulate human perception, intelligence and problem solving. It is intuitive that the software engineering approach should share a similar worldview, philosophy, and belief system as the systems being delivered. 

Because of people are deemphasized and with the emphasis on technology and process, the software industry continue to completely disdain the behavioral sciences although the high level of human interactions involved in the development of software makes the quality of the human interactions one of its most fundamental elements. Defining the business objectives, inventing the software requirements, and realizing the software systems require the highest level of collaboration, communication, coordination, and cooperation. Large software projects require human skills, knowledge, and social attributes of the highest caliber. Albeit temporary, software projects are the most complex form of organizations. People from diverse jobs, roles, groups, and organizations come together to solve a business problem. These organizations have different organizational, group, role, and personal group, agendas, objectives, and requirements. This makes people the primary system inputs, not some pig iron, wood, or fabric. If the people system is dysfunction then the software development system and the project management system are also dysfunctional. In other words, the project with the best technology and the most optimized processes will fail when the psychological, sociological, cultural dynamics of software projects are ignored.

A behavioral approach is needed in software engineering because software projects consist of humans. The technology processes, and artifacts used by projects vary; however, the one constant is the existence and importance of the humans in software projects. People design and develop software. It is people who identify the business problems, needs, project objectives, and requirements. People also analyze these requirements and develop designs, code the software, test the results, and implement the system. All these tasks are essentially mental tasks that require high degree of human interaction and not mechanical tasks that are accomplished in a self-contained manner.

The management variable including leadership management philosophies, strategies, management decisions, and policies and procedures are important. Organizational elements like effectiveness in managing, recruiting, training, and keeping skilled, talented, and motivated staff. It is also the tasks performed by the project roles, the human attributes required by these roles, the relationships between the various roles, the structure of work, and the rewards that people receive. It is the personal relationships with the user managers, customers and users as well as the effectiveness and efficiency of the project tasks that developers perform. It is how the project is structured, who makes the key decisions, and the availability of information needed to support the key decisions. Finally it is the project culture that includes the attitudes, beliefs, behaviors, norms and values that create human-social energy called productivity applied to the project

It takes no stretch of the imagination to realize that the better the ecosystem coordinate, communicate, collaborate, and cooperate, the better the systems results would be. An overemphasis on technology over human needs have created major inefficiencies and ineffectiveness. Machine oriented projects are rife with negativity. Work is stressful because of the need to obtain information or outputs from others in order to complete one task and meet schedules. Excessive politics and infighting between staff, managers and specialists occur very frequently. Overspecialization causes turf wars as groups attempt to protect its turf and maintain power, prestige, and position. Projects are rife with negative behaviors include hostility, resistance, finger pointing, misleading information, politics, distrust, lack of involvement and even subversion.

A New Software Approach

The first really big effort I saw to get the problems under control was in the late ’80s with the CMM®. We thought if we got the processes right, then the productivity would be better, and we would make fewer errors. We tried to fix the problems with structured programming, and then structured design, and then structured analysis. But we’ve heard the same mantra over and over: Each of these things will cause errors to disappear and productivity to improve by an order of magnitude. What we have now is an organized process doing things in an orderly way—and producing the same stuff we did before. I always refer back to a phrase that said, “When processes are optimized, people are interchangeable.” If we get the process right, it doesn’t really matter who the people are. But it’s the people who are the problem. It’s also the people who are the answer. Dr Randall Jensen (2011 Issue Crosstalk)

A software approach consists of three major components, a perception, paradigm, and practice. The perception describes the conscious or unconscious understanding, generalization, and assumptions about software work. The paradigm describes the theoretical elements and the applied behavioral elements involved in software engineering work. The theoretical elements include the underlying theories, concepts, philosophies, sources metaphors, and mental models that direct the human-social behaviors and technical practices. The practices are technical aspects of the work that include the technology, processes, and artifacts (methods, techniques, models, and tools) that transform requirements into software systems.

The new approach is called the Organic Software Engineering Approach (OSEA). The purpose is to replace the SDLC and solve the core software problem. It is time to ditch the industrial revolution mentally and recognize that people, designs, and decisions are the solution to the core software problem. Clinging to SDLC and spending monies on maintaining bureaucratic capability maturity models that limit flexibility, agility, and adaptability is not the answer. Processes are not good at making decisions. People are not machines! Machines cannot evolve. The mentality of software engineers is far beyond that of an ox. You cannot do time and motion studies on the human brain. Lines of code cannot be measured like the old keypunch machines and typewriters. The cookie cutter, checklist, and control charts approach to software engineering does not work. Because of the emphasis on process, the most important aspects of software projects are generally ignored.

Modern project managers must address three major problems. One, major project decisions are often made unilaterally and in an ad-hoc, off-the-cuff manner that is bereft of analysis. Two, software project management is so decision-rich, that no one individual can consistently make all the right decisions. Three, often project decisions are “killer decisions,” that when wrong could cause a project to fail. In the course of managing software projects, project managers make numerous decisions including project design, functional element delivery, and strategic, tactical, and operational. Modern software projects need structured decision-making approaches that include objective setting, context analysis, situational analysis, and alternative definition and alternative selection.

Grave Risks

The battle of process over people is about software superiority. The enterprise, whether government or business, that gains software superiority will be at a major competitive advantage. Software superiority is the advantage gained from the deliverance, on time and within budget of innovative, creative, and high-quality systems that are aligned with customer needs. Systems that achieve purpose, satisfy the customers, and attain system performance objectives at significantly reduced costs. Software superiority leads to information superiority is the operational advantage derived from the ability to collect, process, and disseminate an uninterrupted flow of information while exploiting an adversary’s inability to do the same. Following upon the heels of software and information superiority is enterprise superiority. That is the shift in an industry’s balance of power that occurs when business uses information superiority to gain the flexibility needed to quickly adapt to changes in economic factors. These factors include globalization, competition, market conditions, and business mergers and acquisitions. In the DOD, it includes new weapon systems, increased military systems performance, technology advancement, and the changes in the military’s asymmetric, war fighting environment.

Fortunately, friends and potential enemies have the same software problems as we do. However, other countries, some maybe not so friendly, are also striving for software superiority, either for military or economic advantage. The Washington Time reported that “China is stepping up its long-running struggle with the United States over access to technology. China this year embarked on a campaign to target advanced industries such as aerospace, medicine and information technology for its next stage of development”. It seems that the Chinese has been encouraging creating and supporting innovative startups for years.  However, the China may not cling to the mechanistic model that keeps us grounded in the Dark Ages. While offshore and cheap labor costs almost decimated the American software industry, China could sound the death knell. While the other countries are exploring an organic software universe, we will remain earthbound, trapped in the mines of reductionism digging for that ultimate technical solution, the software “Silver Bullet.”

A New Software Engineering Philosophy

Business-IT alignment is a dynamic state in which a business organization is able to use information technology (IT) effectively to achieve business objectives - typically improved financial performance or marketplace competitiveness. Some definitions focus more on outcomes (the ability of IT to produce business value) than means (the harmony between IT and business decision-makers within the organizations).It is important to consider the overall value chain in technology development projects as the challenge for the value creation is increasing with the growing competitiveness between organizations that has become evident (Bird, 2010). The concept of value creation through technology is heavily dependent upon the alignment of technology and business strategies. While the value creation for an organization is a network of relationships between internal and external environments, technology plays an important role in improving the overall value chain of an organization. However, this increase requires business and technology management to work as a creative, synergistic, and collaborative team instead of a purely mechanistic span of control. Technology can help the organization recognize improved competitive advantage within the industry it resides and generate superior performance at a greater value, [1] according to Bird ([3]). Excerpts from Wikipedia on Business-IT Alignment

We all understand the impact software technology and how it has changed the world. However, most do not understand is the enormous potential of software when developed under the right philosophy. For example, what are the potential gains from a software development philosophy that aligns software systems with business needs? This is called Business-IT alignment. The reasons for the inability to align the business needs with system needs are hierarchic. At the conceptual it is the wrong software development philosophy and approach. At the project level, the inability to align the system with the business occurs because of conflicting objectives. Software engineers are trained and rewarded to deliver requirements specifications on time and within budget. Although “technically correct”, those delivered systems often provide very limited business and customer value. However, business customers want high quality software systems that meet business purpose, add customer value, satisfy users, and increases quality, productivity, and innovation. Because of the conflictive objectives and opposing views, software projects often are characterized by negative developer/customer interactions including finger pointing, lack of support and user involvement, conflicts, excessive politics, personnel resistance, and lack of cooperation and communication.

Of course, the objectives of the business units must prevail. No project has even been initiated in order to determine how well the development organization could meet requirements on time and within budget. But the development community does not see it this way and the question is why? Why the extraordinary obsession with schedules? What makes software developers go as far as to implement immature systems that often do not work, require extensive changes to make work, or produce marginal results when projects fall to far behind schedule?

The answer is the underlying software approach. A software approach represents a particular way of developing software systems. It is the equivalent of a software development conceptual framework, ideology or philosophy. It is a philosophy in that it represents a particular system of software engineering thought that includes the theories and concepts, or set of theories and concepts, beliefs, principles, and mental models that underlie the development of large-scale software systems. It integrates the belief systems, worldviews, perception, practices and behaviors that software engineers use to realize software systems. It is like the software engineering “collective consciousness” that conveys how software practitioners collectively think, judge, decide, act, behave and practice.

A software approach consists of three major components, a perception, paradigm, and practice. The perception describes the conscious or unconscious understandings of software work. For example, the current traditional understanding is that software development is a linear system while the new software perception is that software development is a complex adaptive system. The paradigm describes the theoretical elements and the behavioral elements involved in software engineering work. The theoretical elements include the underlying theories, concepts, philosophies and mental models that direct the human-social behaviors and technical practices. The behavioral elements include the project’s cultural, managerial, and organizational factors. The practices are the technical aspects of the work that include the technology, processes, and artifacts (methods, techniques, models, and tools) that transform requirements into software systems).

Gaining the ability to align software with the business require the replacement of the mechanistic with the system paradigm. The mechanistic paradigm dictates how software developers think, judge, decide, act, behave and practice the traditional software engineering approach. The mechanistic paradigm disavows all that “gooey soft stuff”. It dictates software projects operate as closed systems, behave like machines that ignore human behaviors, focus singularly on technology and process, and above, all emphasize efficiency through meeting schedules and budgets. As closed systems, projects have minimal relationships with the environment. Purpose is loosely coupled, functionally oriented, and generic in nature. The project purpose is to develop the requirements specified in the requirements document.

As a result, software projects exhibit more machine-like behaviors rather than human behaviors and focus more on internal measurements than external. Developers are trained and rewarded to meet schedule and budgets while external environmental measures like business purpose achievement are ignored. The mechanistic paradigm underlies the Software Development Life Cycle (SDLC). It is defined by the phases, activities, and tasks need to transform requirements into software systems. The SDLC uses a monolithic approach much like a pseudo assembly line to transform requirements into specifications, designs, code, tests and implementation. The end result is a manufactured software system tool that is designed to be used until broken. Like other manufactured tools, a bucket for example, quality is measured by the defects that prevent it from functioning correctly. The SDLC also includes the Agile, Lean, CMMI and Six Sigma because of their underlying scientific management principles.

The systems paradigm dictates how software developers think, judge, decide, act, behave and practice the new organic approach called the Organic Software Engineering Approach (OSEA). The OSEA suggests software projects are part of an ecosystem that grows software system services. The concept of the ecosystem emphasizes the need for harmony between the IT developers and business managers, customers, and users within the project. OSEA projects deliver organic software systems that are like living systems such as animal, plant, fungus, or micro-organism. For example, organisms are responsive to stimuli, reproduction, change, maintenance and growth. Software respond to stimuli, often reproduce on other platforms, incur system changes, grow in size and attempt to maintain system integrity during the process. Tools do not share common characteristic with software.

The systems paradigm dictates that projects operate as open systems, behave like socio-techno systems, focus on all aspects of project and system performance and emphasize the joint optimization of time, costs, quality and effectiveness. Ludwig von Bertalannfy, the father of system theory cautioned, that overemphasis on one variable in a function can sub-optimize the whole. He is correct; an overemphasis on schedules and budget is the core reason for project failures. As an open system, the OSEA is tightly coupled and specific to the needs of its environment, which are the business units. There is a strong functional bond, an alignment with the business units that philosophically and ideologically are beyond the capability of the closed system, SDLC. In the OSEA, purpose consists of distinct mission, vision, values, and objectives. It is tightly coupled, environmentally cohesive, and specific to the needs of the business. Quality is also different. Instead of defects, quality is categorized by the achievement of purpose, satisfaction, commitments, and system performance. Included in system performance are functionality, system support services, human factors, security, software quality factors, system operational performance, and defects.

Summary

It is time for a change. The CMM/I have proven that. The CMM/I, the epitome of software mechanistic thought, was widely herald as the “solution”. However, like the rest of the progression of “silver bullets”, the expectations were unrealized. Instead of the expected transformational benefits, improvements are disappointingly measured by return on investment. Clearly, it has not lived up to the original hype and is at an evolutionary dead end. When organizations reach Level 5 there is nowhere else to go yet, the major software problems remain? In addition, a 2003 GAO study estimated that the DOD wastes forty percent, 8 billion, of their annual 21 billion dollar software budget on rework due to quality issues. The DOD out-sources development contracts to organizations with higher CMM/I levels.

The purpose of this blog-series is to identify another way of developing large-scale software systems and to identify the transformational results that will occur one the change is implemented. The underlying mechanistic paradigm must be replaced. It is the strength of the mechanistic dogma that causes software engineers to continue to develop software in the same way, despite the futility of their efforts. Every five years, the size of software projects increases an order of magnitude so the software problem must be solved. Large systems are essential to the economy because business and the government use these systems to run their organizations. The inability to plan, organize, manage and direct moderate to large sized software projects is a major problem. If the software problem remains unresolved, the increasing importance of software systems and the growth in software size and complexity will put the world's economy at risk. The increases in project and system size will drive costs up exponentially and large projects will experience even higher costs, lower productivity, and greater failures.

Only strategic change can rectify this problem because the problem is systemic, philosophical, and cultural. It is systemic because virtually all moderate to large-scale software projects fail to meet expectations, often double or triple schedule and budgets, or are cancellations. It is philosophical and cultural because the adherents of the current software development practices accept that that the vast majority on their efforts including virtually all large-scale system development will result in failure but do not change. The problems associated with the development of large-scale software systems require philosophical and culture change. The philosophy change includes a new perception, conceptual framework, behaviors, and practices that will create new software engineering cultures that has the capability to develop large-scale, high quality systems. In other words, we must create new management systems, project structures, work systems, decision and information databases, human resources skills, work settings, reward systems and artifacts, processes, and technologies. There is also the need for a capability support system that supports the four project subsystems, software engineering, project management, project design, and human resources.

Advanced Software Estimating with Near Zero Variance

Near zero-variance is defined as the ability to achieve schedule and budget predictability with an absolutely minimal variance between project estimates and actual. It is achieved through an organic software estimating approach that is contextually based. Organic software estimating is a component of the Organic Software Engineering Approach that consists of new software engineering perception, paradigm, behaviors, and practices. The source of the estimating problem is the software incarnation of Frederick Taylor’s scientific management. We call it the SDLC; its goal is efficiency; and that is achieved through budget and schedule attainment. As a result, estimators rely on empirical cost models that use empirical data from “best in the class projects”. It is the software equivalent of Taylors time and motion studies. The only problem is that you cannot perform time and motion studies on the human brain.

Any proposed solution to the software problems must deal with estimating. The ability to meet schedules and budgets is a core problem that must be tackled. Achieving both system and project predictability is contingent upon the accuracy of the estimating system. Current estimating models grossly underestimate resource requirements so most projects are doomed from the start. The carnage caused by poor estimating practices is devastating. In economic terms it can be calculated in the billions and even trillions of dollars in wasted software costs and lost economic opportunity costs. Socially, it can be calculated in terms of voluntary and involuntary terminations, demotions, and careers put on hold. The same mechanistic paradigm that gives the world Sweat Shops give software developers Death Marches.

Estimating has always been problematic. The idea that some cost model based on empirical data could accurately estimate all projects past, present and future discounts the uniqueness of software development organizations and their projects. In order to improve estimating accuracy, estimators have been researching and creating various quantitative software development models to describe the software development process. However, the models are too generic and lack the granularity needed to estimate specific software projects. The default parameters can not accurately represent the wide variability of software project contexts. All of this is understandable, without organizational, technical, and project specific data the accuracy of the cost model are limited. These organizational, technical and project specific variables include but are not limited to:

1) Software organizations use different organizational factors, management policies, technologies, and management to create development cultures with different behaviors, attitudes, norms, and values. The culture determines attitudes towards organizational objectives, the amount of enthusiasm and energy applied to tasks, how work is performed or not performed, how internal groups support or hamper project teams, and ultimately how things get done or not get done.

2) There are significant productivity differences in software engineering approaches like the OSEA and SDLC, methodologies, and artifacts. This includes the waterfall, incremental, evolutionary, RAD, Agile or other life cycle models. The life cycle approach represents the difference in the project's course of development and hence the project's phase, events, activities, work products and tasks. It significantly impacts effort, schedules, and costs.

3) Project categories, source, and type have a major estimating impact. Whether a team is developing software, purchasing a package, replicating a system, or converting a system have significant impact on productivity. So does project source like internal development, project developed under contract, project produced for marketing, and many others. Project technical types embedded systems, time critical real-time, scientific or engineering, system programming, distributed and network, data processing and database, expert and artificial intelligence, image and pattern recognition, large scale simulation and modeling

4) Language groups and generation including 5GL, 4GL 3GL and assembler languages have a major impact on productivity.

5) System size and complexity have a major impact on productivity. As the size of the software product increases linearly, project resource requirements for development, communications, integration, coordination, and documentation increase exponentially.

6) The alignment of the development system used to develop the system with the develop system needed to develop the project is important. The greater the contradictions or redundancies between development system and project reality in terms of needed technology, processes, and artifacts, the lower the productivity.

7) Variations in staff caliber including skills, knowledge, experience, and motivation vary widely and affect productivity. Development team context includes development team experience, project support, and project management experience, management support, technical support, and project team quality.

8) User context includes user knowledge, experience, attitudes, support, and organizational turbulence. User context is a key productivity element. User context and the complexity of the problem will help determine the stability of the project and the subsequent volatility that causes rework.

9) Problem context includes problems size, problem complexity, precedent, problem domain maturity, and partial functionality. In the problem context, the major cost drivers are precedent. Precedent refers to domains that are new the organizational and includes technology, computer science, and application domains. The uncertainty that comes with precedence is a major reason for estimating errors. In many situations it takes twice the effort to accomplish unfamiliar tasks as it takes to accomplish familiar tasks.

10) Business context includes purpose instability, business importance, and business integration. Purpose instability leads to product instability including changes in scope and requirements. High business importance requires extensive and often unproductive business management scrutiny. High integration requires extensive coordination, communication, collaboration, and cooperation.

11) Platforms have the potential for major differences in productivity. Platforms and their languages share different characteristics that affect productivity. Mainframe, PC, server, Internet/Web, embedded real time, robotics, mobile system, network, palms, and games systems all have the potential for significant variations in productivity.

12) The quality philosophy has a significant impact on productivity. In the SDLC, the primary quality issue is defects. Does the tool work or not? Organic software estimating is customer oriented so quality includes purpose achievement, customer value, satisfaction, business productivity and performance. The OSEA performance include functionality, system support, human factors, security, software quality factors, system operational performance, and defects.

Information Architecture

A key concept of organic software estimating is that organizations share common characteristics with living systems including plants and animals. People makeup organizations so thinking of an organization as a living system instead of a machine enables managers to harness the natural, organic, tendencies of organizations. For example, like any other living system, organizations send outputs to its environment and receive energy-inputs from the environment. High-order living systems also need training, feedback from experience, and a memory in order to survive, grow, and proliferate. As organizations are living systems, it is intuitive that in order to survive, grow and flourish, there is a need for feedback-control mechanisms and a corporate memory in the form of databases.

The information/decision architecture contains the project management assets including the data bases, tools, methodologies, processes and methods for estimating, planning, scheduling, monitoring, and managing software projects. History databases provide the data needed to create the profiles, relationships, and models. Estimates are more accurate because the estimator does not have to make assumptions inherent in cost model development. Estimators can align both the technical and project variables so that the software organization can statistically listen to their own organizational data, identify what was said, and apply the knowledge towards better management of the software development project. Databases allow a manager to find similar sets or clusters of projects that most closely match the new project and use statistical analysis routines to identify the projects with the highest degree of similarity and relevance to the new project. The results are a set of role model projects that become the basis of the quantitative models that guide the project team towards project success.

The databases contain history data from previous project and are used to create environmental profiles, relationships, and models based upon variable projects or clusters. Variable projects are selected by the estimator at estimating time and clusters involve a semi-fixed set of statistically similar projects in the project database. An environmental profile consists of characteristic and distribution profiles that typify a normal project in a cluster. Characteristic values identify standards in terms of process, performance, product, and quality. Distribution profiles describe standards in the form of the allocation of effort, duration, staff, and costs. Relationships are equations derived from statistical analysis of previous similar, history projects in the cluster. Relationships predict the values of unknown factors based upon known factors. For example, from software size, an independent variable, we can create equations to estimate size, effort, duration, and staff counts. Models are developed by averaging the behavior of similar sets of history projects in the local database. Models describe the typical behavior of a project over time.

Software Functional Elements

Software functional elements are the basic estimating elements used in organic software estimating. Software functional elements are also used in organic software requirements engineering and organic software development. Software functional elements consist of systems, subsystems, software functions, capabilities and features. Software functional elements are hierarchic systems that consist of relational components that are also hierarchic systems. Each system, subsystem, software functions, capability and feature consists of eight objects. The objects are purpose, environment, boundary, inputs, components, outputs, restrictions and feedback control. Features as the prime software estimating element can either compliment or replace function points and other related size measures. Feature can probably be categorized by the number of object elements such as the number of inputs, process services, and outputs.

The software functional element is the basis for organic software development because of the user orientation. Customers and users don't know anything about software requirements but they do understand the capabilities and features that they need to do their work. In addition, each software functional element has emergent quality needs that often cannot be decomposed to a lower level software functional element. Finally, software functional elements are self-organized systems that are design to support a specific external function including a business function or system function.

Organic design is based upon the concepts that software functional elements are more like biological functions than machine functions although machine functions do exist in living organisms. Muscles and bones are good example. Generally, biological systems contain vertical, multi-level relationships in a system hierarchy. Think in terms of cardiovascular, endocrine, digestive, lymphatic and respiratory systems. Biological systems are self-organized systems where each system supports a body functions. For example, the respiratory system is the system of organs in the body responsible for the intake of oxygen and the expiration of carbon dioxide. In mammals, it consists of the lungs, bronchi, bronchioles, trachea, diaphragm, and nerve supply. Organic design structures projects and systems like biological organisms. Just like the respiratory system support the intake of oxygen and the expiration of carbon dioxide, software functional elements support specific business elements including processes, sub-processes, functions, activities and tasks.

Software functional elements can be estimated top-down where the requirements are determined for the systems, subsystems, software functions, capabilities, and features. Software functional elements can also be estimated bottom-up as the lower level software functional elements are used to create higher level elements. The lowest level software functional element and the core estimating element is the feature that provides the services needed by the business community. The new quality engineering approach is based upon the organic service model. The objective is the creation of adaptable, flexible, customer-specific, and self-organized systems that are designed to provide the services that meet the specific needs of the business. Services need a high level of interfaces with its customers so software projects must be designed for a high degree of user interactions. Software services tend to grow so there is a need for design plasticity and flexibility. Instead of rigid software products, developers should build self-organized systems that are designed for self-maintenance, self-renewal, and self-evolution.

Project Specific Modeling

Project specific modeling applies a Business analytics (BA) approach to estimating. BA is defined in Wikipedia “Business analytics (BA) refers to the skills, technologies, applications and practices for continuous iterative exploration and investigation of past business performance to gain insight and drive business planning. Business analytics focuses on developing new insights and understanding of business performance based on data and statistical methods. In contrast, business intelligence traditionally focuses on using a consistent set of metrics to both measure past performance and guide business planning, which is also based on data and statistical methods.”

Project specific modeling uses a human approach to estimating through the comparison of dissimilarities and similarities. Project specific modeling provide profiles, relationships and models that are based upon the needs of a particular project team in a particular organization that is staffed with particular set of people who contain particular skills, knowledge and character attributes. The project team use particular technology, artifacts, and processes to develop a particular system for particular users that execute particular business processes in a particular business environment. Similarity is based upon the concept that in the same organization, similar types of projects will evolve in similar ways. Similarity is a key concept in the development of models, profiles, and relationships. The greater the similarity, the more similar will be the experiences and the more comparable the evolution of the project. The more similar the experiences and comparable the evolution of the project, the more consistent are productivity and other relationships. The more consistent are the productivity and other relationships, the more accurate the derived quantitative models. The more accurate the quantitative models, the greater the predictability of future project outcomes. The greater the predictability of the future project outcomes, the greater the likelihood of project success.

Similarity is a common concept. Appraisers use similarity in assessing the market value of homes. Consider the data of recent home sales as local data, your neighborhood as the development organization, the size of the house as the size of the software product, and the features as special software cost drivers. The appraiser looks at recent sells of homes of similar square feet and the number of bedrooms and baths in the neighborhood. Then special features and other amenities are added to the primary estimates. The estimated market value is validated by comparable homes sold in the particular zip code or neighborhood. This is in contrast to empirical estimating methods where estimate the prices of home on the basis of the country or world-wide sales.

In project specific modeling the estimator select similar reference projects by searching the database for subsystems that shares the same attributes as the new subsystem. An estimator will select multiple sets of projects with similar independent variables from the history database and use the set of data points with the highest correlation coefficient to estimate the size and productivity of new project. The selected similarity sets are passed to a statistical model that displays the regression equation, R-squared value, and correlation coefficient for each similarity group. The group of similar projects with the highest R-squared has the strongest statistical relationship. The derived relationships are used to estimates of other size factors, productivity, duration, staff count, dollars and schedules. The results are a set of role model projects that become the basis of the quantitative models that guide the project team towards project success.

Then the project team must make adjustments. For example, to estimate a high risk project, an estimator develops an attribute profile of the new project and searches through the database for similar projects or a specific cluster. In this case, the estimator looks for technical attributes, contextual attributes, and project specific attributes. There are no exact matches but the more similar the characteristics the better. For this example, this project team consisted of junior developers and was contextually characterized as low development team experience. Let’s say that Insight, the advanced project management prototype that is the basis of this article, identified these reference projects as the most statistically significant. So the estimating equation would be based upon the referent history projects where teams had very low development experience.

The context is called “Low Development Team Experience”. If the project manager can replace the juniors with more senior staff, then this particular human resource variable is controllable. Otherwise, if he is struck with the juniors, then the variable is uncontrollable and mitigation is required. Mitigation is the process by which systems adjust the relational elements in order to compensate for a risk or deficiency in another element. In this case, the project’s four subsystems (software engineering, project management, human resource, and project design) are adjusted. For example, the software engineering subsystem options call for plans that are more detailed with an increased sampling rate of inspections, code readings, and walkthroughs. The project management subsystem options calls for a longer development cycle and an increase in schedule and duration. The human resource subsystem options calls for the availability of technical leads with good technical mentoring skills to help the juniors. The project design subsystem calls for smaller teams so that the juniors can get the attention that they need from the technical leads.

Economic Revitalization thru a New Software Engineering Approach

This blog is about the use of a new software engineering approach that will facilitate the strategic change that will revitalize the economy. Strategic change is the response by government and business to the strategic factors inherent in the modern business world. The strategic factors include competition, business mergers and acquisitions, trade blocs, globalization, and market and economic conditions. It also includes advances in computers such as hand-held, portable, desktop, cluster or cloud computing. The new software engineering approaches such as organic software engineering that supports social requirements, computer-supported collaboration, and social computing are also included. Finally, new telecommunications such as optical fiber, high frequency, and associated switches and routers are included.

Turbulent, dynamic, uncertain and constantly changing economic environments require business organizations to constantly monitor their environment, reassess their objectives and goals, and then to adapt to the situational context. Strategic change includes the scanning the business environment, analyzing the specific environmental contexts, and establishing organizational purpose. It also includes the realignment the subsystems (job engineering, human resource planning, organizational design, and management) in order to achieve purpose, mission, and vision. Strategic change is needed because many companies have not adapted to global realities. In order to remain competitive, business has to examine these old ways of doing things, identify the conceptual framework that underlie these systems, and ascertain the effectiveness and efficiency of these systems in the new global reality. The world has become asymmetrical and the new global reality often no longer fit the current causal, linear, operational realities. The global winner will be the business that uses information technology to support the redesign, reinvention, and reengineer of their business units. Although American companies cannot compete from low labor costs but can compete on the basis of better quality, time to market, localization, flexibility, adaptability, innovation, agility, and quality.

Reorganization is very important because the strategic factors are forcing often disparate internal and external organizations to realign their business structures in order to work more closely together. The current stovepipe traditional formal organizations generally lack the capacity for strategic change because these structures present serious barriers to the collaboration, cooperation, communication, and coordination required by integrated systems. The closed system, vertically designed business structures where information is controlled from the top-down does not function well in horizontally, integrated systems that must cross organizational boundaries. A more open system organizational design is needed because the interactions, relationships and exchanges between business units are changing and growing even more dynamic, uncertain, and complex. Businesses must act more like biological living systems rather than machines bureaucracies because organic organizations are more adaptive, flexible, and less bureaucratic.

Software technology is the means by which organizations can become organic. As software has increased the value of physical products from cars to cameras, it can also help business unit organize and better compete. New and emerging organizational designs, technologies, and a software engineering approach can help get high quality products, services, information, and knowledge to the customers in a timely manner. It can also integrate and align their financial, managerial, job engineering, human resource planning and organizational subsystems with the situational contexts and business purpose. Examples of key design attributes of innovative organizational designs include:

• Deliberative participative planning through implementation

• Centralized planning but distributed execution and accountability

• Shared responsibility and authority among all members

• System continuous innovations – all subsystems involved

• Flexible, porous, adaptive, response, fleet of foot designs

• Self-directed, self-managed, automated work teams

• Creative, innovative, entrepreneurial, empowered teams and members

• Knowledge sharing, continuous, generative, appreciative learning

• Focused high-performance commitment thinking “outside the box”

• Continuous communication with front line people cross functional, distributed, centralized hierarchical

• Crossing, functional boundaries, “boundarylessness”, “stretch”

• Resistance of stagnation and decline through change, challenge, growth, and opportunity

For example, consider a centralized planning but distributed execution and accountability
design attribute. This is much like the Organic Software Engineering Approach (OSEA). In this form of project design, strategic planning and tactical planning are centralized and the operation planning and software development are distributed. Large projects operate as an ecosystem with three basic communities, the business, the technical, and the external partner. The business community consists of user management teams, customer teams, and user teams. The technical community consists of project management, software engineers and support, and developer teams. The partners are the vendors, prime contractors, consultants, and other organizations that are outside the company. The ecosystem also consists of cross functional teams from various communities and neighborhoods.

The user management team develops the business strategies. The customer teams define the tactical software requirements in terms of the high-level software functional elements such as the systems, subsystem, software functions and capabilities. The project management team identifies the development strategy while the software engineering team develops the tactical development plans that include the technologies, processes, and artifacts (methods, tools, techniques, procedures, models) that will be used to develop the software system. The project shifts from being centralized to distributed when the user/developer teams are assigned their respective subsystems to develop. The integrated user/developer teams will have autonomy at the point of action and control. These teams will identify, prioritize, and schedule the features for each release. At this point the feature requirements are detailed, estimated, planed, assigned, scheduled, tracked, controlled and the software created.

Compare this approach with the traditional software development approach where a project starts with the developers rushing pell mell into the requirements phase. Once the requirement specification document is completed, the SDLC phases, activities, and tasks are defined, estimated, planed, assigned, scheduled, tracked, and controlled in a monolithic manner. The sole obsession is to keep the project on schedules. While the ecosystem concept puts people and the business units in the forefront of the software development process, the SDLC puts activities and schedules in the forefront. In the SDLC developers are like machines whose purpose is use the detailed processes, methodologies, and procedures much like on an assembly line to develop the software.

Information Technology and Strategic Change

Information technology is required to support the innovative designs attributes that will facilitate strategic change. Enterprises are using advances in information technology to help business units respond to the strategic factors in an organic manner. The differences between the machine systems and organic systems represent a fundamental redefinition of the software ideology. An organic ideology calls for open system projects that maximize the relationship with its environment. Organic projects are designed to achieve what the business customers want, high quality software systems that support strategic change, competition, business purpose, customer value, quality, productivity, innovation, and user satisfaction. Organic systems rely on people and the principles and practices of concurrent engineering, systems engineering, and quality engineering. Organic design structures projects and systems like biological organisms. Generally, biological systems contain vertical, multi-level relationships in a system hierarchy. Think in terms of cardiovascular, endocrine, digestive, lymphatic and respiratory systems. Biological systems are self-organized systems where each system supports a body functions. For example, the respiratory system is the system of organs in the body responsible for the intake of oxygen and the expiration of carbon dioxide. In mammals, it consists of the lungs, bronchi, bronchioles, trachea, diaphragm, and nerve supply.

IT helps business units to realign their business structures for greater integration between the various interlocking business groups. Different business units can be organized in different ways according to their situational context and business objectives. A strong convergence of various technologies enables business organizations to engage in diverse, complex, internal and external activities. It allows the business units to keep pace with the changes in technology, demographics, and economic conditions in a turbulent competitive environment. Without information technology, obtaining the necessary structural support among disparate customer organizational units is challenging. Consider the difference in an organization that supports with centralized strategic and tactical planning with flexible, adaptive, and distributed with creative, innovative, and empowered user/developer teams with centralized, hierarchical workers performing manual, operational work.

Software Total Factor Productivity

Software Total Factor Productivity (STFP) represents the productivity gains that business customers derive from creating more outputs with the same or fewer capital and labor costs. Systems that allow the customer to reduce inputs in terms of labor hours and business costs while increasing customer outputs increase productivity. Business growth creates more jobs as the increases in efficiency and effectiveness enables the company to expand its operations. As the success of a business generally depends on its ability to deliver more real value for consumers without using more labor, capital or other inputs, the measure of success of software system is the ability to help the business users increase output value without having to use more labor, capital or other inputs.

The success of software project includes the ability to deliver customer value. Customer value is the advantages that business accrues from a software system. It is the reason why companies initiate software projects. According to Robert Sessions, the loss of business value due to project and system failure is 6 trillion dollars worldwide and 1.22 trillion dollars USA. Sessions gave as an example the new electronic fraud detection system that the IRS spent $185 million to develop. The project was abandoned in 2006. According to a 2008 report by the Treasury Inspector General for Tax Administration, the Federal Government lost approximately $894 million in fraudulent refunds during 2006 because the system was not operational. That was just for 2006 alone, there is always the following years.

Realized systems often provide marginal or insufficient business value. This happens when system objectives, purpose, designs, and requirements are misaligned with the needs of the business. When the system is delivered much of the feature set goes unused because the features are not required, the users are overwhelmed with too much functionality, or the quality of the system is so poor that the users are afraid to explore beyond absolutely what is needed. A poor defect prone system that is hard to use along with inadequate software support systems could increase user dissatisfaction and decrease customer productivity. Finally, business organizations often require more than just a software solution. Many projects require the ability to integrate complimentary business solutions including organizational design, business process improvements, and reengineering under the same project umbrella.
Software Engineering Approach

A new software engineering approach based upon systems thinking must replace the mechanistic, linear, reductionism attributes of the traditional software development approach. A machine thinking ideology like the current SDLC calls for closed system projects that focus on internal needs and minimize relationships with the business environment. Closed systems projects can’t support the achievement of business value, productivity, customer satisfaction and strategic change because that is not their underlying purpose. In closed systems, software engineers are trained, encouraged, and rewarded to deliver requirements specifications on time and within budget. The results are, at best, “technically correct” systems that provide limited business or productivity value.

The SDLC is the software incarnation of the Taylorism. Even the Agile methodologies are based upon software production technique that borrows from lean manufacturing and emphasizes closed systems, efficiency, technology and process. The SDLC supports centralized, hierarchic organizations that rely on the use of detailed standard process, detailed methodologies, and detailed procedures. It ignores the importance of human behaviors and relies solely on technology and process in order to maximize efficiency. Frederick Taylor wrote that manual workers are considered to be like machines and solely motivated by money. He specifically exempted mental workers although mental workers like teachers and software engineers are treated like manual laborers. Software engineers should not be treated like machines because people, how they are managed, their skills, experiences, and knowledge, and their cultures are essential to software success. Large projects require very high levels of coordination, communication, cooperation and collaboration, the 4Cs. People their interactions and relationships are required in order to identify the right objectives, invent the right requirements, and design, develop, and support the right system. They are needed to adapt to various situational contexts, develop the strategies to achieve objectives, develop the tactical plans needed to the concurrently engineer the project, and create and execute the operational plans needed to realize the system.

Organic System Requirements

Joe Goguen, late computer science professor at UCSD identified the requirements gap. He once stated, “It is not quite accurate to say that requirements are in the minds of clients; it would be more accurate to say that they are in the social system of the client organization. They have to be invented, not captured or elicited, and that invention has to be a cooperative venture involving the client, the users, and the developers. The difficulties are mainly social, political, and cultural and not technical.”

Organic systems include the business organization’s social requirements while SDLC, ignores social requirements. SDLC requirements are the raw materials to a deterministic, manufacturing-like software engineering approach that creates poor quality functional tools. Organic software requirements consist of the functional, non-functional, and social requirements that are inputs to an evolutionary, biological oriented approach that create purposeful systems that are aligned with the business units. Purposeful systems are adaptive systems that select the most effective strategies, tactics, and operations based upon the situational context and objectives. Purposeful systems not only include the organizations work tasks and activities, it also includes the managerial, organizational, technical support, human resources and cultural elements. In inventing the organic requirements and system designs, the requirements engineer must conceptually understand the customer organization’s underlying mental model, philosophy, concepts, theories, and principles. They must understand the management constructs, cultural construct, and the organizational construct. The organizational construct includes information about the design structures and team structures, human resources and reward system, job engineering and design, and information and decision making. The cultural construct include cultural behaviors, attitudes, beliefs, norms, and values.

Purposeful systems require the definition of the purpose object as the design principle and unifying element that glue the systems objects and business units together. The purpose objects consist of the business unit’s mission, vision, and values. System definition modeling define “things” like organizations, projects, activities and many other as systems that consists of four subsystems each with eight objects. The objects are purpose, environment, boundary, inputs, processes, outputs, restrictions and feedback control. Mission is the reason why the organization exists. A mission defines something to be accomplished in terms of desired results, objectives, and goals. Values define the preferred behaviors and what is most important to the business organization in terms of thinking, feeling, saying, and doing. Values are defined, developed, and achieved through commitment, communications, and discipline. The vision is the selected outcomes that produce the desired results. Establishing what's in the vision dictates knowledge of the desired outcomes, the current outcomes, and the gap between the two. Consider a current outcome as an organization with centralized planning and control, hierarchic bureaucracy, and process control. Consider the desired outcome as the vision of a distributed, flexible, adaptive organization with creative, innovative, and empowered teams. Define the gap between the two as the social requirements for an innovative design attribute.

Supporting the new and innovative design attributes is not simply a matter of reorganizing an organization. Software systems that support innovative project design attributes require the social requirements that include different communication channels, cross-functional processes, human resources roles, and the data flows. Social system requirements must support and encourage good relationships through improvements in the quality of the interactions between the various cultures, groups, and roles that interact. As technical and process functionality are important to the work context, the ability to communicate, coordinate, collaborate, and cooperates is just as important. As such, there are need for social requirements that support collaboration, communication, cooperation, and coordination. Feedback is also a prerequisite for purposeful systems. In this case the social requirements might align the visual representation of the user interface with the desired behaviors. Studies show that the presentation of performance feedback designed in the user interface can radically alter a decision-maker’s responses and behaviors.

Consider the difference in project management. SDLC project management stresses efficiency through the planning and control of the resources needed to ensure that a project is finished on time and within budget. Project management is essentially task management that includes the creation of complex task networks that use sophisticated planning and tracking software with Pert Charts, critical paths, and earned values. Under the OSEA, however, project management has a more social orientation. With distributed project teams managing their tasks and creating the features, project managers can devote attention to boundary management. Boundary management includes open systems planning, leading the project ecosystem, and maintaining business managers, customer, and users relationships. It also includes clarifying purpose, developing strategies, and maintaining negative and positive feedback channels. Finally it means coordinating technical management, business management and external partner management, leading the development community, and responding to a changing environmental needs and wants.

Organic projects require new software requirements and system design skills, knowledge, and experiences. In addition to the technical specialists, organic projects need client facing generalists that perform activities like social network analysis, cognitive task analysis, software ethnography, organizational assessment and design, and reengineering. While there are some software engineers that require STEM knowledge the vast majority of IT software developers do not. The new roles and skill sets will combine technical knowledge with a background in the organizational, managerial, and behavioral sciences. People will need knowledge and skills in organizational design, quality and productivity, human resources, software ethnographers, organizational psychologists, cognitive software engineers, business and domain specialists, and systems theorists. This will result in the creation of more software jobs because the decrease in development costs will increase new projects and thus create a demand for both technical specialists and generalists.

Summary

“An important question that has been debated for almost a decade is whether computers contribute to productivity growth. Productivity isn't everything. However, as noted by the economist Paul Krugman, in the long run it is almost everything. Productivity growth determines our living standards and the wealth of nations. This is because the amount a nation can consume is ultimately closely tied to what it produces. By the same token, the success of a business generally depends on its ability to deliver more real value for consumers without using more labor, capital or other inputs.” Erik Brynjolfsson of the MIT Sloan School of Management

The question rather computers contribute in ongoing only because of the SDLC. If the SDLC adds value and subsequently increases productivity that that is an unintended consequence. According to Wikipedia adding value requires coordination and collaboration; changes in organizational processes, committed leadership; flexible jobs and adaptable, capable employees, supportive organizational culture and attitudes, and investment in information technology. However, the benefits from investments in information technology have been marginal and create concerns about computer productivity. The article proves that the OSEA will address these issues.

The End of Offshore Development

The offshore of software jobs could in the long term decimate the American IT industry and reduce American software competitiveness. Software is vital because it fuels the U.S. economy and serves as a catalyst for innovation, communication, and potential economic growth. However, over the last decade, American corporations, citing cheap labor costs transferred 2.4 million jobs to India. Concurrently, since 2002, the number of students graduating with computer science and related degrees has declined 58.5%. A 2006 study on “Globalization and Off-shoring of Software” by the ACM indicated that because of off-shoring, there is no way of ensuring lifetime employment for American software engineers. Who would want to study, graduate, and enter a profession with no guarantee of lifetime employment?  The research firm Gartner Inc. predicts that up to 15 percent of tech workers will drop out of the profession by 2010, not including those who retire or die. According Business Week “The U.S. high-technology industry lost 115,800 jobs in 2010, a 2 percent decline that marked the industry’s second consecutive year of falling employment.” The situation, however, is different in India, particularly among the three largest IT companies. Tata Consultancy Services Ltd., the largest Indian IT-service provider, currently with over 202,000 employees plans to hire a total of 60,000 workers this year. Infosys Ltd, the second largest over 133,000 employees plan to hire 45,000, while Wipro the third largest with over 122,000 workers plans to increase from 25 to 35%, 30,000 to 40,000 respectively. Since there is no domestic industry for India IT to support, the vast majority of these are American based.

The result is a diminishing labor pool of software engineers and the end of American software dominance. As the IT industry contracts, American developers limited to Aerospace work and beset by the high costs of software development and poor quality problems will face survival problems similar to the American automobile and commercial electronic workers in the 1980s. Eventually software research and development as well as innovation will follow the jobs.  The DOD will be particularly hard hit because of its reliance on software, the risk of offshore development, and a dwindling pool of citizen software talent eligible for security clearances.  

Software Development Life Cycle

The Software Development Life Cycle (SDLC) that underpins current software engineering practices and methodologies is the reason for offshore development. Simply stated, it does not work for software engineering. It is the core software development problem because it applies a manufacturing, assembly line, manual work, and linear oriented approach to software development. Software requirements are so volatile that a mechanistic, monolithic process that focuses on task management and ignores human issues and behaviors cannot work. Ignored are the human social dynamics that include people, their needs, desires, and fears as well as the impact upon project behavior, motivation, and morale. Projects are more than a sophisticated planning and control systems; projects require collaborating, communications, coordination and cooperating between people. It is the behavioral aspects of software development that make large software projects as much human systems than technical systems. As the late Watts Humphries once stated, software developers resent being treated like grunge. The assumption that human behavior can be ignored and software engineers treated like manual laborers create project performance problems and reduce staff productivity.

Organic Software Engineering

The process of software development and evolution is an ambitious undertaking involving complex, incomplete, sometimes inconsistent and often fuzzy factors.  Variables concerning design, quality, reliability, stakeholder interests and objectives, moving targets, and constraints such as budget and timeline must all be considered throughout a dynamic life cycle. The challenge is to provide sound methodological support for enabling good decisions about processes and products, risks and bottlenecks as well as for selection of tools, methods and techniques.  A need also exists to certify critical software systems to ensure their dependability, relying on evaluation of the software development process as well as the properties of the system.  Software engineering research topics will identify and develop innovative means of meeting the challenge. TECHNOLOGIES OF INTEREST – DARPA 

Closed system software development approaches like the SDLC cause software projects to operate like machines. The requirements phase cranks up the project machine that activates a series of sequential phases, activities, and tasks until the system is implemented. It uses a common solution approach that includes detailed processes and methodologies. The problem is that the machine response is too inflexible to handle DARPA’s accurate description of software reality. How could a machine deal with the inconsistencies, fuzzy factors, uncontrollable variables, performance factors and changing targets?

Open systems such as biological and social systems however operate quite differently from mechanistic systems. Organic systems are designed for the inconsistencies, fuzzy factors, uncontrollable variables, performance factors and changing targets. Instead of a common solution approach, there is the concept of equifinality. All software projects are unique because each project has a set of different system objects including purpose, environment, boundary, inputs, outputs, process, restrictions, and feedback control. It suggests that similar results may be achieved from different initial conditions and in from many different ways. In software development, the implication is that software projects may achieve project system objectives from different situational contexts. Instead of cranking up the monolithic project machine, biological projects define their objectives and goals, organize the project ecosystem, evaluate their situational context, develop the strategies, identify the quality needs, select the tactics, and create the operational plans that when executed will deliver the software in stages of systems growth

The Organic Software Engineering Approach (OSEA) solves the problems associated with the almost infinite number of project variations that make each software projects unique. Software projects are like the Tower of Babel. There are the multitudes of technologies and development methods, tools, techniques, approaches and processes. Software projects are also rife with context variations, unknowns, initial starting conditions, restrictions, constraints, and controllable and uncontrollable variables. The OSEA consists of a perception, paradigm, conceptual framework, behavioral model, and a software development practice called the Organic Software Development Practice (OSDP). The OSDP replaces the mechanistic characteristic associated with the SDLC with biological characteristics associated with living systems. Software projects are not simple, causal, linear systems where the requirements are to simply define the tasks, estimate the effort and duration, and then schedule and monitor the progress. Software projects are complex adaptive systems that fit the above DARPA description of software reality. Software projects that share biological characteristics best handle the uncertainty and need for adaptation that is associated with software development. These systems require a corporate memory in terms of history data and feedback mechanisms that allow a project to learn, grow, survive and be successful. Some would call this organizational learning.

The OSDP consists of two components, the Organic Software Engineering Life Cycle (OSELC) and the Organic Software Engineering Environment (OSSEE). While the OSELC provides the universal system processes that are applicable to all software projects, the OSSEE provides the infrastructure, paradigm, models, and methodologies that allow adaptation to any type of software project situation. The OSELC replaces the SDLC. While the latter structure projects around the tasks that must be performed, the OSELC structure projects around the system processes that underlie software projects.

The OSELC system processes include systems engineering, concurrent engineering, software functional elements, organic design, quality engineering, and joint optimization. Software systems engineering defines how complex software engineering projects should be designed and managed. Concurrent engineering promotes the concurrent development of project plans and system products. It manages the entire software system engineering lifecycle though parallelization, participation of all the constituents in the ecosystems, and an emphasis on business value, customer satisfaction, productivity, and quality. Software functional elements are the new software functional model whose parts consist of systems, subsystems, software functions, capabilities, and features. Organic design partition projects, systems, and software functional elements like biological organisms that support distinct body functions. Think in terms of cardiovascular, endocrine, digestive, lymphatic and respiratory systems. Quality engineering embeds the required quality elements into the software functional elements. Joint optimization balances efficiency in terms of time, costs with effectiveness in terms of business value, customer satisfaction, and productivity.

The OSSEE provides the capabilities to assess, design, and manage the project as a socio-technical system. The Software Paradigm Model (SPM) enables a project to assess, design and manage its social system, including conceptual, managerial, organizational, and cultural constructs. Cultures in terms of attitudes, values, norms, beliefs, customs, practices, and social behavior have a significant impact on project performance. Cultures can be high performance where practitioners thrive on achievement and success or cultures can be negative like the schedule-first cultures that implement immature system when projects fall too far behind schedule. Management has a very significant impact of the project culture. So do the organizational factors that include job engineering, people, information and decision-making, project structure and the reward systems.

The system methodologies determine the manner in which software projects will plan, organized, managed, developed, and realized software systems. The system methodologies differ from the traditional methodologies in that the latter are very detailed and designed for strict adherence. System methodologies offer a set of guidelines, principles, and practices that can be tailored to a specific situational context. The OSSEE methodologies are strategic project management, project ecosystem design, tactical project management, functional element management, quality engineering, and distributed team project management. The system methodologies allow the design of a project, definition of strategic objectives and strategies, development of the tactical plans, management the software functional elements, embedment of quality, and creation of the software services. The software developers and users are assigned to distributed teams that are tasked to develop high-level software functional element like a subsystem. These integrated teams identify and prioritize the capabilities and features needed to support their business processes and then develop the operational plans needed to create the software. These practices are not ad-hoc, at project time, the systems engineer selects from the capability libraries the processes, techniques, methods, and tools that best fit the needs of the technical, project management, human resource, and organization subsystems.

 The Cause of Offshore Development

The traditional software development practices ignore the human aspects of software engineering, particularly the interactions between developers and the customers. Generally, IT staff is not the loyal “company” people that focus on the needs of the business units that they support. The problem is the closed system, ivory tower, mental model that keeps software developers aloof and disengaged from the business units.  Software engineers are trained, encouraged, and rewarded to deliver requirements specifications on time and within budget. Developers attempt to regulate requirements changes in order to meet schedules and when project fall too far behind schedule they will implement immature systems. However, business customers want high quality software systems that meet business purpose, add customer value, satisfy users, and increase quality, productivity and innovation. These differences have hardened the attitudes of many business units and executives against IT organizations. More often than not, software projects are characterized by negative interactions including finger pointing, lack of support and user involvement, conflicts, excessive politics, personnel resistance, and lack of cooperation and communication. These conflict severely affects the quality of the delivered systems.

It requires no stretch of the imagination to realize that if project developers and the business customers could improve their relationships, the results would better. The quality of the interaction and the nature of the relationships are more important to the development of quality, high performance systems than discipline processes. How well people interact with each other is essential to creating business value. People interact in order to identify the business problems, solutions, needs, project objectives, and business requirements. People also interact in order to determine and analyze the software requirements, create the designs, code the software, and test the systems. People interact in order to deploy, support, maintain, operate and train other people how to use the system.

The rancor between internal business units and internal software organizations has made it easy to offshore software engineering jobs. Business units are reluctant to pay the high American salaries for results that do not significantly improve the business. Both inshore and offshore development projects use the same problem-prone, archaic development approaches; subsequently the results from American developers and their offshore counterparts are about the same. Viewed by senior executives as costs centers that fail to produce significant business value, the decision to obtain the cheapest IT job rate, via H-1B and offshore development, is both business and social driven.   

Offshore Incompatibility

The open systems oriented OSEA provides the innovation that will significantly improve the business, technical and human-social aspects of software development. It provides business, developer, and economic needs and wants. The business units receive high quality systems that meet business purpose, add customer value, and satisfy customers through increases in system quality, productivity and innovation. Developer receives reductions in software costs by almost a factor of 10, decreases in cycle time, and attainment of schedules and budgets. The economy will see the end of the productivity paradox, the return of American-based software jobs, and a revitalized economic system. Software will become even more a significant part the value chain than enables American businesses to successful compete despite higher labor costs. As software has increased the value of physical products from cars to cameras, it can also increase the value of business services.

The OSEA is incompatible with offshore development because offshore software companies cannot support the constantly high level of interactive and relationship needs including collaboration, communication, coordination, and collaboration. High interactions are required in order to identify the right objectives, invent the right requirements, and design the right system. High interactions are needed to adapt to various situational contexts, develop the strategies to achieve objectives, develop the tactical plans needed to the concurrently engineer the project, and create and execute the operational plans needed to realize the system. High interactions are needed to receive the feedback, measure against objectives, identify the meaning of any variances, and select solutions.

Although, the American software developer cannot compete with offshore companies on a basis of labor costs, the OSEA allow them to compete on the basis of overall costs, efficiency, effectiveness, quality, time to market, and productivity. The American software engineer now has the winning home court advantage because in order to improve business value, customer satisfaction, and productivity business and developers teams must meet “eye to eye to help see things eye to eye”. Offshore developers and their H1-B counterparts will have difficult time teaming with the business managers, customers, and users because of cultural, accents, and skill nuances. Client-facing generalists with knowledge of the organizational science, behavioral sciences and system theory will replace some of the back-end specialists. Companies will have to hire American workers for customer facing jobs creating even more jobs for Americans including veterans.

There are internal risks in not changing our approach to software development, however. The Washington Time reported that “China is stepping up its long-running struggle with the United States over access to technology. China this year embarked on a campaign to target advanced industries such as aerospace, medicine and information technology for its next stage of development”. However, the Chinese may not share the affliction of many American software engineers. Some American software engineers ignore the need for change although the vast majority of software projects fail to add the business value, customer satisfaction and economic productivity. Like Keebler Elves, they happily pursue their goals of meeting schedules on time and within budget despite the constant failures. Many will never change their view because they are trapped in the dogma of the mechanistic paradigm and the hopelessness of the SDLC. While others will use the OSEA to explore the software universe, others will remain trapped in the mines of reductionism digging for that ultimate technical solution, the “Silver Bullet.”