A New Software Engineering Philosophy

Business-IT alignment is a dynamic state in which a business organization is able to use information technology (IT) effectively to achieve business objectives - typically improved financial performance or marketplace competitiveness. Some definitions focus more on outcomes (the ability of IT to produce business value) than means (the harmony between IT and business decision-makers within the organizations).It is important to consider the overall value chain in technology development projects as the challenge for the value creation is increasing with the growing competitiveness between organizations that has become evident (Bird, 2010). The concept of value creation through technology is heavily dependent upon the alignment of technology and business strategies. While the value creation for an organization is a network of relationships between internal and external environments, technology plays an important role in improving the overall value chain of an organization. However, this increase requires business and technology management to work as a creative, synergistic, and collaborative team instead of a purely mechanistic span of control. Technology can help the organization recognize improved competitive advantage within the industry it resides and generate superior performance at a greater value, [1] according to Bird ([3]). Excerpts from Wikipedia on Business-IT Alignment

We all understand the impact software technology and how it has changed the world. However, most do not understand is the enormous potential of software when developed under the right philosophy. For example, what are the potential gains from a software development philosophy that aligns software systems with business needs? This is called Business-IT alignment. The reasons for the inability to align the business needs with system needs are hierarchic. At the conceptual it is the wrong software development philosophy and approach. At the project level, the inability to align the system with the business occurs because of conflicting objectives. Software engineers are trained and rewarded to deliver requirements specifications on time and within budget. Although “technically correct”, those delivered systems often provide very limited business and customer value. However, business customers want high quality software systems that meet business purpose, add customer value, satisfy users, and increases quality, productivity, and innovation. Because of the conflictive objectives and opposing views, software projects often are characterized by negative developer/customer interactions including finger pointing, lack of support and user involvement, conflicts, excessive politics, personnel resistance, and lack of cooperation and communication.

Of course, the objectives of the business units must prevail. No project has even been initiated in order to determine how well the development organization could meet requirements on time and within budget. But the development community does not see it this way and the question is why? Why the extraordinary obsession with schedules? What makes software developers go as far as to implement immature systems that often do not work, require extensive changes to make work, or produce marginal results when projects fall to far behind schedule?

The answer is the underlying software approach. A software approach represents a particular way of developing software systems. It is the equivalent of a software development conceptual framework, ideology or philosophy. It is a philosophy in that it represents a particular system of software engineering thought that includes the theories and concepts, or set of theories and concepts, beliefs, principles, and mental models that underlie the development of large-scale software systems. It integrates the belief systems, worldviews, perception, practices and behaviors that software engineers use to realize software systems. It is like the software engineering “collective consciousness” that conveys how software practitioners collectively think, judge, decide, act, behave and practice.

A software approach consists of three major components, a perception, paradigm, and practice. The perception describes the conscious or unconscious understandings of software work. For example, the current traditional understanding is that software development is a linear system while the new software perception is that software development is a complex adaptive system. The paradigm describes the theoretical elements and the behavioral elements involved in software engineering work. The theoretical elements include the underlying theories, concepts, philosophies and mental models that direct the human-social behaviors and technical practices. The behavioral elements include the project’s cultural, managerial, and organizational factors. The practices are the technical aspects of the work that include the technology, processes, and artifacts (methods, techniques, models, and tools) that transform requirements into software systems).

Gaining the ability to align software with the business require the replacement of the mechanistic with the system paradigm. The mechanistic paradigm dictates how software developers think, judge, decide, act, behave and practice the traditional software engineering approach. The mechanistic paradigm disavows all that “gooey soft stuff”. It dictates software projects operate as closed systems, behave like machines that ignore human behaviors, focus singularly on technology and process, and above, all emphasize efficiency through meeting schedules and budgets. As closed systems, projects have minimal relationships with the environment. Purpose is loosely coupled, functionally oriented, and generic in nature. The project purpose is to develop the requirements specified in the requirements document.

As a result, software projects exhibit more machine-like behaviors rather than human behaviors and focus more on internal measurements than external. Developers are trained and rewarded to meet schedule and budgets while external environmental measures like business purpose achievement are ignored. The mechanistic paradigm underlies the Software Development Life Cycle (SDLC). It is defined by the phases, activities, and tasks need to transform requirements into software systems. The SDLC uses a monolithic approach much like a pseudo assembly line to transform requirements into specifications, designs, code, tests and implementation. The end result is a manufactured software system tool that is designed to be used until broken. Like other manufactured tools, a bucket for example, quality is measured by the defects that prevent it from functioning correctly. The SDLC also includes the Agile, Lean, CMMI and Six Sigma because of their underlying scientific management principles.

The systems paradigm dictates how software developers think, judge, decide, act, behave and practice the new organic approach called the Organic Software Engineering Approach (OSEA). The OSEA suggests software projects are part of an ecosystem that grows software system services. The concept of the ecosystem emphasizes the need for harmony between the IT developers and business managers, customers, and users within the project. OSEA projects deliver organic software systems that are like living systems such as animal, plant, fungus, or micro-organism. For example, organisms are responsive to stimuli, reproduction, change, maintenance and growth. Software respond to stimuli, often reproduce on other platforms, incur system changes, grow in size and attempt to maintain system integrity during the process. Tools do not share common characteristic with software.

The systems paradigm dictates that projects operate as open systems, behave like socio-techno systems, focus on all aspects of project and system performance and emphasize the joint optimization of time, costs, quality and effectiveness. Ludwig von Bertalannfy, the father of system theory cautioned, that overemphasis on one variable in a function can sub-optimize the whole. He is correct; an overemphasis on schedules and budget is the core reason for project failures. As an open system, the OSEA is tightly coupled and specific to the needs of its environment, which are the business units. There is a strong functional bond, an alignment with the business units that philosophically and ideologically are beyond the capability of the closed system, SDLC. In the OSEA, purpose consists of distinct mission, vision, values, and objectives. It is tightly coupled, environmentally cohesive, and specific to the needs of the business. Quality is also different. Instead of defects, quality is categorized by the achievement of purpose, satisfaction, commitments, and system performance. Included in system performance are functionality, system support services, human factors, security, software quality factors, system operational performance, and defects.

Summary

It is time for a change. The CMM/I have proven that. The CMM/I, the epitome of software mechanistic thought, was widely herald as the “solution”. However, like the rest of the progression of “silver bullets”, the expectations were unrealized. Instead of the expected transformational benefits, improvements are disappointingly measured by return on investment. Clearly, it has not lived up to the original hype and is at an evolutionary dead end. When organizations reach Level 5 there is nowhere else to go yet, the major software problems remain? In addition, a 2003 GAO study estimated that the DOD wastes forty percent, 8 billion, of their annual 21 billion dollar software budget on rework due to quality issues. The DOD out-sources development contracts to organizations with higher CMM/I levels.

The purpose of this blog-series is to identify another way of developing large-scale software systems and to identify the transformational results that will occur one the change is implemented. The underlying mechanistic paradigm must be replaced. It is the strength of the mechanistic dogma that causes software engineers to continue to develop software in the same way, despite the futility of their efforts. Every five years, the size of software projects increases an order of magnitude so the software problem must be solved. Large systems are essential to the economy because business and the government use these systems to run their organizations. The inability to plan, organize, manage and direct moderate to large sized software projects is a major problem. If the software problem remains unresolved, the increasing importance of software systems and the growth in software size and complexity will put the world's economy at risk. The increases in project and system size will drive costs up exponentially and large projects will experience even higher costs, lower productivity, and greater failures.

Only strategic change can rectify this problem because the problem is systemic, philosophical, and cultural. It is systemic because virtually all moderate to large-scale software projects fail to meet expectations, often double or triple schedule and budgets, or are cancellations. It is philosophical and cultural because the adherents of the current software development practices accept that that the vast majority on their efforts including virtually all large-scale system development will result in failure but do not change. The problems associated with the development of large-scale software systems require philosophical and culture change. The philosophy change includes a new perception, conceptual framework, behaviors, and practices that will create new software engineering cultures that has the capability to develop large-scale, high quality systems. In other words, we must create new management systems, project structures, work systems, decision and information databases, human resources skills, work settings, reward systems and artifacts, processes, and technologies. There is also the need for a capability support system that supports the four project subsystems, software engineering, project management, project design, and human resources.