This book has only one major purpose—to trigger the beginning of a new field of study: computer programming as a human activity, or in short, the psychology of computer programming. There are, by various estimates, hundreds of thousands of programmers working today. Each of them could be functioning more efficiently, with greater satisfaction, if he and his manager would only learn to look upon the programmer as a human being, rather than another one of the machines. At the moment, programming—sophisticated as it may be from an engineering or mathematic point of view—is so crude psychologically that even the tiniest insights should help immeasurably. – Gerald Weinberg, The Psychology of Computer Programming, Preface
This blog series presents revolutionary advancements in systems engineering (software development, project management, software improvements). The objective is to solve the software problem and end the software crisis. The strategy is to start at the beginning and create a new software engineering approach that incorporates the behavioral aspects of software engineering. In other words, to adopt a new software approach that focuses more on people, design, and decision-making rather than process, technology, and best practices. A software approach represents a particular way of developing software systems. It integrates the belief systems, worldviews, perception, practices and behaviors that software engineers use to realize software systems. It is like the software engineering “collective consciousness” that conveys how software practitioners think, judge, decide, act, behave and practice.
An emphasis on people is needed because this country and its software engineers cannot compete using the current software engineering approach that is based upon early 20th century manufacturing model that emphasizes efficiency. There is a need for a 21st-century model that focuses on business alignment as the means of increasing customer values, satisfaction, business purpose, innovation, high quality, and high performance. Business alignment is the means by which software engineering will evolve to this higher level. This is where software systems will move from supporting the business to improving the business.
The current software approach produces a generic solution to generic problems instead of a particular solution to a particular business problem. The former emphasizes a process driven approach while the latter emphasizes a design driven approach. A process driven approach favors hordes of different specialists, process maturity, detailed methodologies and common best practices while the design driven approach favors people both domain generalists and technical specialists, decision making, and design as the key differentiator for successful projects. The design driven approach is not new. It is called concurrent engineering and is highly successful. In other words, like we design the software system; we must also design the project system that will realize the software system based upon the project’s situational context and project objectives.
Since Gerald Weinberg wrote “The Psychology of Computer Programming” in 1971, nothing has changed; the behavioral aspects of software programming have been ignored and software engineering has remained trapped in the dogma of mechanistic paradigm. Simply stated, the mechanistic paradigm is based upon Newtonian concepts that the universe and nature are closed systems machines. Responses to organizational life must be impersonal and machine like. This is why software engineering is stuck in the Dark Ages as I characterize this period of non software engineering enlightenment. Although there have been major advances in hardware technology, software remain entombed in mechanistic dogma. The mechanistic paradigm is the reason why there has been no major technological breakthrough in software engineering. Systems evolve, machines do not. Even though billions have been spent on software technology and process improvements, the ability to achieve the most marginal objectives like meeting requirements, on time and within budget remain alarmingly elusive.
Obviously technology process-centrism is not the answer because it has not worked. It produces software system that provide marginal business value and create the productivity paradox where billions are spent on software without the commiserate increases in productivity. Since inception, software organizations have searched for the technology or process “silver bullet” as the solution. In fact, one can conclude that the history of software improvement represents a long line of “silver bullets” that include Structured Programming, Information-Engineering, Object-Oriented, CASE, ISO, CMMI, Agile, and many others. Now the new savior is Cloud.
Obviously technology process-centrism is not the answer because it has not worked. It produces software system that provide marginal business value and create the productivity paradox where billions are spent on software without the commiserate increases in productivity. Since inception, software organizations have searched for the technology or process “silver bullet” as the solution. In fact, one can conclude that the history of software improvement represents a long line of “silver bullets” that include Structured Programming, Information-Engineering, Object-Oriented, CASE, ISO, CMMI, Agile, and many others. Now the new savior is Cloud.
Dr Randall Jensen and Les Dupaix( “Commandments for a Productive Development Environment,” Jan. 2006 Issue Crosstalk) stated that after all forty years of technology and software advancement, the average productivity for DOD software increased from 60 line of code per person month in 1960 to 100 lines of code per person month in 2000. The authors proposed better management as a possible answer and cited Gerald Weinberg who argued that management represented the major source of productivity improvement and Barry Boehme who proposed that poor management more than any other factor was responsible for increases in software costs. One can easily conclude that low developer productivity is a function of the behavioral factors. According to Watts Humphrey, the hierarchical management style associated with traditional software development practices does not work because people resent being treated like grunge. Applying a factory management style to highly intelligent professionals that are specifically educated for their job and who love the challenges software development presents is problematic. Software engineers should be considered as part of the Creative Class. According to Richard Florida, “To such people, work offers spiritual as well as economic gratification. They may come to the office dressed in jeans and sneakers, but they happily work 12-hour days, view their co-workers as close friends, and look to their jobs for a sense of personal fulfillment, growth, and even identity”. However, the centralized, hierarchy approach to project management treat people like grunge and cause the morale and motivation problems that decrease productivity.
Technology and process centrism also seems contradictory to quality, productivity, customer value, satisfaction and commitments. In 2003, a US Government Accounting Office (GAO) study indicated that the DOD software budget was 21 billion dollars of which eight billion (40%) was wasted on software rework because of schedule and quality problems. Another report, “The Challenges of Complex IT Projects” from the UK’s prestigious Royal Academy of Software Engineering and the British Computer Society estimated that the European Union wasted 140 Billion and the US 150 Billion Dollars per year in software rework costs. According to Barry Boehm, a 20% improvement in software productivity will be worth $45 billion in the U.S and $90 billion worldwide.
People
“Now one of the very first requirements for a man who is fit to handle pig iron as a regular occupation is that he shall be so stupid and so phlegmatic that he more nearly resembles in his mental make-up the ox than any other type. The man who is mentally alert and intelligent is for this very reason entirely unsuited to what would, for him, be the grinding monotony of work of this character. Therefore the workman who is best suited to handling pig iron is unable to understand the real science of doing this class of work.” Frederick Taylor Principles of Scientific Management
In the same book as the above excerpt, Frederick Winslow Taylor wrote that people are like machines solely motivated by money. Although Taylor wrote this most highly influential book in 1911, it pervades current hardware and software thought. In 1911, the principles were applied to factory laborers worldwide, now such principles are applied to technology workers in China, Mexico, and other countries. Technology companies have become very rich, successful, and powerful using the same inhuman principles that in the early 20th century gave rise to the union movement. Unions were formed because management ignored human issues and needs. American software engineers have not been immune from Taylor’s principles. Although he explicitly stated that his principles of Scientific Management should not be used on intelligent people as mental workers. It has. These same principles that gave the world “Sweat Shops”, created the union movement, and exploits people world-wide gives to software engineering the “Death Marches”. This is why the SDLC, the incarnation of Taylor principles to software engineering, is a failure.
Any software development approach that ignores human behaviors in software projects is doomed to failure. There is no way a machine, assembly line; approach can develop the complex, adaptive software systems including ultra-large systems, systems of systems, complex adaptive application systems, computational social science systems, and cognitive computing systems. Cognitive computing is the development of computer techniques to emulate human perception, intelligence and problem solving. It is intuitive that the software engineering approach should share a similar worldview, philosophy, and belief system as the systems being delivered.
Because of people are deemphasized and with the emphasis on technology and process, the software industry continue to completely disdain the behavioral sciences although the high level of human interactions involved in the development of software makes the quality of the human interactions one of its most fundamental elements. Defining the business objectives, inventing the software requirements, and realizing the software systems require the highest level of collaboration, communication, coordination, and cooperation. Large software projects require human skills, knowledge, and social attributes of the highest caliber. Albeit temporary, software projects are the most complex form of organizations. People from diverse jobs, roles, groups, and organizations come together to solve a business problem. These organizations have different organizational, group, role, and personal group, agendas, objectives, and requirements. This makes people the primary system inputs, not some pig iron, wood, or fabric. If the people system is dysfunction then the software development system and the project management system are also dysfunctional. In other words, the project with the best technology and the most optimized processes will fail when the psychological, sociological, cultural dynamics of software projects are ignored.
A behavioral approach is needed in software engineering because software projects consist of humans. The technology processes, and artifacts used by projects vary; however, the one constant is the existence and importance of the humans in software projects. People design and develop software. It is people who identify the business problems, needs, project objectives, and requirements. People also analyze these requirements and develop designs, code the software, test the results, and implement the system. All these tasks are essentially mental tasks that require high degree of human interaction and not mechanical tasks that are accomplished in a self-contained manner.
The management variable including leadership management philosophies, strategies, management decisions, and policies and procedures are important. Organizational elements like effectiveness in managing, recruiting, training, and keeping skilled, talented, and motivated staff. It is also the tasks performed by the project roles, the human attributes required by these roles, the relationships between the various roles, the structure of work, and the rewards that people receive. It is the personal relationships with the user managers, customers and users as well as the effectiveness and efficiency of the project tasks that developers perform. It is how the project is structured, who makes the key decisions, and the availability of information needed to support the key decisions. Finally it is the project culture that includes the attitudes, beliefs, behaviors, norms and values that create human-social energy called productivity applied to the project
It takes no stretch of the imagination to realize that the better the ecosystem coordinate, communicate, collaborate, and cooperate, the better the systems results would be. An overemphasis on technology over human needs have created major inefficiencies and ineffectiveness. Machine oriented projects are rife with negativity. Work is stressful because of the need to obtain information or outputs from others in order to complete one task and meet schedules. Excessive politics and infighting between staff, managers and specialists occur very frequently. Overspecialization causes turf wars as groups attempt to protect its turf and maintain power, prestige, and position. Projects are rife with negative behaviors include hostility, resistance, finger pointing, misleading information, politics, distrust, lack of involvement and even subversion.
A New Software Approach
The first really big effort I saw to get the problems under control was in the late ’80s with the CMM®. We thought if we got the processes right, then the productivity would be better, and we would make fewer errors. We tried to fix the problems with structured programming, and then structured design, and then structured analysis. But we’ve heard the same mantra over and over: Each of these things will cause errors to disappear and productivity to improve by an order of magnitude. What we have now is an organized process doing things in an orderly way—and producing the same stuff we did before. I always refer back to a phrase that said, “When processes are optimized, people are interchangeable.” If we get the process right, it doesn’t really matter who the people are. But it’s the people who are the problem. It’s also the people who are the answer. Dr Randall Jensen (2011 Issue Crosstalk)
A software approach consists of three major components, a perception, paradigm, and practice. The perception describes the conscious or unconscious understanding, generalization, and assumptions about software work. The paradigm describes the theoretical elements and the applied behavioral elements involved in software engineering work. The theoretical elements include the underlying theories, concepts, philosophies, sources metaphors, and mental models that direct the human-social behaviors and technical practices. The practices are technical aspects of the work that include the technology, processes, and artifacts (methods, techniques, models, and tools) that transform requirements into software systems.
The new approach is called the Organic Software Engineering Approach (OSEA). The purpose is to replace the SDLC and solve the core software problem. It is time to ditch the industrial revolution mentally and recognize that people, designs, and decisions are the solution to the core software problem. Clinging to SDLC and spending monies on maintaining bureaucratic capability maturity models that limit flexibility, agility, and adaptability is not the answer. Processes are not good at making decisions. People are not machines! Machines cannot evolve. The mentality of software engineers is far beyond that of an ox. You cannot do time and motion studies on the human brain. Lines of code cannot be measured like the old keypunch machines and typewriters. The cookie cutter, checklist, and control charts approach to software engineering does not work. Because of the emphasis on process, the most important aspects of software projects are generally ignored.
Modern project managers must address three major problems. One, major project decisions are often made unilaterally and in an ad-hoc, off-the-cuff manner that is bereft of analysis. Two, software project management is so decision-rich, that no one individual can consistently make all the right decisions. Three, often project decisions are “killer decisions,” that when wrong could cause a project to fail. In the course of managing software projects, project managers make numerous decisions including project design, functional element delivery, and strategic, tactical, and operational. Modern software projects need structured decision-making approaches that include objective setting, context analysis, situational analysis, and alternative definition and alternative selection.
Grave Risks
The battle of process over people is about software superiority. The enterprise, whether government or business, that gains software superiority will be at a major competitive advantage. Software superiority is the advantage gained from the deliverance, on time and within budget of innovative, creative, and high-quality systems that are aligned with customer needs. Systems that achieve purpose, satisfy the customers, and attain system performance objectives at significantly reduced costs. Software superiority leads to information superiority is the operational advantage derived from the ability to collect, process, and disseminate an uninterrupted flow of information while exploiting an adversary’s inability to do the same. Following upon the heels of software and information superiority is enterprise superiority. That is the shift in an industry’s balance of power that occurs when business uses information superiority to gain the flexibility needed to quickly adapt to changes in economic factors. These factors include globalization, competition, market conditions, and business mergers and acquisitions. In the DOD, it includes new weapon systems, increased military systems performance, technology advancement, and the changes in the military’s asymmetric, war fighting environment.
Fortunately, friends and potential enemies have the same software problems as we do. However, other countries, some maybe not so friendly, are also striving for software superiority, either for military or economic advantage. The Washington Time reported that “China is stepping up its long-running struggle with the United States over access to technology. China this year embarked on a campaign to target advanced industries such as aerospace, medicine and information technology for its next stage of development”. It seems that the Chinese has been encouraging creating and supporting innovative startups for years. However, the China may not cling to the mechanistic model that keeps us grounded in the Dark Ages. While offshore and cheap labor costs almost decimated the American software industry, China could sound the death knell. While the other countries are exploring an organic software universe, we will remain earthbound, trapped in the mines of reductionism digging for that ultimate technical solution, the software “Silver Bullet.”
