by Brooke Nielsen

Obtaining zero defect software is impossible for any complex system being built. Because of this, software must be delivered and used in a condition of less then perfect. In software that is not life-critical, how must a software development team decide when the software is good enough to ship? Is it a certain date has arrived, a certain functionality is built, or some quality metric achieved?

I believe that none of the above criteria will result in the software being good enough to deliver. One must adopt a utilitarian view of quality. Utilitarianism is the belief that the rightness or wrongness of an action is a function of how it affects other people. This notion applied to software quality means that software is good enough only when the potential positive consequences of creating or employing it acceptably outweigh the potential negatives.

The difficulty is in accurately weighing the consequences. This can be accomplished through structured risk management. We evaluate the risks of creating or employing something and identify factors that make the risk more or less likely to occur. As part of this assessment, there should be discussions of the severity of short term and long term consequences if the risk were to occur. By managing risk, the criteria for good enough software will be established.

A survey of many articles and books relating to software delivery criteria was completed. I tried to read text from all viewpoints, such as authors who believed perfect software is obtainable, or that a defined, repeatable process with quality metrics establish delivery criteria, and another author whom believed following the SEI’s capability maturity model processes actually may reduce product quality.

I started my research by reading an article that claims near defect free software can be realized. If there are ways to deliver bug-free software in a timely matter, then there will never be an issue of if it good enough. The article listed nine steps to delivering defect free software. First, believe that defect free software is possible. The premise of this is if developers believe they can write high quality code, they spend more effort on doing so and eliminate bugs as a result. I think attitude does account for something, but a developer can have a great attitude and still misunderstand the requirements hence introducing a defect. The next step is related, think defect free software is important. If developers and managers believe in the benefits of defect free software they will spend the money to get it. The third step is committing to defect free software. This is saying that if you commit to defect free, they necessary sacrifices in functionality and schedule will need to be made. The next steps cover: create simple code designs, tracing all lines of code written, and perform code reviews. These are good practices and will help reduce defects. In large systems, it may not be possible to execute every line of code. Also there is no validation against requirements mentioned anywhere in these steps. The last steps cover testing: build automated QA into code, build and test daily, use automated checking whenever possible (i.e. compiler warnings). These also are good practices. After reading this, I was convinced that delivering defect free software does not guarantee it is "good enough". There is more to good enough software than the quality. There are aspects of timing, functionality, quality and cost combined.

Information Week ran a column in 1996 entitled Utilitarians at the gate. It provided an industry prospective on how companies like Microsoft and Nike decide which bugs to fix and which to release. The article states "the challenge for good enough software developers -- like product developers in most industries -- is to carefully assess trade-offs between cost, features, delivery date, and quality." It says that the SEI approach is that documenting, measuring and making improvements to a defined, repeatable process will result in a higher quality product. However, companies practicing good enough software should not follow the SEI recommendations and instead rely on testers to find and fix software problems. Microsoft does this, with a developer/tester ratio of 1:1. The article also says that the commercial software industry is so chaotic today, that practicing a defined formal process is difficult to do successfully. It suggests a development strategy of co-locating IS staff in business units, because it will give developers a clearer sense of internal customers' priorities. Nike used to have IS determine what was good enough in terms of delivering a fixed set of requirements at a certain defect level. Instead, Nike has now co-located IS development in the business units and it is the business units who decide what is good enough.

The paper The Challenge of "Good Enough" Software, by James Bach provides a very good (and radical) analysis of quality in software development. He starts out by explaining the concept of quality is a complex abstraction and is in the eye of the beholder. Any measurable quality factors are smoke and mirrors and should not be bothered with. He says, "Instead of using our heads to create a universal metric of quality, and then working to optimize everything against that metric, we can use our heads to directly consider the problems that quality is supposed to solve. When those problems are solved, we automatically have good enough quality." I think this makes sense. I have seen a project manager concentrate on improving the statistics of a project instead of the root of the problem. I’m not against metrics, but I think they should only be used to help solve a problem instead of becoming a goal. The paper says that if the number of known defects in the product is a reasonable quality metric, the traditional approach would be to improve quality by removal or prevention of defects. However, following the utilitarian approach, the consequences of each defect would be determined and a decision made on which problems were important to fix. Bach says that in taking this approach the quality metric becomes irrelevant and while prevention is a concern, it should only be focused on the prevention of important problems to the extent necessary. There are five key process ideas to obtain good enough software. They are utilitarian strategy, an evolutionary strategy (i.e. ongoing improvement), heroic teams, a dynamic infrastructure, and dynamic processes. These key processes ideas applied by an empowered team will produce good enough software. Bach says the essence of developing good enough software is "… to be good at knowing the difference between important and unimportant; necessary and unnecessary."

I spoke with a few colleagues in software development to get their opinions and experiences in relation to this topic. All of the professionals I spoke with are in non life-critical commercial development.

Two of the people I spoke with said they practiced good enough software, though had not heard it referred to as that. Alan VanSant, an Enterprise Computing Consultant for IBM and other companies said that the customers he has worked for define the criteria of when things should go out. In many cases Mr. VanSant has recommended that a client postpone delivery or consider architectural improvements before releasing the product because of problems that may impact the end users. In other cases, Mr. VanSant has encouraged the organization the deliver the product as is, and not spend so much time trying to apply architectural design patterns for the sake of doing so.

The second person I spoke to has only worked in organizations with a chaotic development process. He believed that process is necessary to deliver software at all. In many cases, the lack of recording of requirements or ambiguous requirements resulted in the customers demanding features that were clearly out-of-scope. Because the organization failed to follow a process at all, nothing was ever good enough. His viewpoint is that good enough software can only be achieved once an defined and practiced repeatable process is in place.

Before completing much research on the topic I wrote:

"It is accepted in the web domain that we will go into production with known defects, but all efforts are done to minimize defects that end up costing money to the organization sponsoring the project. It is critical to find out which areas result in the highest risks if failure occurs. In the system we just deployed today at work, we went in with 20-some known defects, but none of these will result in a product "re-run", which is a direct money loss to the company. The customer was happy with that."

After completing my research, I still hold the opinion that good enough software is related to the risks of deploying it with the known (and unknown) defects. However, I have found a clearer understanding of how they relate and what role process plays. I feel that process is an important and perhaps unspoken element to good enough software. An organization whom is immature and is developing adhoc solutions will never be in a postion of realistically evaluate the risks and consequences of delivering a product at a certain quality level.

[1] Bach, James, The Challenge of "Good Enough" Software, 1996

[2] McWilliams, Brian S., Utilitarians at the gate, Information Week 1996

[3] Yourdon, Ed, A Reengineering Concept for IT Organizations: 'Good-Enough' Software, ComputerWorld Italia 1995.

[4] Colligan, Terence M., Nine Steps to Delivering Defect-Free Software, 1997.