This is the appendix from the best-selling book Testing Computer Software, 2nd ed.

Copyright © 1988 by Cem Kaner
Copyright © 1993 by Cem Kaner, Jack Falk, Hung Quoc Nguyen

This is part 11 of 13.

HARDWARE

Programs send bad data to devices, ignore error codes coming back, try to use devices that aren't there, and so on. Even if the problem is truly due to a hardware failure, there is also a software error if the software doesn't recognize that the hardware is no longer working correctly.

Click here - Hardware - to continue...

This is the appendix from the best-selling book Testing Computer Software, 2nd ed.

Copyright © 1988 by Cem Kaner
Copyright © 1993 by Cem Kaner, Jack Falk, Hung Quoc Nguyen

This is part 10 of 13.

LOAD CONDITIONS

Programs misbehave when overloaded. A program may fail when working under high volume (lots of work over a long period) or under stress (maximum amount of work all at once). It may fail when it runs out of memory, printers or other "resources." It may fail because it's required to do too much in too little time. All programs have limits. The issues are whether a program can meet its stated limits and how horribly it fails when those limits are exceeded.

Also, some programs create their own load problems, or, in multi-processing situations, make problems for others. They hog computer time or resources or create unnecessary extra work to such an extent that other processes (or themselves later) can't do their tasks.

Click here - Load Conditions - to continue...

This is the appendix from the best-selling book Testing Computer Software, 2nd ed.

Copyright © 1988 by Cem Kaner
Copyright © 1993 by Cem Kaner, Jack Falk, Hung Quoc Nguyen

This is part 9 of 13.

RACE CONDITIONS

In the classic race, there are two possible events, call them EVENT_A and EVENT_B. Both events will happen. The issue is which comes first. EVENT_A almost always precedes EVENT_B. There are logical grounds for expecting EVENT_A to precede EVENT_B. However, under rare and restricted conditions, EVENT_B can "win the race," and occur just before EVENT_A. We have a race condition whenever EVENT_B precedes EVENT_A. We have a race condition bug if the program fails when this happens. Usually the program fails because the programmer didn't anticipate the possibility of EVENT_B preceding EVENT_A, so he didn't write any code to deal with it.

Few testers look for race conditions. If they find an "irreproducible" bug, few think about timing issues (races) when trying to reproduce it. Many people find timing issues hard to conceptualize or hard to understand. We provide more than our usual amount of detail in the examples below, hoping that this will make the overall concept easier to understand.

Click here - Race Conditions - to continue...

This is the appendix from the best-selling book Testing Computer Software, 2nd ed.

Copyright © 1988 by Cem Kaner
Copyright © 1993 by Cem Kaner, Jack Falk, Hung Quoc Nguyen

This is part 8 of 13.

ERRORS IN HANDLING OR INTERPRETING DATA

Data are passed from one part of a program to another, and from one program to another. In the process, the data might be misinterpreted or corrupted.

Click here - Errors in Handling or Interpreting Data - to continue...

This is the appendix from the best-selling book Testing Computer Software, 2nd ed.

Copyright © 1988 by Cem Kaner
Copyright © 1993 by Cem Kaner, Jack Falk, Hung Quoc Nguyen

This is part 7 of 13.

CONTROL FLOW ERRORS

The control flow of a program describes what it will do next, under what circumstances. A control flow error occurs when the program does the wrong thing next. Extreme control flow errors stop the program or cause it to run amok. Many simple errors lead to spectacular misbehavior.

Click here - Control Flow Errors - to continue...

This is the appendix from the best-selling book Testing Computer Software, 2nd ed.

Copyright © 1988 by Cem Kaner
Copyright © 1993 by Cem Kaner, Jack Falk, Hung Quoc Nguyen

This is part 6 of 13.

Initial and Later States

Before you can use a function, the program may have to initialize it. Typical initialization steps include identifying the function's variables, defining their types, allocating memory for them, and setting them to default values (such as 0). The program may have to read a disk file that contains defaults and other configuration information. What happens when the file is not there? Initialization steps might be done when the program is loaded (data defaults can be loaded into memory along with the program), when it is started, when the function is first called, or each time the function is called.

Initialization needs and strategy vary widely across languages. For example:

• In many languages, a function's local variables keep their values from one call to the next. If a variable is supposed to keep the same value, the function can set the variable's value once and leave it alone thereafter. The function usually has to reset other variables to their starting values.
• In other languages, local variables are erased from memory on exit from the function. Whenever the program calls a function, it must redefine its variables, allocate memory for them, and assign starting values.
• Some languages allow the programmer to specify whether a variable should stay in memory or be erased after each function call.
• Some compilers provide initialization support. The programmer can specify a starting value for a variable; the compiler will make sure that this loads into memory with the program. If the programmer doesn't assign an initial value, the compiler sets it to 0. Other compilers, even for the same language, do not provide this support. The function must set each variable's value the first time it's called. To avoid resetting each variable every time it's called, the function must know whether it's been called before.

Initialization failures usually show up the first time the function is called or the second time, if it doesn't reinitialize variables correctly. Reinitialization failures may be path-dependent. If you reach a function in a "normal" way, it works fine. However, if you take an "abnormal" route, the program might branch into the function at some point after the initialization code. Programmers often treat backing up to modify data or redo calculations as abnormal.

Click here - Initial and Later States - to continue...

This is the appendix from the best-selling book Testing Computer Software, 2nd ed.

Copyright © 1988 by Cem Kaner
Copyright © 1993 by Cem Kaner, Jack Falk, Hung Quoc Nguyen

This is part 5 of 13.

CALCULATION ERRORS

The program calculates a number and gets the wrong result. This can happen for one of three types of reasons:

• Bad logic: There can be a typing error, like A-A instead of A+A. Or the programmer might break a complex expression into a set of simpler ones, but get the simplification wrong. Or he might use an incorrect formula, or one inapplicable to the data at hand. This third case is a design error. The code does what the programmer intendedóit's his conception of what the code should do that is wrong.
• Bad arithmetic: There might be an error in the coding of a basic function, such as addition, multiplication, or exponentiation. The error might show up whenever the function is used (2 + 2 = -5) or it might be restricted to rare special cases. In either case, any program that uses the function can fail.
• Imprecise calculation: If the program uses floating point arithmetic, it loses precision as it calculates, because of roundoff and truncation errors. After many intermediate errors, it may claim that 2 + 2 works out to -5 even though none of the steps in the program contains a logical error.

This area is huge and this section only begins to scratch its surface. For an introduction to the larger area, read Conte and deBoor (1980) and Knuth (1981). For a second source on topics in Conte and deBoor, try Carnahan, Luther & Wilkes (1969).

This is the appendix from the best-selling book Testing Computer Software, 2nd ed.

Copyright © 1988 by Cem Kaner
Copyright © 1993 by Cem Kaner, Jack Falk, Hung Quoc Nguyen

This is part 4 of 13.

BOUNDARY-RELATED ERRORS

A boundary describes a change-point for a program. The program is supposed to work one way for anything on one side of the boundary. It does something different for anything on the other side. The classic "things" on opposite sides of boundaries are data values. There are three standard boundary bugs:

• Mishandling of the boundary case: If a program adds any two numbers that are less than 100, and rejects any greater than 100, what does it do when you enter exactly 100? What is it supposed to do?
• Wrong boundary: The specification says the program should add any two numbers less than 100 but it rejects anything greater than 95.
• Mishandling of cases outside the boundary: Values on one side of the boundary are impossible, unlikely, unacceptable, unwanted. No code was written for them. Does the program successfully reject values greater than 100 or does it crash when it gets one?

We treat the concept of boundaries more broadly. Boundaries describe a way of thinking about a program and its behavior around its limits. There are many types of limits: largest, oldest, latest, longest, most recent, first time, etc. The same types of bugs can happen with any of them so why not think of them in the same terms?

This is the appendix from the best-selling book Testing Computer Software, 2nd ed.

Copyright © 1988 by Cem Kaner
Copyright © 1993 by Cem Kaner, Jack Falk, Hung Quoc Nguyen

This is part 3 of 13.

ERROR HANDLING

Errors in dealing with errors are among the most common bugs. Error handling errors include failure to anticipate the possibility of errors and protect against them, failure to notice error conditions, and failure to deal with detected errors in a reasonable way. Note that error messages were discussed above.

This is the appendix from the best-selling book Testing Computer Software, 2nd ed.

Copyright © 1988 by Cem Kaner
Copyright © 1993 by Cem Kaner, Jack Falk, Hung Quoc Nguyen

This is part 2 of 13.

The user interface (UI) includes all aspects of the product that involve the user. The UI designer tries to strike a balance between

• functionality
• time to learn how to use the program
• how well the user remembers how to use the program
• speed of performance
• rate of user errors
• the user's satisfaction with the program

In seeking a good balance, the designer weighs the experience and needs of the people she expects to use the program against the capabilities of the equipment and available software technology. An error in the UI results in a suboptimal match between the user and the program.

Because tradeoffs are unavoidable in UI design, a good designer might deliberately make any of many of the "errors" listed below. Don't take this list as gospel. If you are at all unsure, listen to the designer's reasoning before condemning one of her choices. See Baecker & Buxton (1987), Helander (1991), Laurel (1990, 1991), Rubenstein & Hersh, (1984), Schneiderman (1987), and Smith and Mosier (1984) for excellent introductions to user interface design, including extended discussion of many of the issues raised in this Appendix.

Throughout this Appendix we write as if you were the user of the program. As a tester of it, you will certainly use it heavily. Realize that other people will also use the program and they will have different problems from you. Try to be empathetic.

This is the appendix from the best-selling book Testing Computer Software, 2nd ed.

Copyright © 1988 by Cem Kaner
Copyright © 1993 by Cem Kaner, Jack Falk, Hung Quoc Nguyen

This is part 1 of 13.

This Appendix describes over 400 bugs. The descriptions are short, including only what we considered the most interesting information. It's worthwhile reading this list, even though you may find it boring. On first reading it provides a frame of reference, details, and background about problems you should look for while testing. Its greater value is as an organized list of program problems for future reference. A good list, built up with time and experience, can be a powerful tool.

By Hans Buwalda, CTO, LogiGear Corporation


Introduction

Keyword based testing is gaining ground. More and more organizations see this model, in which tests are not scripted but written as a series of keywords with arguments, as a valuable alternative to record and playback, or scripting of tests. A good theoretical basis for keywords can be found in the well known automation book Software Test Automation, by Dorothy Graham and Mark Fewster, and also in numerous articles and white papers on the LogiGear website. In this article I want to list some of the typical advantages and risks of keywords.

Other Articles by this Author

• Action Figures
• Agile Test Development with ABT
• Automation and Requirements
• Bonus Bugs
• Business Test Policies
• Capitalizing Testware as an Asset
• Getting Automated Testing Under Control
• Is Action Based Testing an Automation Technique?
• Is Test Automation the Same as Programming Tests?
• Key Success Factors for Keyword Driven Testing
• The First Holy Grail of Test Design
• Key Principles of Test Design
• Key Success Factors for Keyword Driven Testing
• The Potential and Risks of Keyword Based Testing
• The Second Holy Grail of Test Design
• Soap Opera Testing
• Test Automation: Divide and Conquer
• Test Governance
• The Third Holy Grail of Test Design

Comprehensive template to help develop a Load/Performance Test Plan.

By Rob Pirozzi

Introduction

All to often, senior management judges software testing though the lens of potential cost savings. Test automation and outsourcing are simplistically looked at as simply methods to reduce the costs of software testing. But the sad truth is that simply automating or offshoring a poor software testing effort will simply yield an automated and offshore poor testing effort. And if the same simplistic approach was taken in the planning of automation and offshoring, the costs may not even be lower.

It is critically important for management to define "success" for their software testing efforts. While cost is one metric, it is not the sole metric. Other issues that should also be taken into account include:

• Breadth of testing coverage
• Test reusability
• Value of the data reported by software testing
• The quality of the software under test

Using these criteria, management should come up with a clear definition of success that is consistent with their testing strategy and testing methodology. Of course, if there is no strategy and methodology, then that must be developed first - but that is a topic for another article

This paper will discuss the importance of these other non-cost success criteria.

By Hans Buwalda

Introduction

A common issue that I come across in projects is the relationship between test automation and programming. In this article I want to highlight some of the differences that I feel exist between the two.

After many decades of often slow and hard fought over progress, software engineering is now a well established profession, with recognized methods and practices. Automated testing, however, is still a relatively new and often misunderstood phenomenon. Often software engineers get involved in software testing and try to apply their programming knowledge and experience. For many aspects of software testing this is a good thing, but over the years I have come to believe that automated testing poses its own distinct properties and challenges, and a practitioner should try to understand those and work with them.

This article will explore the differences between software engineering and designing automated tests.

By Rob Pirozzi

Introduction

Many companies have come to realize that software testing is much more than a task that happens at the end of a software development cycle. They have come to understand that software testing is a strategic imperative and a discipline that can have a substantial impact on the success of an organization that develops software. Many of these companies are coming to embrace the ideas that are central to the latest phase in the evolution of software testing, Software Testing 3.0 (See the links below for more on Software Testing 3.0).

Implementing Software Testing 3.0 is a major undertaking, and as such, should not be embarked upon without adequate planning. While implementing Software Testing 3.0 will require substantial effort, the benefits in terms of software quality, cost and time savings, and mitigation of risk from in market failures are substantial.

Software Testing 3.0 has both organizational and process ramifications. These ramifications need to be fully understood to insure success implementing Software Testing 3.0. This paper will explore these organizational and process ramifications to help with an organization's planning efforts.

This white paper traces the evolution of software testing through its first two phases, and then presents the current state-of-the practice: Software Testing 3.0, an independent and strategic approach to software quality. The paper concludes with a discussion for senior executives on how and why software testing can finally meet and exceed management expectations, illustrated by real-world case studies showing how two companies benefited from Software Testing 3.0 to test better and faster while lowering their costs.

By Hung Q. Nguyen and Michael Hackett

Objectives Introduce DAST, the Diagnostic Approach to Software Testing. Share our experiences using this method. Provide you with a process guideline.

Other classic articles...

By Hung Q. Nguyen

Today's Agenda

Applied Gray-box Testing

• The Testing Paradigms
• Gray-box Testing Strategies
• Specific Test Strategies, Designs & Examples

This is a web adaptation of a presentation given by LogiGear founder and CEO Hung Q. Nguyen for a RADVIEW Webinar