By Hans Buwalda, CTO, LogiGear Corporation

As I write this article I am sitting at a table at StarEast, one of the major testing conferences. As you can expect from a testing conference, a lot of talk and discussion is about bugs and how to find them. What I have noticed in some of these discussions, however, is a lack of differentiation between the different types of bugs that I think is essential for testing success.

To set the stage I would like to distinguish between different categories of bugs (For a more extensive differentiation of bugs look at the very interesting presentation by Giri Vijayaraghavan and Cem Kaner "Bug taxonomies: Use them to generate better tests").

By Hung Q. Nguyen and Rob Pirozzi

All too often, software development organizations look at automating software testing as a means of executing existing test cases faster. Very frequently there is no strategic or methodological underpinning to such an effort. The approach is one of running test cases faster is better, which will help to deliver software faster.

Even in organizations that start to build a test automation effort on a good foundation of strategy, methodology, and supporting tools, taking such a narrow view of the test automation effort misses a huge dynamic.

How LeapFrog Successfully Outsources 80% of Its Software Testing

When LeapFrog embarked upon an ambitious strategy for new market leadership in the intensely competitive children's education consumer electronics industry, improved operational efficiency was critical to success. LogiGear helped LeapFrog change how they approached software testing to enable them to meet those demands. This case study will describe in detail how LogiGear helped LeapFrog introduce the highest volume of new platform and product introductions in the company’s history while reducing testing costs and allowing the LeapFrog QA team to work more reasonable hours confident in the knowledge that their testing needs were being met. Download...

Click here - Making the Leap to Vietnam with LogiGear - to download the case study (pdf format)...

May 2008

The following is a transcript of a May 7, 2008 interview with Hung Q. Nguyen, founder and CEO of LogiGear Corporation and coauthor of the best selling textbook Testing Computer Software.

Interviewer: When it comes to software testing, what concerns or issues are you hearing from software developers?

Hung Q. Nguyen: The most pressing concern that I hear from companies that develop software is that they are not testing fast enough. With advancements in technology and development tools the rates for building software have become very fast. Development is outpacing software testing which is driving a huge demand for test automation. However, test automation, done incorrectly can be more of a hindrance.

Click here - Hung Q. Nguyen Discusses Software Testing - to continue...

This is the appendix from the best-selling book Testing Computer Software, 2nd ed.

Copyright © 1988 by Cem Kaner
Copyright © 1993 by Cem Kaner, Jack Falk, Hung Quoc Nguyen

This is part 13 of 13.

TESTING ERRORS

This section deals with technical, procedural, and reporting errors made by testers and Testing Groups. Even though these aren't problems in the programs per se, you'll run into them when testing programs. Our focus is on suggestions for dealing with these problems, since they're under your control.

Click here - Common Software Errors - Testing Errors - to continue...

This is the appendix from the best-selling book Testing Computer Software, 2nd ed.

Copyright © 1988 by Cem Kaner
Copyright © 1993 by Cem Kaner, Jack Falk, Hung Quoc Nguyen

This is part 12 of 13.

SOURCE, VERSION, AND ID CONTROL

If you're supposed to have Version 2.43 of the program, but some of the pieces you have are from 2.42 and others are advance bits of 2.44, you have a mess. You must know what you have, you must be able to tell from the code what you have, and what you have must be what you're supposed to have. If not, report a bug.

Some people calls these Bureaucracy Bugs because they reflect failures of labeling and procedure rather than operational failures. Only bureaucrats would worry about such things, right? Wrong. Or maybe right, but so what? They must be worried about otherwise the products shipped to customers will not be what you think.

Click here - Common Software Errors - Source, Version, and ID Control - to continue...

This is the appendix from the best-selling book Testing Computer Software, 2nd ed.

Copyright © 1988 by Cem Kaner
Copyright © 1993 by Cem Kaner, Jack Falk, Hung Quoc Nguyen

This is part 11 of 13.

HARDWARE

Programs send bad data to devices, ignore error codes coming back, try to use devices that aren't there, and so on. Even if the problem is truly due to a hardware failure, there is also a software error if the software doesn't recognize that the hardware is no longer working correctly.

Click here - Hardware - to continue...

This is the appendix from the best-selling book Testing Computer Software, 2nd ed.

Copyright © 1988 by Cem Kaner
Copyright © 1993 by Cem Kaner, Jack Falk, Hung Quoc Nguyen

This is part 10 of 13.

LOAD CONDITIONS

Programs misbehave when overloaded. A program may fail when working under high volume (lots of work over a long period) or under stress (maximum amount of work all at once). It may fail when it runs out of memory, printers or other "resources." It may fail because it's required to do too much in too little time. All programs have limits. The issues are whether a program can meet its stated limits and how horribly it fails when those limits are exceeded.

Also, some programs create their own load problems, or, in multi-processing situations, make problems for others. They hog computer time or resources or create unnecessary extra work to such an extent that other processes (or themselves later) can't do their tasks.

Click here - Load Conditions - to continue...

This is the appendix from the best-selling book Testing Computer Software, 2nd ed.

Copyright © 1988 by Cem Kaner
Copyright © 1993 by Cem Kaner, Jack Falk, Hung Quoc Nguyen

This is part 9 of 13.

RACE CONDITIONS

In the classic race, there are two possible events, call them EVENT_A and EVENT_B. Both events will happen. The issue is which comes first. EVENT_A almost always precedes EVENT_B. There are logical grounds for expecting EVENT_A to precede EVENT_B. However, under rare and restricted conditions, EVENT_B can "win the race," and occur just before EVENT_A. We have a race condition whenever EVENT_B precedes EVENT_A. We have a race condition bug if the program fails when this happens. Usually the program fails because the programmer didn't anticipate the possibility of EVENT_B preceding EVENT_A, so he didn't write any code to deal with it.

Few testers look for race conditions. If they find an "irreproducible" bug, few think about timing issues (races) when trying to reproduce it. Many people find timing issues hard to conceptualize or hard to understand. We provide more than our usual amount of detail in the examples below, hoping that this will make the overall concept easier to understand.

Click here - Race Conditions - to continue...

This is the appendix from the best-selling book Testing Computer Software, 2nd ed.

Copyright © 1988 by Cem Kaner
Copyright © 1993 by Cem Kaner, Jack Falk, Hung Quoc Nguyen

This is part 8 of 13.

ERRORS IN HANDLING OR INTERPRETING DATA

Data are passed from one part of a program to another, and from one program to another. In the process, the data might be misinterpreted or corrupted.

Click here - Errors in Handling or Interpreting Data - to continue...

This is the appendix from the best-selling book Testing Computer Software, 2nd ed.

Copyright © 1988 by Cem Kaner
Copyright © 1993 by Cem Kaner, Jack Falk, Hung Quoc Nguyen

This is part 7 of 13.

CONTROL FLOW ERRORS

The control flow of a program describes what it will do next, under what circumstances. A control flow error occurs when the program does the wrong thing next. Extreme control flow errors stop the program or cause it to run amok. Many simple errors lead to spectacular misbehavior.

Click here - Control Flow Errors - to continue...

This is the appendix from the best-selling book Testing Computer Software, 2nd ed.

Copyright © 1988 by Cem Kaner
Copyright © 1993 by Cem Kaner, Jack Falk, Hung Quoc Nguyen

This is part 6 of 13.

Initial and Later States

Before you can use a function, the program may have to initialize it. Typical initialization steps include identifying the function's variables, defining their types, allocating memory for them, and setting them to default values (such as 0). The program may have to read a disk file that contains defaults and other configuration information. What happens when the file is not there? Initialization steps might be done when the program is loaded (data defaults can be loaded into memory along with the program), when it is started, when the function is first called, or each time the function is called.

Initialization needs and strategy vary widely across languages. For example:

• In many languages, a function's local variables keep their values from one call to the next. If a variable is supposed to keep the same value, the function can set the variable's value once and leave it alone thereafter. The function usually has to reset other variables to their starting values.
• In other languages, local variables are erased from memory on exit from the function. Whenever the program calls a function, it must redefine its variables, allocate memory for them, and assign starting values.
• Some languages allow the programmer to specify whether a variable should stay in memory or be erased after each function call.
• Some compilers provide initialization support. The programmer can specify a starting value for a variable; the compiler will make sure that this loads into memory with the program. If the programmer doesn't assign an initial value, the compiler sets it to 0. Other compilers, even for the same language, do not provide this support. The function must set each variable's value the first time it's called. To avoid resetting each variable every time it's called, the function must know whether it's been called before.

Initialization failures usually show up the first time the function is called or the second time, if it doesn't reinitialize variables correctly. Reinitialization failures may be path-dependent. If you reach a function in a "normal" way, it works fine. However, if you take an "abnormal" route, the program might branch into the function at some point after the initialization code. Programmers often treat backing up to modify data or redo calculations as abnormal.

Click here - Initial and Later States - to continue...

This is the appendix from the best-selling book Testing Computer Software, 2nd ed.

Copyright © 1988 by Cem Kaner
Copyright © 1993 by Cem Kaner, Jack Falk, Hung Quoc Nguyen

This is part 5 of 13.

CALCULATION ERRORS

The program calculates a number and gets the wrong result. This can happen for one of three types of reasons:

• Bad logic: There can be a typing error, like A-A instead of A+A. Or the programmer might break a complex expression into a set of simpler ones, but get the simplification wrong. Or he might use an incorrect formula, or one inapplicable to the data at hand. This third case is a design error. The code does what the programmer intendedóit's his conception of what the code should do that is wrong.
• Bad arithmetic: There might be an error in the coding of a basic function, such as addition, multiplication, or exponentiation. The error might show up whenever the function is used (2 + 2 = -5) or it might be restricted to rare special cases. In either case, any program that uses the function can fail.
• Imprecise calculation: If the program uses floating point arithmetic, it loses precision as it calculates, because of roundoff and truncation errors. After many intermediate errors, it may claim that 2 + 2 works out to -5 even though none of the steps in the program contains a logical error.

This area is huge and this section only begins to scratch its surface. For an introduction to the larger area, read Conte and deBoor (1980) and Knuth (1981). For a second source on topics in Conte and deBoor, try Carnahan, Luther & Wilkes (1969).

This is the appendix from the best-selling book Testing Computer Software, 2nd ed.

Copyright © 1988 by Cem Kaner
Copyright © 1993 by Cem Kaner, Jack Falk, Hung Quoc Nguyen

This is part 4 of 13.

BOUNDARY-RELATED ERRORS

A boundary describes a change-point for a program. The program is supposed to work one way for anything on one side of the boundary. It does something different for anything on the other side. The classic "things" on opposite sides of boundaries are data values. There are three standard boundary bugs:

• Mishandling of the boundary case: If a program adds any two numbers that are less than 100, and rejects any greater than 100, what does it do when you enter exactly 100? What is it supposed to do?
• Wrong boundary: The specification says the program should add any two numbers less than 100 but it rejects anything greater than 95.
• Mishandling of cases outside the boundary: Values on one side of the boundary are impossible, unlikely, unacceptable, unwanted. No code was written for them. Does the program successfully reject values greater than 100 or does it crash when it gets one?

We treat the concept of boundaries more broadly. Boundaries describe a way of thinking about a program and its behavior around its limits. There are many types of limits: largest, oldest, latest, longest, most recent, first time, etc. The same types of bugs can happen with any of them so why not think of them in the same terms?

This is the appendix from the best-selling book Testing Computer Software, 2nd ed.

Copyright © 1988 by Cem Kaner
Copyright © 1993 by Cem Kaner, Jack Falk, Hung Quoc Nguyen

This is part 3 of 13.

ERROR HANDLING

Errors in dealing with errors are among the most common bugs. Error handling errors include failure to anticipate the possibility of errors and protect against them, failure to notice error conditions, and failure to deal with detected errors in a reasonable way. Note that error messages were discussed above.

This is the appendix from the best-selling book Testing Computer Software, 2nd ed.

Copyright © 1988 by Cem Kaner
Copyright © 1993 by Cem Kaner, Jack Falk, Hung Quoc Nguyen

This is part 2 of 13.

The user interface (UI) includes all aspects of the product that involve the user. The UI designer tries to strike a balance between

• functionality
• time to learn how to use the program
• how well the user remembers how to use the program
• speed of performance
• rate of user errors
• the user's satisfaction with the program

In seeking a good balance, the designer weighs the experience and needs of the people she expects to use the program against the capabilities of the equipment and available software technology. An error in the UI results in a suboptimal match between the user and the program.

Because tradeoffs are unavoidable in UI design, a good designer might deliberately make any of many of the "errors" listed below. Don't take this list as gospel. If you are at all unsure, listen to the designer's reasoning before condemning one of her choices. See Baecker & Buxton (1987), Helander (1991), Laurel (1990, 1991), Rubenstein & Hersh, (1984), Schneiderman (1987), and Smith and Mosier (1984) for excellent introductions to user interface design, including extended discussion of many of the issues raised in this Appendix.

Throughout this Appendix we write as if you were the user of the program. As a tester of it, you will certainly use it heavily. Realize that other people will also use the program and they will have different problems from you. Try to be empathetic.

This is the appendix from the best-selling book Testing Computer Software, 2nd ed.

Copyright © 1988 by Cem Kaner
Copyright © 1993 by Cem Kaner, Jack Falk, Hung Quoc Nguyen

This is part 1 of 13.

This Appendix describes over 400 bugs. The descriptions are short, including only what we considered the most interesting information. It's worthwhile reading this list, even though you may find it boring. On first reading it provides a frame of reference, details, and background about problems you should look for while testing. Its greater value is as an organized list of program problems for future reference. A good list, built up with time and experience, can be a powerful tool.

By Hans Buwalda, CTO, LogiGear Corporation


Introduction

Keyword based testing is gaining ground. More and more organizations see this model, in which tests are not scripted but written as a series of keywords with arguments, as a valuable alternative to record and playback, or scripting of tests. A good theoretical basis for keywords can be found in the well known automation book Software Test Automation, by Dorothy Graham and Mark Fewster, and also in numerous articles and white papers on the LogiGear website. In this article I want to list some of the typical advantages and risks of keywords.

Other Articles by this Author

• Action Figures
• Agile Test Development with ABT
• Automation and Requirements
• Bonus Bugs
• Business Test Policies
• Capitalizing Testware as an Asset
• Getting Automated Testing Under Control
• Is Action Based Testing an Automation Technique?
• Is Test Automation the Same as Programming Tests?
• Key Success Factors for Keyword Driven Testing
• The First Holy Grail of Test Design
• Key Principles of Test Design
• Key Success Factors for Keyword Driven Testing
• The Potential and Risks of Keyword Based Testing
• The Second Holy Grail of Test Design
• Soap Opera Testing
• Test Automation: Divide and Conquer
• Test Governance
• The Third Holy Grail of Test Design

Comprehensive template to help develop a Load/Performance Test Plan.

By Rob Pirozzi

Introduction

All to often, senior management judges software testing though the lens of potential cost savings. Test automation and outsourcing are simplistically looked at as simply methods to reduce the costs of software testing. But the sad truth is that simply automating or offshoring a poor software testing effort will simply yield an automated and offshore poor testing effort. And if the same simplistic approach was taken in the planning of automation and offshoring, the costs may not even be lower.

It is critically important for management to define "success" for their software testing efforts. While cost is one metric, it is not the sole metric. Other issues that should also be taken into account include:

• Breadth of testing coverage
• Test reusability
• Value of the data reported by software testing
• The quality of the software under test

Using these criteria, management should come up with a clear definition of success that is consistent with their testing strategy and testing methodology. Of course, if there is no strategy and methodology, then that must be developed first - but that is a topic for another article

This paper will discuss the importance of these other non-cost success criteria.

By Hans Buwalda

Introduction

A common issue that I come across in projects is the relationship between test automation and programming. In this article I want to highlight some of the differences that I feel exist between the two.

After many decades of often slow and hard fought over progress, software engineering is now a well established profession, with recognized methods and practices. Automated testing, however, is still a relatively new and often misunderstood phenomenon. Often software engineers get involved in software testing and try to apply their programming knowledge and experience. For many aspects of software testing this is a good thing, but over the years I have come to believe that automated testing poses its own distinct properties and challenges, and a practitioner should try to understand those and work with them.

This article will explore the differences between software engineering and designing automated tests.

By Rob Pirozzi

Introduction

Many companies have come to realize that software testing is much more than a task that happens at the end of a software development cycle. They have come to understand that software testing is a strategic imperative and a discipline that can have a substantial impact on the success of an organization that develops software. Many of these companies are coming to embrace the ideas that are central to the latest phase in the evolution of software testing, Software Testing 3.0 (See the links below for more on Software Testing 3.0).

Implementing Software Testing 3.0 is a major undertaking, and as such, should not be embarked upon without adequate planning. While implementing Software Testing 3.0 will require substantial effort, the benefits in terms of software quality, cost and time savings, and mitigation of risk from in market failures are substantial.

Software Testing 3.0 has both organizational and process ramifications. These ramifications need to be fully understood to insure success implementing Software Testing 3.0. This paper will explore these organizational and process ramifications to help with an organization's planning efforts.

This white paper traces the evolution of software testing through its first two phases, and then presents the current state-of-the practice: Software Testing 3.0, an independent and strategic approach to software quality. The paper concludes with a discussion for senior executives on how and why software testing can finally meet and exceed management expectations, illustrated by real-world case studies showing how two companies benefited from Software Testing 3.0 to test better and faster while lowering their costs.

By Hung Q. Nguyen and Michael Hackett

Objectives Introduce DAST, the Diagnostic Approach to Software Testing. Share our experiences using this method. Provide you with a process guideline.

Other classic articles...

By Hung Q. Nguyen

Today's Agenda

Applied Gray-box Testing

• The Testing Paradigms
• Gray-box Testing Strategies
• Specific Test Strategies, Designs & Examples

This is a web adaptation of a presentation given by LogiGear founder and CEO Hung Q. Nguyen for a RADVIEW Webinar