Cover

Half-title

Title

Copyright

Dedication

Contents

Foreword to the Second Edition

Foreword to the First Edition

Acknowledgments

Second Edition Acknowledgments

First Edition Acknowledgments

1 Introduction

1.1 Purpose of the Book

1.2 Readership

1.3 How to Read This Book

1.4 Structure and Content of This Book

Part One The Traditional Testing Process

2 An Overview of Testing

2.1 Introduction

2.2 The Challenge of Testing

2.3 What Is Testing?

2.4 Verification and Validation

2.5 What Is the Cost of Not Testing?

2.6 Testing – the Bottom Line

2.7 Additional Information

3 Testing Techniques

3.1 Introduction

3.2 General Testing Techniques

3.2.1 Positive and Negative Testing

3.2.2 White Box and Black Box Testing

3.2.3 Experienced Based Testing

3.2.4 Automated Software Testing

3.3 Functional Testing Techniques

3.3.1 Equivalence Partitioning

3.3.2 Boundary Value Analysis

3.3.3 Intrusive Testing

3.3.4 Random Testing

3.3.5 State Transition Analysis

3.3.6 Static Testing

3.3.7 Thread Testing

3.3.8 Pairwise Testing

3.4 Nonfunctional Testing Techniques

3.4.1 Configuration/Installation Testing

3.4.2 Compatibility and Interoperability Testing

3.4.3 Documentation and Help Testing

3.4.4 Fault Recovery Testing

3.4.5 Performance Testing

3.4.6 Reliability Testing

3.4.7 Security Testing

3.4.8 Stress Testing

3.4.9 Usability Testing

3.4.10 Volume Testing

3.5 Further Reading on Testing Techniques

4 The Management and Planning of Testing

4.1 Introduction

4.2 The Organization of Testing

4.3 Roles and Responsibilities

4.3.1 Overview

4.3.2 Testing Manager

4.3.3 Test Team Leader

4.3.4 Test Analyst

4.3.5 Tester

4.3.6 Independent Test Observer

4.3.7 Supplementary Testing Roles

Test Automation Architect

Test Automation Analyst

Exploratory Tester

4.4 The Testing Phases

4.4.1 Overview

4.4.2 Unit Testing

4.4.3 Integration Testing

4.4.4 System Testing

4.4.5 Systems Integration Testing

4.4.6 Acceptance Testing

4.4.7 Regression Testing

4.4.8 When to Stop Testing

4.5 Role of the V Model in Planning

4.6 The Management of Test Requirements

4.7 The Role and Use of Configuration Management

4.8 The Role and Use of Defect Tracking

4.9 The Role of Risk in Test Planning and Management

5 Unit Testing

5.1 Overview

5.2 Unit Test Approach

5.3 Unit Test Data Requirements

5.4 Roles and Responsibilities

5.5 Planning and Resources

5.6 Inputs

5.7 Testing Techniques for Unit Testing

5.8 Outputs

6 Integration Testing

6.1 Overview

6.2 Integration Test Approach

6.3 Integration Test Data Requirements

6.4 Roles and Responsibilities

6.5 Planning and Resources

6.6 Inputs

6.7 Testing Techniques for Integration Testing

6.8 Outputs

7 System Testing

7.1 Overview

7.2 System Test Approach

7.3 System Test Data Requirements

7.4 Roles and Responsibilities

7.5 Planning and Resources

7.6 Inputs

7.7 Testing Techniques for System Testing

7.8 Outputs

8 Systems Integration Testing

8.1 Overview

8.2 Systems Integration Test Approach

8.3 System Integration Test Data Requirements

8.4 Roles and Responsibilities

8.5 Planning and Resources

8.6 Inputs

8.7 Testing Techniques for Systems Integration Testing

8.8 Outputs

9 User Acceptance Testing

9.1 Overview

9.2 User Acceptance Test Approach

9.3 User Acceptance Test Data Requirements

9.4 Roles and Responsibilities

9.5 Planning and Resources

9.6 Inputs

9.7 Testing Techniques for User Acceptance Testing

9.8 Outputs

10 Operations Acceptance Testing

10.1 Overview

10.2 Operations Acceptance Test Approach

10.3 Operations Acceptance Test Data Requirements

10.4 Roles and Responsibilities

10.5 Planning and Resources

10.6 Inputs

10.7 Testing Techniques for Operations Acceptance Testing

10.8 Outputs

11 Regression Testing

11.1 Overview

11.2 Regression Test Approach

11.3 Regression Test Data Requirements

11.4 Roles and Responsibilities

11.5 Planning and Resources

11.6 Inputs

11.7 Testing Techniques for Regression Testing

11.8 Outputs

12 Improving the Testing Process

12.1 Introduction

12.2 Overview of the Role and Use of Metrics

12.3 Metrics Typically Used Within the Testing Process

Defect Detection Effectiveness Percentage or DDE

Defect Removal Effectiveness Percentage or DRE

Test Case Design Efficiency Percentage or TDE

12.4 Setting Up and Administering a Metrics Program

12.5 A Proposal for a Simple and Effective Metrics Set

12.6 Further Reading

13 Introduction, Adoption, and Maintenance of the Testing Process

13.1 Introduction

13.2 Introduction and Adoption of a Testing Process

13.2.1 Overview

Establishing the Requirement

Strategy for Introduction and Adoption

13.3 Maintenance of the Testing Process

13.4 A Proposal for a Quick Start Test Process

14 Agile Testing

14.1 Introduction

14.2 Overview of Agile Testing

14.3 Agile Quality Management Practices

14.4 A Proposal for an Effective Subset of Agile Best Practices

14.5 Conclusion

Part Two The Testing Process in the Real World: Illustrative Case Studies

British Library

Reuters Product Acceptance Group

Crown Quality Assurance Group

The Wine Society

Automatic Data Processing Limited

Confirmit

15 Case Study 1: The British Library

15.1 Overview of the Organization

15.2 Characteristics of the Testing Requirement

15.3 The Management and Planning of Testing

15.4 Roles and Responsibilities

Testing Manager

Test Team Leader

Test Analyst

Tester

Independent Test Observer

15.5 Testing Phases

15.6 Artifacts

15.7 Process Improvement

16 Case Study 2: Reuters Product Acceptance Group

16.1 Overview of the Organization

16.2 Testing Requirements

16.3 The Management and Planning of Testing

16.4 Roles and Responsibilities

PAG Manager

RDBGUI Acceptance Managers

RDBGUI Test Planner

Test Analyst

Tester

Independent Test Observer

16.5 Testing Phases

Unit, Link, System, and Integration Testing

Acceptance Testing

Regression Testing

User Testing

16.6 Artifacts

16.7 Process Improvement

17 Case Study 3: Crown Quality Assurance Group

17.1 Overview of the Organization

17.2 Testing Requirements

17.3 The Management and Planning of Testing

17.4 Roles and Responsibilities

QA Team Manager

Test Analyst/Tester

Developer Representative

Client Representative

17.5 Testing Phases

System Testing

Acceptance Testing

Crash Testing

Regression Testing

17.6 Artifacts

17.7 Process Improvement

18 Case Study 4: The Wine Society

18.1 Overview of the Organization

18.2 Testing Requirements

18.3 The Management and Planning of Testing

18.4 Roles and Responsibilities

Project Manager

User Representative Tester

Testing Consultant

Supplier Representative

18.5 Testing Phases

Acceptance Testing

Regression Testing

18.6 Artifacts

18.7 Process Improvement

19 Case Study 5: Automatic Data Processing Limited

19.1 Overview of the Organization

19.2 Characteristics of the Testing Requirement

19.3 The Management and Planning of Testing

19.4 Roles and Responsibilities

Product Development Manager

Development Project Manager

Senior Quality Assurance Analyst

Quality Assurance Analyst

19.5 Testing Phases

Skim Testing

Acceptance Testing

Regression Testing

19.6 Artifacts

19.7 Process Improvement

20 Case Study 6: Confirmit

20.1 Overview of the Organization

20.2 Characteristics of the Testing Requirement

20.3 The Management and Planning of Testing

20.4 Roles and Responsibilities

Client Manager

Confirmit Head of Research and Development

Quality Assurance Manager

Development Team

Quality Assurance Team

Client User Representatives

Process Champion

20.5 Testing Phases

20.6 Artifacts

20.7 Process Improvement

20.8 Acknowledgment

Part Three The Appendices

Appendix A Terms of Reference for Testing Staff

A1 Introduction

Testing Manager

Responsibilities

Reporting and Liaison

Characteristics of a Testing Manager

References

Test Team Leader

Responsibilities

Reporting and Liaison

Characteristics of a Test Team Leader

References

Test Analyst

Responsibilities

Reporting and Liaison

Characteristics of a Test Analyst

Reference

Tester

Responsibilities

Reporting and Liaison

Characteristics of a Tester

Reference

Independent Test Observer

Responsibilities

Reporting and Liaison

Characteristics of an Independent Test Observer

References

Testing Programme Board

Responsibilities

Reporting and Liaison

Reference

Appendix B Testing Guides

B1 Introduction

Unit Testing Guide

Purpose of Unit Testing

Approach

Inputs

Testing Techniques (see Chapter 3)

Outputs

Quality Considerations

References

Integration Testing Guide

Purpose of Integration Testing

Approach

Inputs

Testing Techniques (see Chapter 3)

Outputs

Quality Considerations

References

System Testing Guide

Purpose of System Testing

Approach

Inputs

Testing Techniques (see Chapter 3)

Outputs

Quality Considerations

References

Systems Integration Testing Guide

Purpose of Systems Integration Testing

Approach

Inputs

Testing Techniques (see Chapter 3)

Outputs

Quality Considerations

References

UserOperations Acceptance Testing Guide

Purpose of Acceptance Testing

Approach

Inputs

Testing Techniques (see Chapter 3)

Outputs

Quality Considerations

References

Regression Testing Guide

Purpose of Regression Testing

Approach

Inputs

Testing Techniques (see Chapter 3)

Outputs

Quality Considerations

References

Appendix C Test Plan Document Template

C1 Introduction

Test Plan Document Front Sheet

Contents

1 Introduction

1.1 Background

1.2 Structure of the Document

1.3 References

2 Test Approach and Constraints

2.1 Introduction

2.2 Test Objectives

2.3 Test Constraints

3 Risks and Dependencies

3.1 Introduction

3.2 Risks

3.3 Dependencies

4 Test Assumptions and Exclusions

4.1 Introduction

4.2 Assumptions

4.3 Exclusions

5 Entry and Exit Criteria

5.1 Introduction

5.2 Entry Criteria

5.3 Exit Criteria

6 Testing Project Controls

6.1 Introduction

6.2 Roles and Responsibilities

6.3 Training Requirements

6.4 Problem Reporting

6.5 Progress Reporting

A Test Plan

Appendix D Test Specification Document Template

D1 Introduction

Test Specification Document Front Sheet

Contents

1 Introduction

1.1 Background

1.2 Scope

1.3 Structure of the Document

1.4 References

2 Test Requirement

2.1 Introduction

2.2 Test Philosophy

2.2.1 Overview

2.2.2 Functional Areas

2.2.3 Test Result Categories

2.2.4 Exclusions

2.3 Test Environment

2.3.1 Overview

2.3.2 Hardware

2.3.3 Software

2.3.4 Test Data

2.4 Staff Roles and Responsibilities

2.5 Test Identification

2.5.1 Test Scripts

2.5.2 Result Reporting

2.5.3 Acceptance Criteria

2.5.4 Test Error Clearance

2.5.5 Test Documentation

3 Test Procedure

3.1 Introduction

3.2 Pre-test Activities

3.2.1 Test Environment

3.2.2 Test Timescales

3.2.3 Test Liaison

3.2.4 Test Script Preparation

3.3 Conducting the Test

3.3.1 Test Execution Procedure

3.3.2 Filing of Completed Test Result Record Forms

3.4 Post-test Activities

Annex A Test Cases

Appendix E Test Script Template

E1 Introduction

Appendix F Test Result Record Form Template

F1 Introduction

Appendix G Test Log Template

G1 Introduction

Appendix H Test Certificate Template

H1 Introduction

Appendix I Reuse Pack Checklist

I1 Introduction

I2 Creating a Reuse Pack

I3 Contents of a Reuse Pack

I4 How to Use the Reuse Pack

Appendix J Test Summary Report Template

J1 Introduction

Test Summary Report Front Sheet

Contents

1 Introduction

1.1 Background

1.2 Structure of the Report

1.3 References

2 Overview

3 Variances

4 Assessment

5 Results

6 Evaluation

7 Summary of Activities

Appendix K Equivalence Partition Example

K1 Introduction

K2 The Testing Problem

K3 Analyzing the Testing Requirements

K4 Designing the Test Cases

Appendix L Boundary Value Analysis Example

L1 Introduction

L2 The Testing Problem

L3 Analyzing the Testing Requirements

L4 Designing the Test Cases

Appendix M State Transition Example

M1 Introduction

M2 The Testing Problem

M3 Analyzing the Testing Requirements

M4 Designing the Test Cases

Appendix N Pairwise Testing Example

N1 Introduction

N2 The Testing Problem

N3 The Pairwise Testing Solution

N3 Pairwise Testing in Practice

Appendix O Automated Testing Tool Selection Criteria

O1 Introduction

O2 Scoring Scheme

O3 Support for Testing Phases and Techniques

O4 Support for Testing Management

O5 Tool Technical Issues

O6 Tool Defect Analysis Facilities

O7 Tool Documentation Issues

O8 Tool Integration Issues

O9 Tool Usability Issues

O10 Supplier Issues

O11 Training and Consultancy Issues

O12 Contractual and Financial Issues

O13 Miscellaneous Issues

Appendix P Usability Testing Overview

P1 Introduction

P2 Roles and Responsibilities

P3 Usability Test Design

P4 Usability Testing

P5 Post-test Activities

Appendix Q Testing Process Health Check

Q1 Introduction

Q2 Terminology

Q3 Testing and the Software Development Process

Q4 Testing Process

Q5 Roles and Responsibilities

Q6 Testing Phases

Q7 Optional Evaluation Scheme

Appendix R The Testing of Object-Oriented Software

R1 Introduction

R2 Object-Oriented Development

R3 Impact of OO Development on Test Management & Planning

Iterative Development

Rapid Prototyping

Object Reuse

R4 Impact of OO on Test Design

Unit Testing

Integration Testing

Acceptance Testing

Appendix S Pragmatic Test Process Adoption – a Real-World Example

S1 Introduction

S2 Test Process Adoption – a Real-World Example

Introduction

The Testing Challenge

Making a Start

Test Planning

Test Scripts

Test Execution

Bugs, Defects, and Problems

Managing Stakeholder Expectations

Summary

References

Glossary

Introduction

Index