SOFTWARE TESTING
Content
SOFTWARE QUALITY ASSURANCE
3. TEST GENERATION 3.1 SOFTWARE TESTING Software testing is a process of executing a program or application with the intent of finding the software bugs. bugs. It can also be stated as the process of validating and verifying that a software program or application or product: Meets the business and technical requirements that guided it‘s design and development Works as expected cha racteristic. Can be implemented with the same characteristic.
The definition of Software testing is broken into the following parts: 1) Process: Testing is a process rather than a single activity. 2) All Life Cycle Activities: Testing is a process that‘s take place throughout the Software Development Life Cycle (SDLC).
The process of designing tests early in the life cycle can help to prevent defects from being introduced in the code. Sometimes it‘s referred as “verifying the test test basis via the test design”. The test basis includes documents such as the requirements and design specifications.
3) Static Testing: It can test and find defects without executing code. Static Testing is done during verification process. This testing includes reviewing of the documents (including source code) and static analysis. This This is useful and cost effective way of testing. For example: reviewing, walkthrough, inspection, etc.
software code is executed to demonstrate the result 4) Dynamic Testing: In dynamic testing the software of running tests. It‘s done during validation process. For example: unit testing, integration testing, system testing, etc. we want to do. We control the test activities, we report 5) Planning: We need to plan as what we on testing progress and the status of the software under un der test. 6) Preparation: We need to choose what testing we will do, by selecting test conditions and
designing test cases. cases. 1
unit III
SOFTWARE QUALITY ASSURANCE
must check the results and evaluate the software under 7) Evaluation: During evaluation we must test and the completion criteria, which helps us to decide whether we have finished testing and whether the software product has passed the tests. 8) Software products and and related work products: Along with the testing of code the testing of requirement and design specifications and also the related documents like operation, user and
training material is equally important. Software Testing has different goals and objectives. The major objectives of Software testing are as follows:
Finding defects which may get created by the programmer while developing the software. Gaining confidence in and providing information about the level of quality. To prevent defects. To make sure that the end result meets the business and user requirements. To ensure that it satisfies the BRS that is Business Requirement Specification and SRS that is System Requirement Specifications.
To gain the confidence of the customers by providing them a quality product. Software testing is very important because of the following reasons: 1. Software testing is really required to point out the defects and errors that were made during the development phases. 2. It‘s essential since it makes sure of the Customer‘s reliability and their satisfaction in the application. 3. It is very important to ensure the Quality of the product. Quality product delivered to the customers helps in gaining their confidence. 4. Testing is necessary in order to provide the facilities to the customers like the delivery of high quality product or software application which requires lower maintenance cost and hence results into more accurate, consistent and reliable results. 5. Testing is required for an effective performance of software application or product. 6. It‘s important to ensure that the application app lication should not result into any failures because it can be very expensive in the future or in the later stages of the development. 7. It‘s required to stay in the business. business.
3.2. VALIDATION Determining if the system complies with the requirements and performs functions for which it is intended and meets the organization‘s goals and user needs. needs. Validation is done at the end of the development process and takes place after
verifications are completed. It answers the question like: Am I building the right product?
2
unit III
SOFTWARE QUALITY ASSURANCE
Am I accessing the right data (in terms of the data d ata required to satisfy the requirement). It is a High level activity. Performed after a work product is produced against established criteria ensuring that the product integrates correctly into the environment. Determination of correctness of the final software product by a development project with respect to the user needs and requirements.
3.2.1. Software verification and validation
According to the Capability Maturity Model we can also define validation as ―The process of evaluating software during or at the end of the development process to determine whether it satisfies specified requirements‖ requirements‖. A product can pass while verification, as it is done on the paper and no running or functional application is required. But, when same points which were verified on the paper is actually developed then the running application or product can fail while validation. This may happen because when a product or application is build as per the specification but these specifications are not up to the mark hence they fail to address the user requirements. 3.2.2. Advantages of Validation:
1. During verification if some defects are missed then during validation process it can be caught as failures. 2. If during verification some specification is misunderstood and development had happened then during validation process while executing that functionality the difference between the actual result and expected result can be understood. 3. Validation is done during testing like feature testing, integration testing, system testing, load testing, compatibility testing, stress testing, etc. 4. Validation helps in building the right product as per the customer‘s requirement and helps in satisfying their needs.
Validation by theresult testers during the testing. While the product if some deviationisisbasically found in done the actual from the expected result thenvalidating a bug is reported or an 3
unit III
SOFTWARE QUALITY ASSURANCE
incident is raised. Not all incidents are bugs. But all bugs are incidents. Incidents can also be of type ‗Question‘ where the functionality is not clear to the tester. tester. Hence, validation helps in unfolding the exact functionality of the features and helps the testers to understand the product in much mu ch better way. It helps in making the product more user friendly. Validation testing provides answers to questions such as: Does the software fulfill its intended use? Is the company building the right product? Can the project be properly implemented? Are the documents in line with the development process? 3.2.3. Validation Testing Process:
The aim of software testing is to measure the quality of software in terms of number of defects found in it, the number of tests run, and the system covered by the tests. When bugs or defects are found with the help of testing, the bugs are logged and the development team fixes them. Once the bugs are fixed, testing is carried out again to ensure that they are indeed fixed, and no new defects have been introduced in the software. With the entire cycle, the quality of the software increases. The software validation process can be described as follows: Validation Planning To plan all the activities that needs to be included while testing. ⇓
Define Requirements
To set goals and define the requirements for testing ⇓
Select Appropriate Team
To select a skilled and knowledgeable development team (third party included) ⇓
Develop Documents
To develop a user specifications document describing the operating conditions ⇓
Evaluation
To evaluate the software as per the specifications and submit a validation report ⇓
Incorporating Changes 4
unit III
SOFTWARE QUALITY ASSURANCE
To change the software so as to remove any errors found during evaluation.
3.2.4. Techniques
There are various validation techniques like fault injection, dependability analysis, etc., that are commonly used for software testing. Here are a few more. 1. Formal Methods:
This technique makes use of mathematical and logical techniques to analyze the input specifications, the product design, the supporting documents, and the behavior of the product under test. 2. Fault Injection
As the name suggests, faults or bugs are intentionally added to software so as to check its working/functionality under the said conditions. These are of two types - hardware fault injection and software fault injection. In the former, faults in physical hardware are injected, while in the latter, errors are injected in the software or in the system's memory. 3. Dependability Analysis
This technique is used to increase the dependability of the product. The hazards in the software are identified, and methods are suggested to reduce theses potential hazards. 4. Hazard Analysis
Every software has its own standard tests to identify hazards. This type of analysis follows the said procedure to identify the hazard and find out its countermeasures. 5. Risk Analysis
This takes the hazard analysis further by identifying countermeasures for each type of hazard that is identified.
3.3. TEST PLANS Test plan is the project plan for the testing work to be done. It is not a test design p lans specification, a collection of test cases or a set of test procedures; in fact, most of our test plans do not address that level of detail. Many people have different definitions for test plans. There are three main reasons to write the test plans: guides our thinking. Writing a test plan forces us to First, by writing a test plan it guides confront the challenges that await us and focus our thinking on important topics. 5
unit III
SOFTWARE QUALITY ASSURANCE
Fred Brooks explains the importance of careful estimation and planning for testing in one of his book as follows: Failure to allow enough time for system test, in particular, is peculiarly disastrous. Since the delay comes at the end of the schedule, no one is aware of schedule trouble until almost the delivery date [and] delay at this point has unusually severe financial repercussions. The project is fully staffed, and cost-per day is maximum [as are the associated opportunity costs], It is therefore very important to allow enough system test time in the th e original schedule. Second, the test planning process and the plan itself serve as the means of communication with other members of the project team, testers, peers, managers and other stakeholders. This communication allows the test plan to influence the project team and the project team to influence the test plan, especially in the areas of organization-wide testing policies and motivations; test scope, objectives and critical areas to test; project and product risks, resource considerations and constraints; and the testability of the item under test. We can complete this communication by circulating one or two test plan drafts and through review meetings. Such a draft will include many man y notes such as the following: Third, the test plan helps us to manage change. During early phases of the project, as we gather more information, we revise our plans. As the project evolves and situations change, we adapt our plans. By updating the plan at major milestone helps us to keep testing aligned with project needs. need s. As we run the tests, we make mak e final f inal adjustments to our plans plan s based on the th e results. You might not have the time – or or the energy – energy – to to update your test plans every time a change is made in the project, as some projects can be quite dynamic.
At times it is better to write multiple test plans in some situations. For example, when we manage both integration and system test levels, those two test execution periods occur at different points in time and have different objectives. For some systems projects, a hardware test plan and a software test plan will address different techniques and tools as well as different audiences. However, there are chances that these test plans can get overlapped, hence, a master test plan should be made that addresses the common elements of both the test plans can reduce the amount of redundant documentation. 3.3.1. IEEE 829 STANDARD TEST PLAN TEMPLATES: 6
Test plan identifier. Test deliverables Introduction. Test tasks. Test items. Environmental needs. Features to be tested. Responsibilities. Features not to be tested. unit III
SOFTWARE QUALITY ASSURANCE
Staffing and training needs. Approach Schedule. Item pass/fail criteria. Risks and contingencies. Suspension and resumption criteria Approvals.
3.4. TEST CASE A test case, in in software software engineering, engineering, is
a set of conditions or variables under which a tester will determine whether an application, an application, software software systemor systemor one of its features is working as it was originally established for it to do. The mechanism for determining whether a software program or system has passed or failed such a test is known as a a test oracle. In some settings, an oracle could be a requirement a requirement or use use case, case, while in others it could be a heuristic. a heuristic. It may take many test cases to determine that a software program or system is considered sufficiently scrutinized to be released. Test cases are often referred to as as test scripts, particularly when written - when they are usually collected into test into test suites. suites. 3.4.1. Formal test cases
In order to fully test that all the requirements of an application are met, there must be at least two test cases for each requirement: one positive test and one negative test. If a requirement has sub-requirements, each sub-requirement must have at least two test cases. Keeping track of the link between the requirement and the test is frequently done using a traceability matrix. matrix. Written test cases should include a description of the functionality to be tested, and the preparation required to ensure that the test can be conducted. A formal written test-case is characterized by a known input and by an expected output, which is worked out before the test is executed. The known input should test a precondition and the expected output should test a post a post condition. condition. 3.4.2. Informal test cases
For applications or systems without formal requirements, test cases can be written based on the accepted normal operation of programs of a similar class. In some schools of testing, test cases are not written at all but the activities and results are reported after the tests have been run. In In scenario testing, testing, hypothetical stories are used to help the tester think through a complex problem or system. These scenarios are usually not written down in any detail. They can be as simple as a diagram for a testing environment or they could be a description written in prose. The ideal scenario test is a story that is motivating, credible, complex, and easy to evaluate. They are usually different from test cases in that test cases are single steps while scenarios cover a number of steps of the key. In its most basic form, a test case is a simple pair of <input, expected outcome>. If a program under test is expected to compute the square root of nonnegative numbers, then four examples of test cases are as shown in Figure 2. In stateless systems, where the outcome depends solely s olely on the current input, test cases are very simple in structure, as shown in Figure 2. 7
unit III
SOFTWARE QUALITY ASSURANCE
A program to compute the square root of nonnegative numbers is an example of a stateless system. A compiler for the C programming language is another example of a stateless system. A compiler is a stateless system because to compile a program it does not need to know about the programs it compiled previously. TB1: <0, 0>, TB2: <25, 5>, TB3: <40, 6.3245553>, TB4: <100.5, 10.024968>.
Figure 2: Examples of basic test cases. TS1: <check balance, $500.00>, <withdraw, ‘‘amount?’’>, <$200.00, ‘‘$200.00’’>, <check balance, $300.00>.
Figure 3: Example of a test case with a sequence of <input, expected outcome>. In state-oriented systems, where the program outcome depends both on the current state of the system and the current input, a test case may consist of a sequence of <input, expected outcome> pairs. A telephone switching system and an automated teller machine (ATM) are examples of state-oriented systems. For an ATM machine, a test case for testing the withdraw function is shown in Figure 3 Here, we assume that the user has already entered validated inputs, such as the cash card and the personal identification number (PIN). In the test case TS1, ―check balance‖ and ―withdraw‖ in the first, second, and fourth tuples represent the pressing of the appropriate keys on the ATM keypad. It is assumed that the user account has $500.00 on it, and the user wants to withdraw an amount of $200.00. The expected outcome ―$200.00‖ in the third tuple represents the cash dispensed by the ATM. After the withdrawal operation, the user makes sure that the remaining balance is $300.00. For state-oriented systems, most of the test cases include some form of decision and timing in providing input to the system. A test case may include loops and timers, which we do not show at this moment.
3.5. TEST GENERATION Any form of test generation uses a source document. In the most informal of test methods, the source document resides in the mind of the tester who generates tests based on the knowledge of the requirements. In several commercial environments, the process is a bit more formal. The tests are generated using a mix of formal and informal methods either directly from the requirements document serving as the source. In more advanced test processes, requirements serve as a source for the development d evelopment of formal models.
8
unit III
SOFTWARE QUALITY ASSURANCE
3.5.1. Test generation strategies Model based: require that a subset of the requirements be modeled using a formal notation (usually graphical). Models: Finite State Machines, Timed automata, Petri net, etc. Specification based: require that a subset of the requirements be modeled using a formal mathematical notation. Examples: B, Z, and Larch. Code based: generate tests directly from the code.
3.5.2. Test generation techniques
Four techniques are considered:
Equivalence partitioning Boundary value analysis Cause effect graphing and Predicate based test generation. Each of these test generation techniques is a black-box technique and useful for generating test cases during functional testing.
9
unit III
SOFTWARE QUALITY ASSURANCE
3.5.3. The test selection problem Requirements and test generation
Requirements serve as the starting point for the generation of tests. During the initial phases of development, requirements requ irements may exist only onl y in the minds of one or more people. These requirements, more aptly ideas, are then specified rigorously using modeling elements such as use cases, sequence diagrams, and state charts in UML. Rigorously specified requirements are often transformed into formal requirements using requirements specification languages such as Z, S, and RSML.
3.6. EQUIVALENCE CLASS PARTITIONING An input domain may be too large for all its elements to be used as test input. However, the input domain can be partitioned into a finite number of sub domains for selecting test inputs. Each sub domain is known as an equivalence class (EC), and it serves as a source of at least one test input. The objective of equivalence partitioning is to divide the input domain of the system under test into classes, or groups, of inputs. All the inputs in the same class have a similar effect on the system under test [14, 15]. An EC is a set of inputs that the system treats identically when 10
unit III
SOFTWARE QUALITY ASSURANCE
the system is tested. It represents certain conditions, or predicates, on the input domain. An input condition on the input domain is a predicate over the values of the input domain. A valid input to a system is an element of the input domain that is expected to return a non error value. An invalid input is an input that is expected to return an error value. Input conditions are used to partition the input domain into ECs for the purpose of selecting inputs. Guidelines for EC Partitioning Equivalence classes can be derived from an input domain by a heuristic technique. One can approximate app roximate the ECs by identifying classes for which different program behaviors are specified. Identification of ECs becomes easier with experience. Myers suggests the following guidelines to identify ECs [16]. 1. An input condition specifies a range [a, b]: Identify one EC for a ≤ X ≤ b and two other classes for X <a and X>b to test the system with invalid inputs. 2. An input condition specifies a set of values: Create one EC for each element of the set and one EC for an invalid member. For example, if the input is selected from a set of N items, then N + 1 ECs are created: (i) One EC for each element of the set {M1}, {M2}, . . . , {MN} (ii) One EC for elements outside the set {M1,M2, . . .,MN}. 3. Input condition specifies for each individual value: If the system handles each valid input differently, then create one EC for each valid input. For example, if the input is from a menu, then create one EC for each menu item.
4. An input condition specifies the number of valid values (say N): Create one EC for the correct number of inputs and two ECs for invalid inputs — inputs — one one for zero values and one for more than N values. For example, if a program can accept 100 natural numbers for sorting, then three ECs are created: 11
unit III
SOFTWARE QUALITY ASSURANCE
(i) One for 100 valid input of natural numbers. (ii) One for no input value. (iii) One for more than 100 natural numbers. 5. An input condition specifies a ―must―must - be‖ be‖ value: Create one EC for a must-be must -be value and one EC for something that is not a must-be value. For example, if the first character of a password must be a numeric character, then we are required to generate two ECs: (i) (ii)
one for valid values, {pswd | the first character ofpswd has a numeric value}, (ii) one for invalid values, {pswd | the first character of pswd is not numeric}.
6. Splitting of EC: If elements in a partitioned EC are handled differently by the system, then split the EC into smaller ECs. Identification of Test Cases from ECs Having identified the ECs of an input domain of a program, test cases for each EC can be identified by the following: Step 1: Assign a unique number to each EC. Step 2: For each EC with valid input that has not been covered by test cases yet, write a new test case covering as many uncovered ECs as possible. Step 3: For each EC with invalid input that has not been covered by test cases, write a new test case that covers one and only one of the uncovered ECs.
In summary, the advantages of EC partitioning are as follows: • A small number of test cases are needed to adequately cover a large input domain. domain. • One gets a better idea about the the input domain being covered with theselected test cases. • The probability of uncovering defects with the selected test cases based on EC partitioning is higher than that with a randomly chosen test suite of the same size. • The EC partitioning approach approach is not restricted to input conditions alone;the technique may also be used for output domains. Example: Adjusted Gross Income. Consider a software system that computes
income tax based on adjusted gross income (AGI) according to the following rules: If AGI is between $1 and $29,500, the tax due is 22% of AGI. If AGI is between $29,501 and $58,500, the tax due is 27% of AGI. If AGI is between $58,501 and $100 billion, the tax due is 36% of AGI.
12
unit III
SOFTWARE QUALITY ASSURANCE
In this case, the input domain is from $1 to $100 billion. There are three input conditions in the example: 1. $1 ≤ AGI ≤ $29,500. $29,500. 2. $29,501 ≤ AGI ≤ $58,500. $58,500. 3. $58,501 ≤ AGI ≤ $100 billion. billion. First we consider condition 1, namely, $1 ≤ AGI ≤ $29,500, to derive two ECs: ECs: EC1: $1 ≤ AGI ≤ $29,500; valid input. input. EC2: AGI < 1; invalid input. Then, we consider condition 2, namely, $29,501 ≤ AGI ≤ $58,500, to derive one EC: EC: EC3: $29,501 ≤ AGI ≤ $58,500; valid input. input. Finally, we consider condition 3, namely, n amely, $58,501 ≤ AGI ≤ $100 billion, to derive two ECs: ECs: EC4: $58,501 ≤ AGI ≤ $100 ≤ $100 billion; valid input. EC5: AGI > $100 billion; invalid input. Note that each condition co ndition was considered separately in the derivation of ECs. E Cs. Conditions Cond itions are not combined to select ECs. Five test cases are generated to cover the five ECs, as shown in Table 9.10. In the EC partition technique, a single test input is arbitrarily selected to cover a specific EC. We need to generate specific test input by considering the extremes, either inside or outside of the defined EC partitions. This leads us to the next technique, known as boundary value analysis, which focuses on the boundary of the ECs to identify test inputs.
3.7. BOUNDARY VALUE ANALYSIS Boundary value analysis is a test selection technique that targets faults in applications at the boundaries of equivalence classes. While equivalence partitioning selects tests from within 13
unit III
SOFTWARE QUALITY ASSURANCE
equivalence classes, boundary value analysis focuses on tests at and near the boundaries of equivalence classes. Certainly, tests derived using either of the two techniques techn iques may overlap. The central idea in boundary value analysis (BVA) is to select test data near the boundary of a data domain so that data both within and outside an EC are selected. It produces test inputs near the boundaries to find failures caused by incorrect implementation of the boundaries. Boundary conditions are predicates that apply directly on and around the boundaries of input ECs and output ECs. In practice, designers and programmers tend to overlook boundary conditions. Consequently, defects tend to be concentrated near the boundaries between ECs. Therefore, test data are selected on or near a boundary. In that sense, the BVA technique is an extension and refinement of the EC partitioning technique. In the BVA technique, the boundary conditions for each EC are analyzed in order to generate test cases. 3.7.1. BVA: Procedure Procedure
1. Partition the input domain using uni-dimensional partitioning. This leads to as many partitions as there are input variables. Alternately, a single partition of an input domain can be created using multidimensional partitioning. We will generate several sub-domains in this step. 2. Identify the boundaries for each partition. Boundaries may also be identified using special relationships amongst the inputs. 3. Select test data such that each each boundary value occurs in at least one ttest est input. input. 3.7.2. Guidelines for BVA:
As in the case of EC partitioning, the ability to develop high-quality effective test cases using BVA requires experience. The guidelines discussed below are applicable to both input conditions and output conditions. The conditions are useful in identifying high-quality test cases. By high-quality test cases we mean test cases that can reveal defects in a program. 1. The EC specifies a range:
If an EC specifies a range of values, then construct test cases by considering the boundary bo undary points of the range and points just beyond the boundaries of the range. For example, let an EC specify the range of −10.0 ≤ X ≤ 10.0. This would result in test data {− {−9 9.9 − 10.0 ,−10.1} and {9.9 , 10.0 , 10.1}. 2. The EC specifies a number of values:
If an EC specifies a number of values, then construct test cases for the minimum and the maximum value of the number. In addition, select a value smaller than the minimum and a value 14
unit III
SOFTWARE QUALITY ASSURANCE
larger than the maximum value. For example, let the EC specification of a student dormitory specify that a housing unit can be shared by one to four students; test cases that include 1, 4, 0, and 5 students would be developed. 3. The EC specifies an ordered set:
If the EC specifies an ordered set, such as a linear list, table, or sequential file, then focus attention on the first and last elements of the set. Example:
Let us consider the five ECs identified in our previous example to compute income tax based on AGI. The BVA technique results in test as follows for each EC. The redundant data points may be eliminated. EC1: $1 ≤ AGI ≤ $29,500; this would result in values of $1, $0, $ – 1, 1, $1.50 and $29,499.50, $29,500, $29,500.50. EC2: AGI <1; This would result in values of $1, $0, $ – 1, 1, $ – 100 100 billion. EC3: $29,501 ≤ AGI ≤ $58,500; this would result in values of $29,500, $29,500.50, $29,501, $58,499, $58,500, $58,500.50, $58,501. EC4: $58,501 ≤ AGI ≤ $100 billion; this would result in values of $58,500, $58,500.50, $58,501, $100 billion, $101 billion. EC5: AGI >$100 billion; this would result in $100 billion, $101 billion, $10000 billion. Remark:
Should we test for an AGI value of $29,500.50 (i.e., between the partitions), and if so, what should be the result? Since we have not been told whether the decimal values are actually possible, the best decision to make is to test for this value and report the result.
3.8. CATEGORY PARTITION METHOD The category partition method (CPM) is a generalization and formalization of a classical functional testing approach. The reader is reminded of the two steps of the classical functional testing approach: (i) Partition the input domain of the functional unit to be tested into equivalence classes. (ii) Select test data from each EC of the partition. p artition. The CPM is a systematic, specification-based methodology that uses an informal functional specification to produce formal test specification. The test designer‘s key job is to develop categories, which are defined to be the major characteristics of the input domain of the function under test. Each category is partitioned into ECs of inputs called choices. The choices in 15
unit III
SOFTWARE QUALITY ASSURANCE
each category must be disjoint, and together the choices in each category must cover the input domain. Grochtmann and Grimm extend this approach by capturing the constraints using a tree structure to reduce the number of impossible test cases. The main advantage of this approach is the creation of a formal test specification written in languages such as test specification language (TSL), Z or testing and test control notation (TTCN) to represent an informal or natural language formal specification. The formal test specification gives the test engineers a logical way to control test generation and is easily modified to accommodate changes or mistakes in the functional specification. The use of Zallows more flexibility in the specification of constraints and more formalities in the representation The category partition testing method is comprised of the following steps: Step 1: Analyze the Specification
The method begins with the decomposition of a functional specification into functional units that can be separately tested. For each e ach functional unit, identify the following: • Parameters of the functional unit • Characteristics of each parameter, that is, the elementary characteristics of the parameters that affect the execution of the unit • Objects in the environment whose state might affect the operation op eration of the functional unit • Characteristics of each environment object Parameters are the explicit input to a functional unit, and environment conditions are the state characteristics of the system at the time of execution of a functional specification. Step 2: Identify Categories
A category is a classification of the major properties(or characteristics) of a parameter or an environmental condition. Consider, for instance, a program P SORT SORT that reads an input file F containing a variable-length array of values of arbitrary type. The expected output is a permutation of input with values sorted according to some total ordering criterion. The environmental condition for P SORT SORT is status of F , which can be classified into three categories, namely Status of F = Does Not Exist, Status of F = Exists But Empty, and Status of F = Exists and Nonempty. The properties of the input parameter categories for the P SORT SORT are as follows: the array size, the type of elements, the minimum element value, the maximum element value, and the positions in the array of the maximum and minimum values. Categories can be derived directly from the functional specification, implicit design in formation, or intuition of the test engineer. Often categories are derived from preconditions and type information about the input parameters and system state components. Step 3: Partition Categories into Choices Partition each category into distinct choices that include all the different kinds of values that are possible for the category. Each choice is an equivalent class of values assumed to have identical properties as far as testing and error detection capability are concerned. The choices 16
unit III
SOFTWARE QUALITY ASSURANCE
must be disjoint and cover all the categories. While the categories are derived from a functional specification, the choices can be based base d on specification and the test engineer‘s past experience of designing effective test cases, such as error guessing. In the sorting program P SORT SORT example, one possible way to partition the category array size is size = 0, size = 1, 2 ⇐ size ⇐ 100, and size >100. These choices are based primarily on experience with likely errors. The selection of a single choice from each category determines a test frame, which is the basis for constructing the actual test cases. A test frame consists of a set of choices from the specification, with each category contributing either zero or one choice. Since the choices in different categories frequently interact with each other in ways that affect the resulting test cases, the choices can be annotated with constraints (see next step) to indicate these relations. In the absence of constraints the number of potential test frames is the product of the number of choices in each category — and and this is likely to be very large. Step 4: Determine Constraints among Choices Constraints are restrictions among the choices within different categories that can interact with one another. A typical constraint specifies that a choice from one category cannot occur together in a test frame with certain choices from another category. Choices and constraints are
derived from the natural language functional specifications but can often be specified by formal methods, thus making their analysis easier to automate. With a careful specification of constraints, the number of potential test frames can be reduced to a manageable number. Step 5: Formalize and Evaluate Test Specification Specify the categories, choices, and constraints using a specification technique that is compatible with the test generation tool, such as TSL, Z, or TTCN, that produces test frames. Most test generation tools also provide automated techniques for evaluating the internal consistency of the formal specification. This evaluation often discovers errors or inconsistencies in the specification of constraints and sometimes leads to discovery of errors in the source functional specification. Step 6: Generate and Validate Test Cases The final step in the test production process is to transform the generated test frames into executable test cases. If the test specification includes post conditions that must be satisfied, then the tool verifies the post conditions. In case a reference Implementation is available, the test cases can be validated by executing them and checking their results against the reference implementation. The validation of test cases is a labor-intensive process in the absence of a reference implementation.
3.9. COMBINATORIAL TESTING 3.9.1. Combinatorial testing: Basic idea
• Identify distinct attributes that can be varied in the data, environment, or configuration 17
unit III
SOFTWARE QUALITY ASSURANCE
– Example: browser could be ―IE‖ or ―Firefox‖, operating system could be ―Vista‖, ―XP‖, or ―OSX‖ ―OSX‖ • Systematically generate combinations to be tested tested – Example: Example: IE on Vista, IE on XP, Firefox on Vista, Firefox on OSX. • Rationale: Test cases should be varied v aried and include possible ―corner cases‖ cases‖ 3.9.2. Key ideas in combinatorial approaches
• Category-partition testing Separate (manual) identification of values that characterize the input space from (automatic) generation of combinations for test cases • Pair wise testing Systematically test interactions among attributes of the program input space with a relatively small number of test cases • Catalog-based testing Aggregate and synthesize the experience of test designers in a particular organization or application domain, to aid in identifying iden tifying attribute values Category partition (manual steps)
1. Decompose the specification into independently testable features. For each feature identify • Parameters Parameters • Environment elements elements For each parameter and environment env ironment element identify elementary characteristics (categories) 2. Identify relevant values for each characteristic (category) identify (classes of) values • Normal values values • Boundary values values • Special values values • Error values values 3. Introduce constraints 18
unit III
SOFTWARE QUALITY ASSURANCE
An informal specification: check configuration
Check Configuration • Check the validity of a computer configuration configuration • The parameters of check -configuration -configuration are: – Model Model – Set Set of components An informal specification: Parameter model:
A model identifies a specific product and determines a set of constraints on available components. Models are characterized by logical slots for components, which may or may not be implemented by physical slots on a bus. Slots may be required or optional. Required slots must be assigned with a suitable component to obtain a legal configuration, while wh ile optional slots may be left empty or filled depending on the customers' needs. An informal specification of parameter set of components:
A set of (slot, component) pairs, corresponding to the required andoptional slots of the model. A component is a choice that can be varied within a model, and which is not designed to be replaced by the end user. Available components compon ents and a default for each slot is determined determ ined by the model. The special value empty is allowed (and may be the default selection) for optional slots. In addition to being compatible or incompatible with a particular model and slot, individual components may be compatible or incompatible with each other. Step1: Identify independently testable units and categories
• Choosing categories categories – No No hard-and-fast rules for choosing categories – Not Not a trivial task! • Categories reflect test designer's judgment judgment – Regarding Regarding which classes of values may be b e treated differently by an implementation • Choosing categories well requires experience and knowledge knowledge – Of Of the application domain and product architecture. 19
unit III
SOFTWARE QUALITY ASSURANCE
The test designer must look under the surface of the specification and identify hidden characteristics Identify independently testable units
Parameter Model – Model Model number – Number Number of required slots for selected model (#SMRS) – Number Number of optional slots for selected model (#SMOS) Parameter Components – Correspondence Correspondence of selection with model slots – – Required Required component selection – – Optional Optional component selection Environment element: Product database – Number Number of models in database (#DBM) – Number Number of components in database (#DBC) Step 2: Identify relevant values
• Identify (list) representative classes of values for each of the categories categories – Ignore Ignore interactions among values for different categories (considered in the next step) • Representative values values may be identified by applying Boundary value testing • select extreme values within a class class • select values outside but as close as possible to the class • select interior (non-extreme) (non-extreme) values of the class - Erroneous condition testing • select values outside the normal domain of the program Identify relevant values of Model
Model number, malformed, not in database, valid v alid values are identified. 20
unit III
SOFTWARE QUALITY ASSURANCE
Number of required slots for selected model (#SMRS)
0 1 Many Number of optional slots for selected model (#SMOS)
0 1 Many Identify relevant values: Component
Correspondence of selection with model slots Omitted slots Extra slots Mismatched slots Complete correspondence Number of required components with non empty selection
0 Less than number required slots Equal to number required slots Required component selection Some defaults All valid
Number of optional components with non empty selection 21
unit III
SOFTWARE QUALITY ASSURANCE
0 < Less than #SMOS Equal to #SMOS Optional component selection Some defaults All valid
selection
Identify relevant values: Database
Number of models in database (#DBM)
0 1 Many Number of components in database (#DBC)
0 1 Many Note 0 and 1 are unusual (special) values. They might cause unanticipated behavior alone or in
combination with particular values of other parameters Step 3: Introduce constraints
• A combination of values for each category corresponds to a test case specification • Example: Example: zero slots and at least one incompatible slot
22
unit III
SOFTWARE QUALITY ASSURANCE
• Introduce constraints to rule out impossible combinations and combinations and to reduce the size of the test suite if too large. Error constraint
[error] indicates a value class that – corresponds corresponds to a erroneous values – need need be tried only once Example: Model number: Malformed and Not in database Err or value value classes
– No No need to test all possible combinations of errors – One test is enough (we assume that handling an error case bypasses other program logic) Property constraints
Constraint [property] [if-property] rule out invalid combinations of values [property] groups values of a single parameter to identify subsets of values with common properties [if-property] bounds the choices of values v alues for a category that can be combined with a particular p articular value selected for a different category Single constraints
[Single] indicates a value class that test designers choose to test only once to reduce the number of test cases Example value some default for required component selection and optional component selection may be tested only once despite not being an erroneous condition Example - Single constraints
Number of required slots for selected model (#SMRS) 0 [single]
1 [property RSNE] [single]
Number of optional slots for selected model (#SMOS) 0 [single]
23
unit III
SOFTWARE QUALITY ASSURANCE
1 [single] [property OSNE]
Number of models in database (#DBM) 1 [single]
Number of components in database (#DBC) 1 [single]
• Category partition Category partition testing gave us – Systematic approach: Identify characteristics and values (the creative step), generate combinations (the mechanical step) but Test suite size grows very rapidly with number of categories • Pairwise (and n-way) n-way) combinatorial testing do – Combine Combine values systematically but not exhaustively – Rationale: Most unplanned interactions are among just two or a few parameters or parameter characteristics Pairwise combinatorial testing
• Category partition works well when intuitive constraints reduce the number of combinations to a small amount of test cases without many constraints, the number of combinations may be unmanageable. • Pairwise combination (instead of exhaustive) Generate exhaustive) Generate combinations that efficiently cover all pairs (triples) of classes classes – Rationale: most failures are triggered by single values or combinations of a few values. Covering pairs (triples,…) reduces the number of test cases, but reveals most faults faults Category-partition approach gives us, – Separation between (manual) identification of parameter characteristics and values and (automatic) generation of test cases that combine them – Constraints Constraints to reduce the number of combinations
24
unit III
SOFTWARE QUALITY ASSURANCE
3.10. DECISION TABLES A major limitation of the EC-based testing is that it only considers each input separately. The technique does not consider combining conditions. Different combinations of equivalent classes can be tried by using a new technique based on the decision table to handle multiple inputs. Decision tables have been used for many years as a useful tool to model software requirements and design decisions. It is a simple, yet powerful notation to describe complex systems from library information management systems to embedded real-time systems. The general structure of a decision table comprises a set of conditions (or causes) and a set of effects (or results) arranged in the form of a column on the left of the table. In the second column, next to each condition, we have its possible values: yes (Y), no (N), and don‘t care (dash). To the right of the values column, we have a set of rules. For each combination of the three conditions {C1, C2, C3}, there exists a rule from the set {R1, R2… R2 … R8}. Each rule comprises a yes, no, or don‘t care response and contains an associated li st of effects {E1, E2, E3}. Then, for each relevant effect, an effect sequence number specifies the order in which the effect should be carried out if the associated sets of conditions are satisfied. For example, if C1 and C2 are true but C3 is not true, then E3 should be followed by E1. The checksum is used for verification of the combinations the decision table represents.
Test data are selected so that each rule in a table is exercised and the actual results are verified with the expected results. In other words, each rule of a decision table represents a test case. The steps in developing test cases cas es using the decision table technique are as follows: Step 1:
The test designer needs to identify the conditions and the effects for each specification unit. A condition is a distinct input condition or an EC of input conditions. An effect is an output condition. Determine the logical relationship between the conditions and the effects. 25
unit III
SOFTWARE QUALITY ASSURANCE
Step 2:
List all the conditions and effects in the form of a decision table. Write down the values the condition can take. Step 3:
Calculate the number of possible combinations. It is equal to the number of different values raised to the power of the number of conditions. Step 4:
Fill the columns with all possible combinations — each each column corresponds to one combination of values. For each row (condition) do the following: 1. Determine the repeating factor (RF): divide the remaining number of combinations by the number of possible values for that condition. 2. Write RF times the first value, then RF times the next, and so forth until the row is full. Step 5:
Reduce combinations (rules). Find indifferent combinations — place a dash and join column where columns are identical. While doing this, ensure that effects are the same. Step 6: Check covered combinations (rules). For each column calculate the combinations it represents. A dash represents as many combinations as the condition has. Multiply for each dash down the column. Add up total and compare with step 3. It should be the same. Step 7:
Add effects to the column of the decision table. Read column by column and determine the effects. If more than one effect can occur in a single combination, then assign a sequence number to the effects, there by specifying the order in which the effects should be performed. Check the consistency of the decision table. Step 8:
The columns in the decision table are transformed into test cases. Decision table – based testing is effective under certain conditions as follows: • The requirements are easily mapped to a decision table. table. • The resulting decision table should not be too large. One can break down a large decision table into multiple smaller tables. • Each column in a decision decision table is independent of the other columns. A decision table is a good way to deal with combinations of things (e.g. inputs). This technique is sometimes referred to as a ‘cause‘cause -effect‘ table. The reason for this is that there is an associated logic diagramming technique called ‘cause‘cause -effect graphing‘ which was sometimes used to help derive the decision table. 26
unit III
SOFTWARE QUALITY ASSURANCE
Decision tables provide a systematic way of stating complex business rules, which is useful for developers as well as for testers.
Decision tables can be used in test design whether or not they are used in specifications, as they help testers explore the effects of combinations of different inputs and other software states that must correctly implement business rules.
It helps the developers to do a better job can also lead to better relationships with them. Testing combinations can be a challenge, as the number of combinations can often be huge. Testing all combinations may be impractical if not impossible. We have to be satisfied with testing just a small subset of combinations but making the choice of which combinations to test and which to leave out is also important. If you do not have a systematic way of selecting combinations, an arbitrary subset will be used and this may well result in an ineffective test effort.
How to Use decision tables for test designing?
The first task is to identify a suitable function or subsystem which reacts according to a combination of inputs or events. The system should not contain too many inputs otherwise the number of combinations will become unmanageable. It is better to deal with large numbers of conditions by dividing them into subsets and dealing with the subsets one at a time. Once you have identified the aspects that need to be combined, then you put them into a table listing all the combinations of True and False for each of the aspects. Let us consider an example of a loan application, where you can enter the amount of the monthly repayment or the number of years you want to take to pay it back (the term of the loan). If you enter both, the system will make a compromise between the two if they conflict. The two conditions are the loan amount and the term, so we put them in a table (see Table 4.2). TABLE 3.2 Empty decision table: Conditions
Rule 1
Rule 2
Rule 3
Rule 4
Repayment amount has been entered: Term of loan has been Entered:
Next we will identify all of the combinations of True and False (see Table 4.3). With two conditions, each of which can be True or False, we will have four combinations (two to the power of the number of things to be combined). Note that if we have three t hings to combine, we will have eight combinations, with four things, there are 16, etc. This is why it is good to tackle small sets of combinations at a time. In order to keep track of which combinations we have, we will alternate True and False on the bottom row, put two Trues and then two Falses on the row 27
unit III
SOFTWARE QUALITY ASSURANCE
above the bottom row, etc., so the top row will have all Trues and then all Falses (and this principle applies to all such tables).
TABLE 3.3 Decision table with input combinations: Conditions
Rule 1
Rule 2
Rule 3
Rule 4
Repayment amount has been entered:
T
T
F
F
T
F
T
Term of loan has been entered:
F
In the next step we will now identify the correct outcome for each combination (see Table 4.4). In this example, we can enter one or both of the two fields. Each combination is sometimes referred to as a rule.
TABLE 3.4 Decision table with combinations and outcomes: Conditions
Rule 1
Rule 2
Rule 3
Rule 4
Repayment amount has been entered:
T
T
F
F
Term of loan has been entered:
T
F
T
F
Actions/Outcomes Process loan amount:
Y
Process term:
Y
Y Y
At this point, we may realize that we hadn‘t thought about what happens if the customer doesn‘t enter anything in either of the two fields. The table has highlighted a combination that was not mentioned in the specification for this example. We could assume that this combination should result in an error message, so we need to add another action (see Table 4.5). This highlights the strength of this technique to discover omissions and ambiguities in specifications. It is not unusual for some combinations to be omitted from specifications; therefore this is also a valuable technique to use when reviewing the test basis. 28
unit III
SOFTWARE QUALITY ASSURANCE
TABLE 3.5 Decision table with additional outcomes: Conditions
Rule 1
Rule 2
Rule 3
Rule 4
Repayment amount has been entered:
T
T
F
F
Term of loan has been entered:
T
F
T
F
Actions/Outcomes Process loan amount:
Y
Process term:
Y
Y Y Y
Error message:
Now, we make slight change in this example, so that the customer is not allowed to enter both repayment and an d term. Now the outcome of our ou r table will change, chan ge, because b ecause there should also be an error message if both are entered, so it will look like Table 4.6.
TABLE 3.6 Decision table with changed outcomes: Conditions
Repayment amount has been entered: Term of loan has been entered:
Rule 1
Rule 3
Rule 4
T
T
F
F
T
F
T
F
Actions/Outcomes Process loan amount:
Y Y
Process term: Error message:
Rule 2
Y
Y
You might notice now that there is only one ‗Yes‘ in each column, i.e. our actions are mutually exclusive – only one action occurs for each combination of conditions. We could 29
unit III
SOFTWARE QUALITY ASSURANCE
represent this in a different way by listing the actions in the cell of one row, as shown in Table 4.7. Note that if more than one action results from any of the combinations, then it would be better to show them as separate rows rather than combining them into one row.
TABLE 3.7 Decision table with outcomes in one row: Conditions
Rule 1
Rule 2
Rule 3
Rule 4
Repayment amount has been entered:
T
T
F
F
Term of loan has been entered:
T
F
T
F
Actions/Outcomes:
Result: Error Process loan Process Error message amount term message. The final step of this technique is to write test cases to exercise each e ach of the four rules in our o ur table. Credit card example:
Let‘s take another example. If you are a new customer and you want to open a credit card account then there are three conditions con ditions first you will get a 15% discount on all your purchases today, second if you are an existing customer and you hold a loyalty card, you get a 10% discount and third if you have a coupon, you can get 20% off today (but it can‘t be used with the ‗new customer‘ discount). Discount amounts are added, if ap plicable. ap plicable. This is shown in Table 4.8. TABLE 3.8 Decision table for credit card example
In Table 4.8, the conditions and actions are listed in the left hand column. All the other columns in the decision table each represent a separate rule, one for each combination of conditions. We may choose to test each rule/combination and if there are only a few this will usually be the case. However, if the number of rules/combinations is large we are more likely to sample them by selecting a rich subset for testing. Now let‘s see the decision table for credit card shown above: above:
Note that we have put X for the discount for two of the columns (Rules 1 and 2) – this this means that this combination should not occur. You cannot be both a new customer and
30
unit III
SOFTWARE QUALITY ASSURANCE
also holding a loyalty card as per the conditions mentioned above. Hence there should be an error message stating this.
We have made an assumption in Rule 3. Since the coupon has a greater discount than the new customer discount, we assume that the customer will choose 20% rather than 15%. We cannot add them, since the coupon cannot be used with the ‗new customer‘ discount as stated in the condition above. The 20% action is an assumption on our part, and we should check that this assumption (and any other assumptions that we make) is correct, by asking the person who wrote the specification or the users.
For Rule 5, however, we can add the discounts; since both the coupon and the loyalty card discount should apply (that‘s our assumption). assumption).
Rules 4, 6 and 7 have only one type of discount and Rule 8 has no discount, so 0%.
If we are applying this technique thoroughly, we would have one test for each column or rule of our decision table. The advantage of doing this is that we may test a combination of things that otherwise we might not have tested and that could find a defect. However, if we have a lot of combinations, it may not be possible or sensible to test every combination. If we are timeconstrained, we may not have time to test all combinations. Don‘t just assume that all combinations need to be tested. It is always better to prioritize and test the most important combinations. Having the full table helps us to decide which combinations we should test and which not to test this time. In the example above all the conditions are binary, i.e. they have only two possible values: True or False (or we can say Yes or No). ?
31
unit III
