Projects in Software Testing

Published on December 2016 | Categories: Documents | Downloads: 36 | Comments: 0 | Views: 320
of 8
Download PDF   Embed   Report

Projects in software testing documents

Comments

Content








Projects in Software Testing

Software Testing, ETS 200
Programvarutestning, 5p.
Version 1.0, 2008, Lp Vt2
Per Runeson, Yeni Li Helgesson, CS, LTH
http://www.cs.lth.se/ETS200/


1. Introduction

This document gives the practical details regarding the project in the course Software Testing.
The project in the course is equivalent to 2 weeks of full time studies (2 credit points). The main
objective of the project is to gain a deeper understanding of a specific area within verification and
validation (V&V) of software products. A project group consists of 4-5 persons, who perform the
project together. All project members should be involved and the total effort should be evenly
distributed among participants.

2. Learning Objectives

The objective of the project is to learn a specific area of software testing. Furthermore, the
structure of the project makes it possible to learn important parts that will be practiced later on in
the education. Hence, the main learning points of the project are:

• Learn a specific area of software testing
• Collect and summarize research information
• Critical thinking beyond the written information
• Present information in a structured way

There are several types of projects to choose among. Examples of project types are:

• Research: solve a research problem; survey the state-of-the-art and identify the research
problems in some area; develop and justify an extension to an existing technique; etc.
• Evaluation: apply and evaluate a technique or evaluate a commercial testing or, analysis
tool.
• Practical: Use an existing technique to test a system or design and implement a
prototype for a system.









3. Activities

The main activities in the project are:
1. Decide on a subject. There are some subjects specified in this document. Other
subjects may be chosen, but has to be decided together with your project supervisor.
2. Find literature. In each subject area, there is a suggestion of research reports to start
reading. Other research literature can be found on http://elin.lub.lu.se (within the
domain of LU, or by using your STIL password). For research projects, 7-10 reports
should be chosen. For evaluation and practical projects, fewer reports are needed.
3. Outline the report. Specify heading one and two and write in short sentences what
will be included in the sections.
4. Read; perform evaluation and practical work. Read and understand the new area
of software testing; perform evaluation and practical work if these kind of projects
have been chosen.
5. Write the report. The report is preferable written in English. The report shall be
written in the IEEE template, see the homepage of the course.
6. Present the report. Communicate the area to other people who have knowledge of
software testing, but not the specific area. This is done during the presentation session.
You have about 15 minutes to describe your work. The presentation is preferable in
English.


4. Schedule

Week Activities Hand in to supervisor Deadline
14 Decide on a subject Decided subject Thu 3/4
Search literature

15-16 Search literature
Read literature
Outline the report Outline Tue 15/4
17-18 Read literature
Write report
Meeting with supervisor Book time
19 Write report
Report to supervisor


Prepare presentation


Presentation Session Tue 13/5
20

Update report If needed, updated report to supervisor Wed 21/5
21












5. Assessment

See the course program.

6. Report

The report shall be written using the IEEE template. The report shall consist of:
• Abstract: summary of all parts in the report. The purpose of the abstract is to attract
people to read your report.
• Introduction: introduce the chosen area. The purpose is to give an introduction to the
reader who is not familiar to the specific area, but knows software engineering and testing
generally.
• Description of the chosen area: summary of the chosen area. The purpose is to describe
why, how and for which purpose the area is used.
• Analysis: analysis, critical thinking and future plans within the chosen area. For research
projects, the purpose is to describe the benefits, drawbacks, what research the area needs
in the future. For evaluation and practical projects, the work should be described here
together with a shorter analysis.
• Conclusion: main conclusion in the report. The purpose is to discuss the main points in
the report again.
Another structure can be chosen after discussion with your project supervisor.

7. Presentation

The project will be presented in the project presentation sessions at the end of the course.
Each project will be given 15 minutes to present their chosen area. The presentation should
cover the main points of summary, analysis and conclusion.

8. Project areas

There are several types of projects to choose among. For research projects, 7-10 reports
should be chosen. For evaluation and practical projects, fewer reports are needed. Examples
of project types are:
• Research: solve a research problem; survey the state-of-the-art and identify the research
problems in some area; develop and justify an extension to an existing technique; etc.
• Evaluation: apply and evaluate a technique or evaluate a commercial testing or, analysis
tool.
• Practical: Use an existing technique to test a system or design and implement a
prototype for a system.
If you would like to choose another area, describe the area to your project supervisor
before starting the work. The keywords in Section 9 can be used when you chose an area and
when you search for literature.

8.1 How do you know your test cases are correct?
Test cases are developed to check whether a software system is implemented correctly
according to the requirements specification. However, if the test cases are badly chosen, they
will not detect the failures. There is a method, called Mutation testing, which can be used to
check whether the test cases are correct.
Mutation Testing: A. M. R. Vincenzi, J . C. Maldonado, E. F. Barbosa and M. E. Delamaro,
“Unit and Integration Testing Strategies for C Programs using Mutation”, Software Testing,
Verification and Reliability, 11(3):249–268, 2001.

8.2 How can you measure that the software system works?
Software testing is performed to detect the failures in the software. There are different metrics
to use in order to measure whether the software is correct. One such measure is reliability,
which is defined as: “The probability for a failure-free operation of a program for a specified
time under a specified set of operating conditions.” (IEEE 610.12-1990). There are several
models and techniques that can be used, for example, reliability growth models, Markov
models, statistical usage testing, usage-based testing and operational profile testing.
Reliability growth models: C. Stringfellow, A. A. Andrews, “An Empirical Method for
Selecting Software Reliability Growth Models”, Empirical Software Engineering, 7(4): 319-
343, 2002.
Markov models: J . A. Whittaker and M. G. Thomason, “A Markov Chain Model for
Statistical Software Testing”, IEEE Transactions on Software Engineering, 20(10):812-824,
1994.

8.3 What techniques are most effective to check the requirements and
design documents?
Static verification is often used in the beginning of the development of software. There are
several different techniques and methods used to check the static representation. The com5
mon feature of these is that they have to be manually checked by reviewers. Software
inspection, reviews, walkthroughs are common techniques, which are used together with
reading techniques.
Software inspection and reading techniques: A. Aybuke, H. Petersson, C. Wohlin, “Stateof-
the-art: Software Inspections after 25 Years”, Software Testing, Verification and
Reliability,12(3):133-154, 2002.

8.4 How can you perform testing on an object-oriented software system?
A special branch of software testing is application-based testing. Object-oriented is one
such application, which needs special treatment and special methods to test the system.
Object-oriented testing: Y. Labiche, P. Thévenod-Fosse, H. Waeselynck, M.-H. Durand,
“Testing Levels for Object-Oriented Software”, Proceedings of the 22nd International
Conference
on Software Engineering, pp. 136-145, 2000.

8.5 How can you perform testing on a distributed system?
A special branch of software testing is application-based testing. Testing distributed systems
need special treatment and special methods.
Testing of distributed systems: S. Goeschl and H. M. Sneed, “Case study of testing a
distributed internet-system”, Software Testing, Verification and Reliability, 12(2):77 - 92,
2002.
8.6 What techniques can be used to know when to stop testing?
Stopping criteria are used in the testing phase to determine when a certain quality level has
been achieved. The quality can for example be defined as the reliability or just the number of
faults left in the system. There are several techniques to use as a stopping criterion. One
could, for example, estimate the number of faults left after an inspection or to estimate the
number failures left in testing.
Fault content estimation; L. C. Briand, K. E. Emam, B. G. Freimut, O. Laitenberger, “A
Comprehensive Evaluation of Capture-recapture Models for Estimating Software Defect
Content”, IEEE Transactions on Software Engineering, 26(6):518-540, 2000.
Stopping criterion in testing: J . D. Musa and A. F. Ackerman, “Quantifying Software
Validation: When to Stop Testing?”, IEEE Software, 6(3):19-27, 1989.

8.7 How is testing performed for Agile processes?
Test-first is a principle that is often used in Agile processes. The principle is based on that test
cases are first derived and then these are used to specify how the system should work as well
as used as test cases when the system is developed. There are some questions, which are of
interest within this area, for example, what is the relation between requirements and test
cases, what is the final quality of such a system and how easy is it to maintain such a system?
Test-first principle: M. M. Muller, and O. Hagner, “Experiment about Test-first
Programming”, IEE Proceedings Software, 149(5):131-136, 2002.

8.8 How can different testing techniques (methods) be compared?
One research methodology is to use empirical methods in order to evaluate which testing
methods are best to use. The methodologies are often divided into experiments, surveys and
cases studies. All these methods are important in order to help software organizations to
choose the right testing technique for their purpose.
Experiment: Sun Sup So, Sung Deok Cha, Timothy J . Shimeall and Yong Rae Kwon, “An
Empirical Evaluation of Six Methods to Detect Faults in Software”, Software Testing,
Verification and Reliability, 12(3):155–171, 2002.
Survey: P. Runeson, C. Andersson and M. Höst, “Test Processes in Software Product
Evolution- A Qualitative Survey on the State of Practice”, Journal of Software Maintenance
and Evolution, 15(1):41-59, 2003.
Case study: T. Berling and T. Thelin, “An Industrial Case Study of the Verification and
Validation Activities”, International Software Metrics Symposium, pp. 226-238, 2003.

8.9 What quality models are used for the testing purpose?
There are many general quality models developed within software engineering, and specific
models for testing. The quality models try to capture the essential parts of software
development (testing) so the final quality will be assured. Some of the general models are
CMM, SPICE, ISO 9000, and some of the specific models are TPI and TIM.
CMM: M. Paulk, B. Curtis, M.B. Chrissis, C.V. Weber, “Capability Maturity Model, Version
1.1”, IEEE Software, pp. 18-27, 1993.
SPICE: J-M. Simon, ‘‘SPICE: Overview for software process improvement’’, Journal of Systems
Architecture, 42(8):633-641, 1996.
TIM: T. Ericson, A. Subotic, S. Ursing, ‘‘TIM -- - A Test Improvement Model’’, Software
Testing, Verification and Reliability, 7(4):229-246, 1997.

8.10 How is the software of a nuclear plant tested?
Safety is concerned with the property of a system that it will endanger human life or the
environment. A safety-critical system is, hence, one by which the safety of equipment or
plant is assured. An example of such a system is an aeroplane or nuclear plant system. Safety-
critical software needs special attention when it comes to testing.
Safety: N. G. Leveson, S. S. Cha and T. J . Shimeall, “Safety Verification of ADA Programs
using Software Fault Trees”, IEEE Software, 8(4):48-59, 1991.

8.11 How can test cases be derived and generated?
Software test cases are often derived from documents produced during the software
development. There are different techniques and languages that can be used. Use cases can be
used to derived test cases and two methods that use this approach are usage-based testing and
operation profile testing. Furthermore, there are techniques that are automated and use UML
diagrams.
Usage-based testing: B. Regnell, P. Runeson and C. Wohlin, “Towards Integration of Use
Case Modelling and Usage-Based Testing”, Journal of Systems and Software, 50(2):117-130,
2000.
Operational profile testing: J . Musa, “Operational Profiles in Software-Reliability
Engineering”, IEEE Software, 10:(2):14-32, 1993.
Specification-based testing: J . Offutt, S. Liu, A. Abdurazik and P. Ammann, “Generating
Test Data from State-based Specifications”, Software Testing, Verification and Reliability,
13(1):25–53, 2003.

8.12 How can process simulation be used for project planning and
learning?
Software process simulation is used in a variety of fields in software engineering, for
example, to support process improvement, software project management training, decision
support and to understand empirical research results. This project can either be practical by
focusing on developing a simulation model of a software development/testing project with the
pupose to assist in the project planning, or it could be focused on evaluating existing models’
usefulness in the decision-making.
Simulation models for decision support: Donzelli, P., “A Decision Support System for
Software Project Management”, IEEE Software, 23(4): 67-75, 2006.
Why using process simulation models: Kellner, M., Madachy, R., Raffo, D., "Software
Process Simulation Modeling: Why? What? How?", Journal of Systems and Software, 46(2-
3):91-105, 1999.

8.13 How can tools for software testing be evaluated?
Tools are important in order to implement software testing effectively in an software
organization. However, although tools are needed, they do not solve the problems in the
testing phase. Several tools exist and they have to be evaluated before a software organization
purchases one.
Evaluation of testing tools: Poston, R. M. and Sexton, M. P., “Evaluating and Selecting
Testing Tools”, IEEE Software, pp. 33-42, 1992.

8.14 How can predictions of the software development project be useful
during the testing process?
The project management is often interested in predicting the outcome of the ongoing software
development projects. Both predictions of how much resources that will be required to deliver
on time, and the quality of the delivered software product are valuable for the planning
process. What metrics are useful for predictions during the test process, and how can these be
used? One model to use for the whole development project is COCOMO, but other
approaches could be applied when focusing on the test process.
COCOMO and other models: Boehm, B., Abts, C., Chulani, S., “Software Development
Cost Estimation Approaches – A Survey”, Annals of Software Engineering, 10(1-4):177-205,
2000.
What is needed for making measurements: Schneidewind, N. E., “Body of Knowledge for
Software Quality Measurement”, IEEE Computer, 35(2):77-83, 2002.

8.15 How can web applications be tested?
The quality of web applications mostly relies on the skill of the individual developer. In order
to meet the quality decided by end-users, the processes of developing and testing web
applications need to be formalized.
Web application testing: Di Lucca G. and Fasolino, A., “Testing Web-based Applications:
The State of the Art and Future Trends”, Information and Software Technology, 48(12):1172-
1186, 2006.

8.16 How do you effectively regression test your system?
Regression testing is the process of validating modified software to detect whether new errors
have been introduced into previously tested code. Because of time and resource constraints
for testing, regression test selection techniques have been proposed, to reduce the expenses.
Regression testing: Rothermel, G. and Harrold, M.J ., “Analyzing regression test selection
techniques”, IEEE Transaction on Software Engineering, 22(8):529-551, 1996.



9. Keywords in Software Testing
The below keywords can be used if you want to decide on a subject not specified above. Another
source that is appropriate to use is the Software Engineering Book of Knowledge. The latest
version can be downloaded on http://www.swebok.org.
White-box testing Black-box testing Gray-box testing
Equivalence partitioning
Boundary value analysis Decision
table Finite-state machine-based
Testing from formal specification
Error guessing Random testing
Operational profile SRET
Reference models from code-
based testing (flow-graph,
call-graph) Control flow-
based criteria Data-flow based
criteria Mutation testing
Coverage measures
Application testing Test Methods Testing in Software Process
Object-oriented testing
Component-based Web-based
GUI testing Testing of
concurrent programs Protocol
conformance testing Testing of
distributed systems Testing of
real-time systems Testing of
scientific software
Connection between require-
ments and testing
Architecture Testability
Statistical usage testing
(usagebased) Specification-
based software testing Code-
based, Fault-based Testing
from formal specification
Random testing
Testing metrics and
measurements Objectives of Testing Evaluation of the testing
Acceptance testing Installation
Alpha, beta Conformance,
functional, correctness Regression
Performance Stress Back-to-back
Recovery Configuration Usability
Attitudes and egoless program-
ming Test process Test
documentation and test-wares
Test organisation vs. company
Size Cost/effort estimation
and other metrics Termination
and stop criterion Test reuse
and test patterns
Coverage measures Fault
seeding Mutation score
Comparison and relative effec-
tiveness of different techniques
Stopping criteria
Empirical methods in
software testing
Quality models and
software testing Estimations in software testing
CMM SPICE Testing
with SQA Testing and
certification
Reliability Reliability growth
models, types and classification of
faults Remaining faults and faults
density
Compare testing techniques
Compare inspection and testing
Testing on the management
level
Inspections Testing targets
Unit
Integration
System
Alpha Beta
Process Reading
techniques





Automated software testing Simulation of the testing proc- Testing connected to
develop
(tools) ment process
ess
Tools for different techniques XP

Cleanroom
Journals
• ACM SIGSOFT software engineering notes /
Software engineering notes
• International Conference on Software
Engineering
• ACM transactions on software engineering and
methodology
• Automated Software Engineering
• Empirical Software Engineering
• IEEE Software
• Information and Software Technology
• Journal of Systems and Software
• Software - Concepts & Tools
• Software Engineering Journal
• IEEE Transactions on Software Engineering
• Software Engineering. IEE Proceedings
• Software Quality Journal
• Software Testing, Verification and Reliability
Conferences
• International Conference &
Workshop on the Engineering of Computer-
Based Systems
• International Symposium on
Empirical Software Engineering
• International Symposium on Software
Metrics
• International Conference on
Automated Software Engineering
• International Symposium on Software
Reliability Engineering
Below, there are references to test and process-related conferences. These are almost the same
references as in Appendix I in the book by I. Burnstein.
10. Journals and Conferences
Below, there are some journals and conferences listed which publish results of software
testing research.
11. Test-Related References
Risk-based

Software Testing: Related Conferences Test-Oriented Web Sites
•www.sqe.com •www.rbsc.com
•www.stickyminds.com •www.softwareqatest.com
•www.testinginstitute.com •www.ssq.org
•www.stc-online.org •standards.ieee.org
•www.qualityweek.com •www.ondaweb.com
Software Process and Quality Information •www.mtsu.edu/~storm
•www.sei.cmu.edu/collaborating/spins/spins.ht
ml
•www.stqemagazine.com
•www.sei.cmu.edu •www.soft.com
•www.software.org •www.sast.se
•www.espi.org •www.aptest.com
•www.sqi.gu.edu.au/spice/contents.html •www.testingeducation.com
•www.asq.org •www.opensourcetesting.org
•www.qaiworldwide.com

Sponsor Documents

Or use your account on DocShare.tips

Hide

Forgot your password?

Or register your new account on DocShare.tips

Hide

Lost your password? Please enter your email address. You will receive a link to create a new password.

Back to log-in

Close