Software Testing General Overview
Content
Damijan Ćavar http://blog.cavar.si http://www.linkedin.com/in/damijanc
Limitations Time Qualit y
Budge t
Every problem not fixed before production will cost 10x more to fix later.
M a n u al te GUI st Tests s Acceptance Tests
Mike Cohn pyramid
Where is usability in this pyramid?
Cost
Unit tests
Why, when, benefits Goal Testing main goal is to create robust application that works according to specifications. Quality Testing has no impact on software quality. Poorly written code will remain the same. Testing just helps developers fix software. When ? When we are going to support the application ? Single user “Code and forget” projects (admin scripts, simple programs) have no need for test cycle. Benefits Discover weak points in implementation (performance, bugs ...). Give more realistic predictions of project completion. Can validate code after changes have been made (If I change this, will my application still work ???). Will make software adaptation by end users much easier.
FURPS ??? FURPS is an acronym representing a model for
classifying software quality attributes (nonfunctional requirements):
Functionality - Feature set, Capabilities, Generality, Security Usability - Human factors, Aesthetics, Consistency, Documentation Reliability - Frequency/severity of failure, Recoverability, Predictability, Accuracy, Mean time to failure Performance - Speed, Efficiency, Resource consumption, Throughput, Response time Supportability - Testability, Extensibility, Adaptability, Maintainability, Compatibility, Configurability, Serviceability, Install ability, Localizability, Portability
FURPS are essential for limiting the tests and should be a
part of every project documentation and confirmed by user.
What should we test part 1 ??? Unit tests In general should be implemented by developer to validate his/her work. They can be prewritten if interface in known in advance. Business logic Most difficult to test. Requires knowledge of business work flow. Usually also done by customer before takeover and therefore not part of the tests, unless business scenarios are defined. GUI Lacks automation (test suites exist but they rely on visual representation - expensive). Very difficult to catch the users interaction and to predict program flow. Best way is by hand unless application design allows testing of GUI functionality. Web Web test is nothing but number of requests called in predefined order. This is now supported with VS TS and by fiddler and … .
What should we test part 2 ??? Security Difficult, requires a lot of knowledge and in general it should be considered in design time. Load Only true test is usually preproduction test with real users. Simulation tools do exist. Goal of load tests is to specify minimum hardware requirement or to estimate the load that end users infrastructure will handle. Should also reveal where the bottle necks in design are. Deploy scenarios This will help support team confront possible difficulties before end user does.
Unit tests ??? There are two basic types: Positive (easy): Validates the function IO.
Negative (difficult): Tests error handling of function. Gives robustness to application. Description of error returned is good quality measurement, helps both users and support engineers to quickly define the problem. At least all expected errors should return descriptive responses.
Programs should check input since white listing is much simpler than black listing.
Business logic tests ??? Explanation in simple words: According to predefined scenario test should reproduce user action and check results of it (purchase an item, reserve a room, insert customer …) Tests must be repeatable an repeated
throughout the lifetime of an application. These test require bussines knowledge and will require a Bussines Analists contribution
GUI ??? General: This test is usually part of business logic test. Standards (general design of forms: double click, buttons look and behavior, context menus …) for GUI design should be defined in advance and is also a part of test. Web specific: When testing websites list of supported browsers and plugins with versions must be provided to do the test Every combination (IE7 + Adobe Flash 6) should be tested.
LOAD ??? General: Load test provides information about maximum load that application will handle in expected response time. This is very important part of FURPS Web load tests are well supported by test suits. MS has provided test tools for MSSQL also. Other platforms (databases) have little or no support.
What is needed for tests??? Clean environment Systems used must be as close to users configuration as possible. OS: must be the same as end users, free of add-ons, unnecessary service packs,
patches, developer tools and with default settings.
Database: same versions and if possible on similar hardware Servers: same as OS, but hardware must be as close as customers, or at least
minimum configuration must be specified (load test). Specifications, scenarios, manuals, code comments All of the above are the tools that testers need in order to create and validate tests. Test data (quantity and contents) Databases should be filled with expected amount of quality data for testing purposes. Empty databases are useless for testing. Limits In theory tests must be independent from infrastructure (database, external services) and should test limited functionality (scenario)
Usability General:
Purpose is to get real users in front of the application and use it. Software is useless if users cannot/will not use it (complicated). Since smartphone was introduced users expect speed and simplicity more than ever. Why was Windows 8 a failure ?
Management point of view Testing costs (a lot of) money : Testing is usually one of the things that is are considered as a cost center on a project and there for struggle is always on to justify testing cost. Testing must not be an excuse for programmers to write poor code. Developers must deliver working code (forms must resize properly, click of a button
must not crush an application ...)
Testing extends development time but shortens support time: Every test done extends the time to finish the project, but it reduces support (bug fix) costs. Finding the balance is tricky. Testing allows to predict and solve problems before they become production bugs.
Testing shortens time needed introduction of changes: In production environments testing is the only way to ensure that new feature does not compromise the existing application. Testing is not always an option(but should be): Usually is a matter of deadline, lack of dedicated resources or budget.
Limitations Time Qualit y
Budge t
Every problem not fixed before production will cost 10x more to fix later.
M a n u al te GUI st Tests s Acceptance Tests
Mike Cohn pyramid
Where is usability in this pyramid?
Cost
Unit tests
Why, when, benefits Goal Testing main goal is to create robust application that works according to specifications. Quality Testing has no impact on software quality. Poorly written code will remain the same. Testing just helps developers fix software. When ? When we are going to support the application ? Single user “Code and forget” projects (admin scripts, simple programs) have no need for test cycle. Benefits Discover weak points in implementation (performance, bugs ...). Give more realistic predictions of project completion. Can validate code after changes have been made (If I change this, will my application still work ???). Will make software adaptation by end users much easier.
FURPS ??? FURPS is an acronym representing a model for
classifying software quality attributes (nonfunctional requirements):
Functionality - Feature set, Capabilities, Generality, Security Usability - Human factors, Aesthetics, Consistency, Documentation Reliability - Frequency/severity of failure, Recoverability, Predictability, Accuracy, Mean time to failure Performance - Speed, Efficiency, Resource consumption, Throughput, Response time Supportability - Testability, Extensibility, Adaptability, Maintainability, Compatibility, Configurability, Serviceability, Install ability, Localizability, Portability
FURPS are essential for limiting the tests and should be a
part of every project documentation and confirmed by user.
What should we test part 1 ??? Unit tests In general should be implemented by developer to validate his/her work. They can be prewritten if interface in known in advance. Business logic Most difficult to test. Requires knowledge of business work flow. Usually also done by customer before takeover and therefore not part of the tests, unless business scenarios are defined. GUI Lacks automation (test suites exist but they rely on visual representation - expensive). Very difficult to catch the users interaction and to predict program flow. Best way is by hand unless application design allows testing of GUI functionality. Web Web test is nothing but number of requests called in predefined order. This is now supported with VS TS and by fiddler and … .
What should we test part 2 ??? Security Difficult, requires a lot of knowledge and in general it should be considered in design time. Load Only true test is usually preproduction test with real users. Simulation tools do exist. Goal of load tests is to specify minimum hardware requirement or to estimate the load that end users infrastructure will handle. Should also reveal where the bottle necks in design are. Deploy scenarios This will help support team confront possible difficulties before end user does.
Unit tests ??? There are two basic types: Positive (easy): Validates the function IO.
Negative (difficult): Tests error handling of function. Gives robustness to application. Description of error returned is good quality measurement, helps both users and support engineers to quickly define the problem. At least all expected errors should return descriptive responses.
Programs should check input since white listing is much simpler than black listing.
Business logic tests ??? Explanation in simple words: According to predefined scenario test should reproduce user action and check results of it (purchase an item, reserve a room, insert customer …) Tests must be repeatable an repeated
throughout the lifetime of an application. These test require bussines knowledge and will require a Bussines Analists contribution
GUI ??? General: This test is usually part of business logic test. Standards (general design of forms: double click, buttons look and behavior, context menus …) for GUI design should be defined in advance and is also a part of test. Web specific: When testing websites list of supported browsers and plugins with versions must be provided to do the test Every combination (IE7 + Adobe Flash 6) should be tested.
LOAD ??? General: Load test provides information about maximum load that application will handle in expected response time. This is very important part of FURPS Web load tests are well supported by test suits. MS has provided test tools for MSSQL also. Other platforms (databases) have little or no support.
What is needed for tests??? Clean environment Systems used must be as close to users configuration as possible. OS: must be the same as end users, free of add-ons, unnecessary service packs,
patches, developer tools and with default settings.
Database: same versions and if possible on similar hardware Servers: same as OS, but hardware must be as close as customers, or at least
minimum configuration must be specified (load test). Specifications, scenarios, manuals, code comments All of the above are the tools that testers need in order to create and validate tests. Test data (quantity and contents) Databases should be filled with expected amount of quality data for testing purposes. Empty databases are useless for testing. Limits In theory tests must be independent from infrastructure (database, external services) and should test limited functionality (scenario)
Usability General:
Purpose is to get real users in front of the application and use it. Software is useless if users cannot/will not use it (complicated). Since smartphone was introduced users expect speed and simplicity more than ever. Why was Windows 8 a failure ?
Management point of view Testing costs (a lot of) money : Testing is usually one of the things that is are considered as a cost center on a project and there for struggle is always on to justify testing cost. Testing must not be an excuse for programmers to write poor code. Developers must deliver working code (forms must resize properly, click of a button
must not crush an application ...)
Testing extends development time but shortens support time: Every test done extends the time to finish the project, but it reduces support (bug fix) costs. Finding the balance is tricky. Testing allows to predict and solve problems before they become production bugs.
Testing shortens time needed introduction of changes: In production environments testing is the only way to ensure that new feature does not compromise the existing application. Testing is not always an option(but should be): Usually is a matter of deadline, lack of dedicated resources or budget.
