In order to understand and use software testing methods there are some basics we need to be aware of. In this page, I will try to clarify these basic issues based on two articles that had great impact in the practice of software testing.
“What is software testing? And Why is it so hard?” by James A. Whitaker
What is software environment for testing purposes?
Software operates in an environment which effects its behavior via the inputs. Inputs can only effect the software using the interfaces provided by the software or the operating system. Whitaker defines 4 main types of interfaces. These are;
Human interfaces: These are the straightforward interfaces used by humans to input data into the software. Graphical user interface and command line interface are the most common human interfaces.
Software Interfaces: Application Programming Interfaces are mainly the software interfaces used to interact with a black box commercial off-the shelf software to make use of a provided service.
File system interfaces: File system is used to store data permanently in softwares. Low-level system APIs are used to handle this service. However, depending on the platform (Linux, Windows or Mac OS) the messages received during an exception will be different. Testers should test the software against these unexpected events like disk full, write not permitted.
Communication interfaces: Mostly the distributed systems rely on communication interfaces for transferring data. While providing convenience to the developers, these interfaces brings vulnerabilities arising from protocols and packaging limitations. These issues must be considered by the testers.
What is typical use scenario?
Typical use scenario refers to operating the software with the most common execution paths or input sequences to simulate most common usage profile. However, testing only the most common is not a good idea considering the significant probability that bugs may reside in those execution paths or input sequences which were ignored.
What is the role of test data adequacy criteria in testing process?
The main challenge testers are facing is the infinitely many execution paths (sequences of code statements representing an execution of a software) and input sequences that represent scenarios to which the software must respond.
The test data adequacy criteria provides a solution to this challenge by choosing the adequate and economical representation or sample of these infinite sets.
This can be achieved by selecting a test set that cause each source statement to be executed at least once and each branching structure to be evaluated once for execution paths. Additionally, each data structure must be initialized once with the chosen test set.
When it comes to input sequence, selected test set must contain each physical input causing each interface control to be stimulated.
The discrimination criterion under input sequence suggests selecting a set of tests that have the same statistical properties as the entire input domain.
What is regression testing and what potential problems are involved in regression testing?
In software development life-cycle, testers detect some bugs in the software and report them. After this, developers usually create a new version of software after removing these bugs. After each new version software needs to be retested. This test is called regression testing.
The potential problems in regression testing (proposed by Whitaker):
Regression test takes additional time which might be useful for testing the code for the new functionality often brought with the new version of software,
Testers usually deviate from the purpose of test data adequacy criteria by seeking only to show the absence of a fault.
In order to get rid of these pitfalls testers should work closely with developer while they are debugging and during the retesting they should increase the test set to cover the newly added parts.
What makes software testing so hard?
First of all, considering the infinitely many execution paths and input sequences the first challenge is to selecting the subsets that represent the overall paths and input sequences. Moreover, the tester need to choose either covering execution paths or input sequences. Covering both of them is not straightforward.
Secondly, testing is an iterative process with each debugged new version of the software. New version brings other challenges related to regression testing and extending the test data adequacy criteria to include new functionality added to the software.
Last challenge is determining when to finish the testing. Software reliability models are promising solving this problem at some level.
There is no deterministic solution to all these challenges. For these reasons, majority of computer scientists such as Whitaker claims that software testing is so hard.
“Seven Principles of Software Testing” by Bertrand Meyer
What is the relationship between software quality and testing?
We can get very little information about the quality of software by looking at the test results. Because we are using only a limited subset of infinitely many execution paths and input sets. This looks like a drop in the ocean as suggested by Meyer. He also adds that any inference to quality is the responsibility of quality assurance. Testers should use all efforts for finding faults rather than evaluating the quality.
Also a famous computer scientist, Dijkstra, suggests that program testing can be used to the presence of bugs not the absence of bugs.
When we detect bugs in a software, we can prove the presence of bugs. However, when we can’t detect any bug, this doesn’t mean that the software is bug free. Even after the development team fixes the faults, there will be bugs. We never have evidence to disprove this.
The main motivation for software testing should be finding the bugs. Because the opposite, that is proving the absence of bugs is beyond the scope of software testing considering the limited subsets we can select from execution paths and input sequences.
What is the relationship between tests and specs? Can one substitute for another?
Tests are only instances of specifications. No matter how large test set we select, we won’t be able to capture the abstraction the specifications provide. I agree with the Mayer on his claim that tests are no substitute for specifications. However, specifications can be used to generate test cases.
What is the rationale for regression testing?
The assumption that previously corrected faults can appear again after fixing other bugs is the main reason for using regression testing. Mayer also adds that any failed execution must yield a test case to remain a permanent part of the project’s test suite.
What is the role of test oracle?
Test oracle addresses the need for automation for large number of tests. Without an oracle the tester needs to examine all test results manually. And obviously this will not scale. Test oracles can be part of the program text in the form of contracts. This is the desired form. However, it is still possible to adopt weaker forms.
What is the relationship between manual and automatic test cases?
Testers think about the problem domain and data structures, and create manual test cases. Owing to the limitations of human being these test cases will be limited in number and scope. In contrast, automatic tests try many values. These values include the extremes that are usually missed by humans.
The first paper was pointing out the deficiency in applying software testing methodologies in software development life cycle. In contrast, second paper is talking about the relatively matured software testing paradigm. This is partly impact of test-driven approach the AGILE method has brought. However, this time there is some obvious wobbling to understand the hierarchy of specifications with the latest approach. Specification can not be replaced by many tests.
I have noticed another difference about applying regression testing. Previous paper was assuming that all test sets from the previous built would be used for retesting the new version of software. And this was criticized by Whitaker. However, the second paper is limiting this to the test cases that had detected bugs previously.
Having discussed these issues here, I will support this section by adding posts in Software Testing category about best practices and language specific solutions.
Recent Comments