Computerization has become an integral part of human life in the modern world and so is the software that runs them. Human interactions with these systems software are very complex. Modern appliances and gadgets at home, cars, banks, factories, schools, airplanes, and critical life-support devices use software products to run their functions.
System failures will not only cause inconveniences and bitter experiences, they can also have serious consequences like loss of lives and properties. It is imperative that software developers develop numerous specifications for software reliability.
Testing is still the best way to ensure software reliability and viability. Software developers have been using many dependable testing methods during different stages of software development. However, testing concurrent software developmental process is still very difficult, because of its inherent problems of non-determinism and synchronization.
The traditional or sequential testing techniques are not enough to trace hidden errors in concurrent programs. As software systems become complex, additional concurrent programs are developed and released for use. Concurrent programs are suited for faster and more efficient running of multiple processes.
What is Concurrent Testing?
To truly understand concurrent testing, it is necessary to understand concurrent programming first. Concurrent programming is used when there is a need to run a number of simultaneous processes and dynamically share information with one another. The dynamic sharing of information is one of the most important characteristics of concurrent programming, which result usually in a faster, but a partially defined order of execution. This means that the program will have only partial control in the order of execution of the different processes those are waiting in the queue.
A process may start sequentially and then at any point in the execution, the whole program in the order of execution may simply change or alter, which is brought about by the shared information with the other processes. The differences or changes in the order of execution of the same data input may produce different or non-determinable results.
Non-determinism and synchronization problems make testing of concurrent software quite challenging and difficult. Non-repeatability of results also makes it difficult to detect errors and consequently the implementation of corrective actions. Furthermore, testing concurrent programs is far more difficult than testing sequential program.
The outcome of a sequential program is repeatable, that is, a given input that is processed under similar conditions will have similar results, no matter how many times you repeat the process. The possible sources of risks and errors in a sequential program may multiply many times over in a concurrent program, because of the sharing of information with the other concurrently running processes.
Non-determinism in a program may give rise to the following difficulties:
Concurrent test programs – Due to the inherent non-deterministic nature, the failure modes are not repeatable in software development process. Its occurrence is probabilistic; with a given set of inputs and conditions, the error may not appear just one time, but in the next test run as well. There might be some problems at the client’s side, but the problem may not manifest itself during trouble shooting verification
Debugging activities – Additional debugging activities are discouraged, as they may only introduce additional variations and result in additional problems.
Concurrent testing will help you verify, whether the system is functioning and operating as it should be. However, when you run a concurrent process, repeatedly under identical conditions and with different results or behavior, how would you ensure the reliability and trust-ability of a concurrent program? With the many testing difficulties highlighted and detected, how can you ensure the efficiency of concurrent software? On the contrary, concurrent testing will help you prevent all those reliability questions and doubts about the software product.
Running Concurrent System Tests – Simple Methods and Techniques
Concurrent tests are the tests that help improve the reliability and robustness of concurrent programs. Although a number of tests are available to test the reliability of software programs, only a few of them can work in combination to provide the best results. Some of these tests are described below:
Code Review: Code review proposes the time-consuming process of reviewing embedded codes and their structures, and procedures to do it. Srivathsa Sankaran is the pioneer programmer, who developed this system of review.
Static Analysis: This useful tool can check and evaluate the coding system before their execution and it can easily detect different errors and bugs in the system. This works almost like the code review test.
Extensive and Random Testing: Cameron Purdy is the software professional, who divulged how to improve the extensibility and coverage area of the test cases by randomizing test inputs. He also gave many invaluable tips on how the testing process can be made efficient one. These tips are:
- Always test near zero and near the overflow
- Test random operations
- Multi-threading the test
- Perform on 5-10 threads only, not on too many threads at a time
Brian Goetz advices the following two basic rules:
- Structure programs to limit concurrent interactions
- Test concurrent building blocks
Fuzz Testing: Elliotte Harold explains that in fuzz testing, you will be exposing a program with randomized feeding of bad data and then wait to see what the program does. It is not a logical test. Rather than attempting to guess what data is likely to provoke a program crash (as a human tester might do), an automated fuzz test simply throws as much random gibberish at a program as possible. The failure modes identified by such testing usually come as a complete shock to programmers, because no one would have thought about them.
Reachability Testing: This test combines both deterministic and non-deterministic testing. Unfortunately, it also needs a large number of sub-tests and is impractical for many applications. There is a proposed research program to enhance the positive characteristics of reachability testing and improve on the negative side to make it an efficient testing tool for concurrent programs.
Extending Concolic Testing: Concolic testing is easily extendable and expendable with race-detection and flipping algorithm. It can be used for testing multi-threaded or concurrent programs. This method was developed originally for the C and Java programs.
Without the extension, concolic testing is only effective for testing sequential programs. Concolic is the combination of two testing methods – concrete and symbolic, which were proposed first by Larsen and Austin in 2003. Based on concolic testing, CUTE and jCUTE were later developed, for testing C and Java programs respectively.
Con Test: This tool is designed for detecting synchronization faults in multithreaded Java programs. Planned to complement and address the deficiencies of unit testing, this testing is not so good at finding concurrent bugs, thus allowing them to escape detection, until at a later stage in the development process.
Innovations in testing methods have the ability to ensure software quality and reliability without sacrificing cost and productivity. A good testing method will give reliable testing results with least numbers of performed tests and least cost and time. Most of the operational concerns or pitfalls like deadlocks, data corruption, or system hangs are classified as “emergent phenomena”. Such operational concerns are not results of code errors. However, they may simply occur as results of various interactions among simultaneous processes within the system. In fact, they may just occur without any reasons.
What can the end user or client do against this “emergent phenomena”? Would this phenomenon make the tester helpless or clueless? The most practical suggestion is to make sure that such troubles never happen again. Testers may need to know and understand the system in their hand and later control its basic functionalities so that all collective effects of the “emergent phenomena” remain at a manageable level.
Based on the observations made, what could be the cause for a system to hang? To prevent such an eventuality, you may need to create a system that resists hanging. It is also possible to create a “constructive emergent phenomena”. This is a useful situation where certain problems solve all by themselves. However, testers will never know when this phenomenon happens.
Everyone knows how difficult it is to test concurrent programs. However, you just cannot avoid conducting concurrent test programs as a majority of software developmental activities are diverse and concurrent. Programs written are duplicates and most of them are concurrent too.
Some examples of concurrency in the real-time scenarios are:
• Cooking where the people working in the kitchen could work on diverse activities like cutting vegetables, boiling broths or even arranging for delivery of cooked food.
• Railway networks where trains run on different tracks or in the same tracks based on timings.
Because of the probabilistic occurrence of concurrent system errors/bugs, you can never ensure that testing will completely screen out or remove all of them. Both the developer and the client should accept this truth. The main goal of testing concurrent systems is to increase the confidence or satisfaction levels on the reliability of the software by minimizing the occurrences of errors and bugs to a universally acceptable level.
