New tools that help software testing

Releasing a product with bugs is potentially very expensive, especially considering the cost of field upgrades, recalls and repairs. Less quantifiable, but at least equally important, is the damage done to reputation and consequent loss of customer goodwill.

Many embedded products are nevertheless released without all the testing that is necessary and/or desirable. Consequently, software quality is receiving increasing focus in embedded application development.

Most existing software testing tools can only execute the tests on a PC, the use of which is limited when testing an embedded application because the actual system will have numerous differences, including hardware interfaces, timing issues and memory constraints among many others. Therefore, it is important to run as many software tests as possible on the actual embedded hardware to minimise inter-platform differences.

The challenge is to find automated in-target software testing tools that can be easily integrated into developers’ embedded development environments. These tools should be able to analyse the application source code, auto-generate a test suite and automatically run it on the target board.

However, a test report showing 100% success may not be sufficient, as the test procedure itself may not have been applied to more than a small fraction of the code.

Understanding the quality of test procedures thus becomes critical in judging whether a product is ready for release.

However, new and advanced tools for embedded test automation and test quality measurement are now emerging.

A key element of the software testing process is ‘unit testing’. Unit tests call up C programing functions with different combinations of input parameter values to drive the code through different execution paths.

Other than for trivial functions, writing unit tests by hand is labour intensive, costly, tedious and leaves the possibility that many execution paths remain untested.

A detailed analysis of how input parameters drive the code is therefore required, so that as many as possible of the important execution paths throughout a function are actually exercised by a test suite.

Another challenge faced by development teams is maintaining the synchronisation of unit tests with code under development. This problem is often exacerbated when delivery schedules become compressed with project progression.

If source code and unit tests become out of sync, the tests will become less useful, especially when they are most needed.

The unit test tools available to PC developers are less useful to embedded developers as they rarely manage compilation, downloading and execution of the test suites on embedded boards.

Tools that create the unit tests automatically, build them into the embedded application and allow them to run on the target via a convenient pathway (such as a JTAG debugger) will result in optimal usage of time and maximum productivity.

An effective approach is to use a full embedded test automation system that is integrated right into the C/C++ development IDE (integrated development environment), so that synchronisation is easy to maintain and ancillary tools such as JTAG debug probes can be used to full effect.

Such tools have not been common previously, but new tools now meet these demanding criteria and bring powerful in-target test automation capabilities to software developers.

The new breed of professional embedded test automation tools can analyse the source code and generate and execute suitable test suites automatically on the target board.

An example is TrueVERIFIER, which auto-generates a test suite (implemented in C source code) and makes many function calls with different combinations of input parameters, thus driving a large number of different execution paths.

Figure 1, shows how a trivial C function can be tested, by calling it many times with different input parameters. The selection of input parameter values is selected and generated automatically by the testing tool.

Once the test suite has been generated (as C source code), it must be compiled, linked and executed on the target system. Many low-cost unit test tools only run unit tests on a Windows PC.

The inability of these tools to integrate with the embedded tool development environment results in additional inefficiency of not being able to execute the tests on the target board.

TrueVERIFIER integrates into a professional embedded IDE, enabling easy synchronisation of test with code development. It also takes advantage of integration with tools such as JTAG debug probe.

Once code has been developed and tested, the focus shifts to quantitatively understanding what actually happened during testing. Code coverage measurement, which is performed using dynamic execution flow analysis, is commonly used to study which parts of the code have been tested.

There are many different types of code coverage analysis from the very simple to the very stringent, such as MC/DC, which is often used for measuring the test quality of safety-critical software, for example, in flight-control and aerospace applications. However, many other projects can also benefit from this advanced test analysis.

Some code coverage analysis types are:

• Statement or block coverage measures how many of the C-statements or code blocks have been executed during a test session. It does not measure how branches affect which C-statements or code blocks are executed;
• Function coverage only measures which or how many of the C-functions have been called during a test session. It does not measure which, or how many, of the function calls in a code section are actually executed, or the quality of the testing of the function itself;
• Function call coverage measures which or how many of the function calls in a code section have actually been called during a test session;
• Branch coverage measures whether all code blocks and alternate branch paths have been executed in a code section (such as both the if- and the else- part in an if-else statement). Branch coverage typically requires a code section to be executed a sufficient number of times, so that all alternative branch directions will be tested. As all branch paths must be executed, all corresponding code blocks are executed as well;
• MC/DC (modified condition/decision coverage) is an advanced type of code coverage analysis. This kind of code coverage is applied to applications of which the highest reliability is expected. It extends branch coverage with the additional requirement that all sub-expressions in complex decisions (such as in a complex if statement) must drive the branch decision independently of the other sub-expressions.

Until recently, there have been few test-quality measurement tools available and they have been limited by one or more of the following problems: testing using weak types of code coverage analysis only; testing in PC environments and not on the embedded target; expensive; difficult to use; and lacking integration with other embedded development tools.

However, TrueANALYZER is highly integrated and supports the wide range of code coverage analysis types previously described.

One of its most powerful features is that the analysis is conducted on the embedded target. Connecting to the target via a JTAG probe, the embedded application is run directly on the target board with the code coverage information recorded for later inspection and quantitative analysis.

As an example of the tool’s capabilities, Figure 2 shows a trivial code section that contains three blocks: a red code block, which is always executed; a green block, sometimes executed depending on the branch decision made in the if statement; and a blue block, again, always executed.

The branch decision in the if statement drives the selection of one out of the two possible execution paths: either directly from the red to the blue code block; or passing from red to blue via the green code block.

Figure 3 shows how the easy to use, in-target TrueANALYZER tool, using only two mouse-clicks, detects all the different types of code coverage, including statement/block coverage, function coverage, function call coverage, branch coverage and MC/DC.

Both the TrueVERIFIER and TrueANALYZER tools are integrated in the TrueSTUDIO C/C++ IDE.

It is this new generation of embedded test tools that will enable software developers to be confident of delivering high quality software, making a significant contribution to new products that work first time.