| Dynamics
|
| Test Plan
| $Revision: 1.10 $
| State approved
|
| Date 05-Dec-1998
| Author Kimmo Mustonen
|
| Review date 08-Dec-1998
| Reviewed by Jouni Malinen
|
| Approval date 08-Dec-1998
| Approved by Jouni Malinen
|
$Id: test_plan.html,v 1.10 1998/12/08 21:04:29 jkmaline Exp $
Table of contents
Summary
1. Introduction
1.1. Terms
1.2. Document structure
2. Testing areas and objectives
2.1. Preliminary module testing
2.2. Module testing
2.3. Integration testing
2.4. System testing
2.5. Objectives
3. Test environment
4. Organization and reports
4.1. Test reports
4.2. Schedule
5. Test cases and acceptance criteria
5.1. Preliminary module tests
5.2. Module tests
5.3. Integration tests
5.4. System test
6. Testing non-functional properties
6.1. Performance
6.2. Updatability
6.3. Portability
6.4. Usability
6.5. Installation
This test plan describes how the testing of the hierarchical IP
tunneling project will be done. The goal of the testing is to guarantee
that the system meets the specified requirements.
The system will be thoroughly tested by dividing the system into
smaller components, called modules, that will be tested first. After
passing this test they will be combined to form entities and finally
the whole system will be tested. When the system test has been
successfully completed, the system should be a stable working
prototype that can be further developed.
The testing will be done mainly in Spektri Kvintti using the Linux
systems installed there. Most tests can be done without physical
attendance.
The programmer of the module is responsible for the first module test,
called preliminary module test. After that there will be the module
test, integration test and system test that will be done by the test
group. The test group is organized by the test manager. The tester
fills in the test report forms that will be used while examining the
test results. Non-functional properties, like
performance, will also be tested.
This is the test plan for the hierarchial IP tunneling project being
done by the Dynamics group. Testing will help to verify that the
system works as it is supposed to work and that it meets the
requirements specified in the
requirements specification
document. Both the functional and non-functional properties will be
tested.
The terminology used in this document follows the custom taken into
use in this project. The terms are used as in RFCs of the mobile IP
field and they are defined in a separate
terminology specification.
This document specifies a few additional terms that are related to
testing. These terms are not used elsewhere and they are thus
explained here.
Module
A module is a package that consists of a group of functions that are
collected together. That functional entity forms a component to the
system that the other components may use.
Equivalence class
A set of input values that the function or module processes the same
way. In order to be able to test all possible input value
combinations, the input values are partitioned into a preferable small
number of sets. The sets are chosen so that any input selected from
that set produces the same kind of operation. For example, if the
input is a signed integer, partitioning to negative numbers, zero and
positive numbers forms three different equivalence classes to be
tested.
ASSERT() macro
ASSERT() macro tests that the specified condition is true every time
that piece of code is executed. The programmers can use ASSERT()
macros to claim something about the internal state of the function.
Usually there are many conditions that must hold for the correct
operation of the function. For example, if a pointer is passed to the
function, the programmer may demand that it must not be a null
pointer. However, a care must be taken when using ASSERT() macros,
because if the condition doesn't hold, the software execution is
immediately stopped. That means that assertions must not be used to
check if the user has entered valid input to the system but only to
verify the internal state of the software.
This document begins with a short summary.
Chapter 1 (Introduction) describes briefly this document. The
terms used in this document are also explained.
Chapter 2 (Testing areas and objectives) defines what is going
to be tested and the main objectives of the testing.
Chapter 3 (Test environment) defines where the tests will be
made. The hardware and software required for testing are described.
Chapter 4 (Organization and reports) explains the testing
itself. The responsibilities and the test result processing are
presented.
Chapter 5 (Test cases and acceptance criteria) defines the test
cases that will be executed and how the results will be reported.
Chapter 6 (Testing non-functional properties) describes how to test other features of the system such as performance and portability.
The whole system will be tested piece by piece to make sure that it is
functioning the way it is defined and planned. In order to make the
system function well, the underlying components must be carefully
designed and implemented. By starting the test from small independent
modules we try to make sure that the upper level modules have a stable
base underneath to rely on.
The programmers of the modules will make the first tests of the
functions and modules they have implemented. The functions of the
modules will contain plenty of ASSERT() macros to make sure that the
functions are internally working the way they are designed to do. If
any of those assertions fail, we will immediately know what was the
place in code where the internal state of the process was not what the
programmer had planned it to be. There are hardly ever too many
ASSERT() macros in the source code, so we will use them a lot.
If the function or module is only used internally, assertions may be
used to check that the input values to the function are valid.
Preferably, the function should have a way to return an error code to
the calling function. If the functions are called with parameters that
come from any external source, like the user or administrator or the
network, all invalid input values must be handled without crashing the
software. That means that anything the user of the tunnel does, must
not cause the assertions to fail.
The programmer will document the module well enough that it can easily
be tested. The test functions used in the early module test are
included so that the module can easily be tested using that test
program. All the valid and invalid inputs should be categorized into
different equivalence classes so that we don't need to test every
possible combination of input values. The module will be tested until
it is functioning as specified and there are no known bugs. When the
module seems to be working well and the programmer cannot figure out
any way to cause erroneous results, the module is ready for module
testing.
In the module testing all the modules will be tested separately. The
documentation is checked and the source code is read through. The
tester checks all the possible input values and checks that the
programmer has correctly separated the input values to equivalence
classes. If there are any problems, the tester contacts the programmer
to check if there is a problem in the documentation.
The tester will choose at least one set of input values from each
equivalence class, also invalid input values. If any one of the test
cases does not behave like expected and defined or there are some
problems with the source code or documentation, the programmer will be
contacted.
In integration testing the sets of modules will be tested together.
The modules that interact with each other are chosen and tested. All
the modules should have already passed the module testing and be ready
to be connected to other modules. The sets of modules to be tested
will be chosen later. If the integration test fails, we will
investigate what causes the problem and the malfunctioning module will
be corrected.
When the integration tests have been successfully completed, the whole
system can be tested. The installation and configuration can be
tested separately. Also the non-functional properties are tested.
When the system test has completed, the system should be stable and be
able to perform the functions as defined. We will keep in mind that the
system will only be a prototype. The most important thing is that the
main system is functioning so that it can be further developed. All
the small details are not as important as in a final end-user
product.
The test environment will mainly be three Linux machines located in
Spektri Kvintti. The machines are equipped with 2.1.xxx kernel that is
the current development version of the kernel. Most of the tests can
be done online through the network, so physical attendance is not
required. The necessary software is installed on the machines.
When testing modules that depend on other modules, the programmer will
make the necessary stub functions or modules required in the module
testing. The programmer will also make the necessary test functions
needed to test the module. The test function must make it possible for
the tester to easily change the input values and check the output
values and possible intermediate values to make sure that the module
is working as specified.
For integration tests the stub versions of the main system components
are needed. These dummy modules are Home Agent, Foreign Agent and
Mobile Node.
The programmer takes care of the preliminary module testing, including
the addition of ASSERT() macros, dividing the input values into
different equivalence classes and creating the test program for the
module. These tests are done at the same time as the module is
programmed. All the other tests are done by the test group organized
by the test manager. The test group consists of the team members and
one or more external volunteers.
The programmer makes the documentation for the module. No separate
test report is required for the preliminary module test. The other
test results are reported by filling the test form. The
module test form
is the same for all modules. It is filled by the tester who finally
tests the module.
The integration test consists of multiple modules that are combined to
make a functional entity. The tester fills in an
integration test form
that contains the names of the modules and the test results.
The system test is used to test that all components of the whole
system work together. The tester fills in the
system test form
that includes the requirements from the requirement specification. The
goal is that the final system passes all these tests.
Module testing is done regularly during the phases. When the
programmer has made a module, done the preliminary test for it and
documented what it should do, the test manager organizes testing of
that module. In each phase of the project, all changed modules will be
tested. The integration tests are done when one whole entity of the
system is ready to be tested. System tests are done at each phase to
verify that the product works at the end of each phase as planned.
Some of the non-functional tests can and also will be done during
different phases, like estimation of the performance, updatability and
portability.
In the P1 phase, all the modules that are used in the prototype will
be tested. Integration tests of the entities to be prototyped will be
done. The requirements that should be implemented in the prototype
will be selected and system tests are run to verify that those
properties have been implemented as required.
At the end of the P2 phase the main properties of the system should be
functional. The module tests are done on the modules that have
changed. Integration tests on the entities, at least
Mobile Node,
Foreign Agent
and
Home Agent
will be completed. All the system test cases will be checked to verify
which of them are still to be implemented. We may be able to test some
of the non-functional properties in the previous phases but the main
goal is to test them in the P2 phase.
At the end of the project, the changed modules and entities are tested
and all the system tests will be performed to make sure that the
system is functioning as documented and planned. The non-functional
properties will also be tested and documented.
The system is divided into modules. All the modules have their own
test cases depending on the module. They will be generated by the
programmer and the tester of the module will check them. When the
modules are functioning, the three main components will be tested
separately in the integration tests. The requirements of the final
product are tested in the system test phase.
The programmer divides the input values to different equivalence
classes. At least one test case of input values from each equivalence
class is performed.
The preliminary module test is accepted when the module works as
defined and planned. This requires that all the test cases that the
programmer could invent have been done and no known faults exist. The
module must function properly with any valid input value and it must
not crash with any valid input value. If possible, the module should
recognize invalid input values. If the input came from the user, the
module must be able to handle the error without causing failed
assertions.
A person from the test group verifies the equivalence classes created
by the programmer and verifies that all the possible cases are
covered.
The module test is accepted when the module works the way it should
and none of the test cases has revealed any bugs. The end user may not
be able to crash the module in the system. If the module is used only
internally and there is no reasonable way to report the invalid input
values, the module may crash because of assertion failed. However,
only internal programming errors may be able to cause that kind of
situation, the end user must not be able to cause this.
Each module test produces a filled version of the
module test form.
When the different modules and their interaction is known, the team
will select a group of interacting modules and test that the modules
work together. At least the three main components of the system will
be tested in the integration test. These components are
Home Agent,
Foreign Agent
and
Mobile Node.
Each test produces a filled versions of the
integration test form.
The functionality of the Home Agent will be tested. The integration
test cases of this component are defined in the
Home Agent test form
and the test results will be filled into the same form.
The functionality of the Foreign Agent will be tested. The integration
test cases of this component are defined in the
Foreign Agent test form
and the test results will be filled into the same form.
The functionality of the mobile node will be tested. The integration
test cases of this component are defined in the
Mobile Node test form
and the test results will be filled into the same form.
At least the system with one Home Agent, two Foreign Agents and one
Mobile Node will be tested. The installation and configuration will be
tested separately. The test cases are in the
system test form.
The test produces a filled version of the test report form.
Usually testing a functional part gives the result quite easily, it is
either working as it should be or not. Measuring non-functional
properties is more difficult. Usually the level of the non-functional
property can only be estimated.
The non-functional parts of this system are performance, updatability,
portability, usability and ease of installation.
The performance of the tunnel can be tested by transferring a big file
for example using FTP through the tunnel. The same file is transferred
also without the tunnel. The resulting speeds can then be compared.
Response times can be tested in a similar way. We choose a node in
the home network and ping it using the tunnel and repeat the same test
without the tunnel. The node in the home network is chosen because
otherwise the routes could be totally different.
The system and especially the protocol version numbering scheme is
analysed to verify that it is possible and easy to create a simple
protocol enhancement or design new protocol versions.
Some parts of the source code will be tested if they can be compiled
using different compilers on different systems. In addition to
gcc/Linux/i386, also gcc/Digital Unix/alpha, cc/Digital Unix/alpha and
SAS/Amiga/m68k will be tested. That way we can have some idea about
the portability of the system. This can also reveal some programming
errors, because different compilers report different kind of things.
Because of the kernel differences, the system will probably not be able
to run on those environments without taking additional actions that we
are not going to do. Thus, porting to other systems will not be done.
The overall usability of the system and the configuration tools will
be estimated. A user who is not involved in the project should be able
to establish tunneling and control the tunnels with the help of the
online help functions after having read the user's manual.
The system will also be installed by users that have not been involved
in the project. If the installation causes problems, the installation
instructions have to be made easier to understand.