Streaming Media Proxy for Mobile Communication

Master Test Plan

Executive Summary

Acceptance testing measures the goals of the whole project described in the Project Plan [PROJPLAN] and the Requirement Specification [REQSPEC]. These high level targets are set by the customer. Therefore, the acceptance testing must be considered more from the customer point of view, as a final approval for the quality of the delivered product. One of the goals is the comprehensive documentation due to the research-oriented nature of the project. The acceptance testing also requires that the results of system testing must be presented as one criteria for successful exit of the project.

The acceptance test plan must be inspected by the customer in order to agree the common goals with the project team. Also the actual testing at the end of the project may be performed together with a representative of the customer, if that is seen feasible. This is decided by the customer. The functional requirements stated in the Requirement Specification [REQSPEC] and revised in the Functional Specification [FUNCSPEC] are not targets of acceptance testing. The testing of these features is covered in detail by the system testing.

The purpose of the system testing is to let the customers and users to check that the system fulfills their actual needs documented in the Requirement Specification [REQSPEC]. It concentrates on the non-functional requirements like installability, usability, maintainability, modularity and documentation of the system. For all these features, it is characteristic that it is challenging to find measurable key factors.

The system testing is performed as black-box testing. In this approach, it is assumed that the implementation details of the system are hidden and unknown to the tester(s). Testing concentrates on the interfaces of the system, especially to its externally observable functionality and input-output  behavior. In practice, this means the tester taking the roles of the end user and the system operator.

Table of Contents

1. Introduction

1.1 Scope

The end product of this project consist of an architecture for streaming media proxy and a prototype that implements a set of the key nodes in that architecture. A major contribution is the project documentation: it defines the end user scenarios and use cases, requirements, functional and technical specifications, test plans and reports. Due to the research-oriented approach of the project, the quality of this documentation is considered essential. The design decisions and the rejected alternatives for those must be reasoned and documented. In other words, there is more emphasis on the documentation than the actual binary code program developed. This emphasis is required by the customer.

The intended audience of this document is the customer, course personnel and project members.

1.2 Acronyms and definitions

1.3 Changes from T2

1.4 Changes from T3

2. Testing environment

2.1 Hardware

2.2 Software

Burana is used for error reporting when the project team finds it feasible. The target is set to start using it at the beginning of phase T4. 

Phase T4: considering the minimal benefits and additional bureaucracy involved, the team decided not to use Burana software for error reporting. Instead, information is stored in test reports as described in chapter 4.2 'Defect handing'. 

2.3 Tools

Netscape browser (4.0 or higher) is used as a client software running on Win98/WinNT operating system. Also a media player supporting mpeg and avi formats (e.g. RealPlayer or Windows Media Player) is required.

CTC++ code coverage tool may be used, if white-box testing method is used. This is decided when planning testing for each phase individually.

Phase T4: All intergration and module tests were automated. See chapters 7 and 8 for details.

3. Training

No other training is seen necessary at this stage. If new testing tools are introduced, more training may be scheduled if needed.

4. Process

4.1 Description

At the bottom, testing concentrates on the individual modules and their functionality, at the level of source code. The modules are developed independently of each other and can therefore be tested separately. Modules are not stand-alone programs, so special software must be created to test these. Often, these are side-products of the implementation. Modules are tested against their design documents. Only the critical modules are tested in this project. These are defined in each phase separately. The end product of the module testing is the corresponding test report.

Integration testing seeks to combine the different modules in to a single subsystem and verify the correctness of their interoperability. The focus is on their interfaces and co-operation. Even if components work independently, new defects may be revealed when they are put together. Integration testing can be performed in two ways: by assembling all the modules together for execution, or by composing the system with a bottom-up approach. The latter, incremental way of organizing integration testing is preferred in this project, since it is easier to locate errors in smaller subsystems. Integration testing relies on the functional specification and technical documentation, concentrating on the architecture point of view. In this project, integration testing tests the SMOXY excluding clients and other external components, and the end product is a summarizing report.

At the system testing level, the whole system is tested. The entry criteria for it is that most of the software is already covered in testing of the previous phases. System testing looks after issues which may be overseen in the previous software-intensive testing levels. Typically, these consist of parameters like poor overall performance, capacity requirements, configuration testing and recovery capability in error situations. These are the mostly the non-functional requirements described in requirements specification, but part of them is also traceable in the functional specification. Going beyond SMOXY, the system testing includes external components, like the clients, to the scope. Executed system testing is one of the requirements of the acceptance testing.

In this project, the acceptance testing verifies the high level project goals. The weight is on the customer point of view. The scope of the acceptance testing was already discussed in the beginning of this document.

Black-box testing is preferred over white-box testing. The concept of black-box testing is capsulated with well-know functionality of the tested object. In other words, the system is started with some input x, the system's output f(x) is measured and compared against the predefined, specified output y. If f(x) = y, the system has passed the test.

If it is felt necessary, the project may perform white-box testing of certain key blocks of the code. The idea of going through the all the software at source code level is felt too exhausting effort with the current resources of the project team, so it will be done on selected areas if so decided later.

The white-box testing is partially covered by selected inspections of the source code. Code inspections are arranged for the critical parts of the software. These are identified in each phase separately.

4.2 Defect handling

• 1. Based on the requirements or design documentation, testing is planned
• 2. Test cases are created and validated
• 3. Execution of the test cases against the software
• 4. Test incident reports (bug reports) are created for each failed test case
• 5. Test report summary is presented
• 6. All test incident reports are allocated, evaluated and corrective actions are taken
• 7. All resolved test incident reports are regression tested. Since a correction may break other parts of the software, it a selected set of another tests may be executed.

The following template is used to report any irregularities while running system, integration or module test cases. The information needed to solve or abandon error report is enclosed together with the original report.

4.3 Resources and timing

Picture 2. Testing effort scheduling.

Starting from the bottom of the V-model, module testing is planned and performed by the persons who develop the corresponding source code. This is feasible because these are the persons who has most knowledge of the required functionality. On the other hand, it violates the basic assumption: testing must be done independently, since the programmers are too biased by their solutions and blind to their errors. However, in this project there are no resources to execute the external testing on modules. Like at every testing levels, it must be documented what was tested and what where the results. Module testing is started at the end of project phase T3, when first source code blocks are created, and it continues until the last component is finished in phase T4. In this project, module testing is the responsibility of developers of the modules.

Integration testing starts during the phase T4 when there is available module tested components. For this, the persons testing the software are different from those developing it, if the division of work allows that. It is very beneficial to cycle developers to test each other's code, since they have the needed experience of the structure of the software. Full time testers often lack this information, especially if the knowledge transfer has not fully taken place yet. The integration testing is an incremental process starting with the basic skeleton of the software and ending up with the full system. Therefore, it lasts until the end of the last implementation phase T4, and is performed by a person responsible for integration testing (to be named).

For system testing, the primary resources are the persons not so heavily involved with the source code development. By having a bit more external to point of view, they don't bear the weight of ownership. The use of testing resources outside the project could be achieved by temporarily exchanging persons with another project group. The feasibility of this idea is studied. System testing planning was started at the end of phase T2, while the actual system testing is performed on phase T4, with rescheduling overhead to phase LU. System testing is the responsibility of the system tester (to be named).

The acceptance testing involves also the customer in the delivery phase, LU. It is in the customer's interest to oversee the final testing rounds in order to familiarize himself with the end product and to evaluate the quality of the product and, more importantly, the quality of the project documentation. This includes estimating the deepness, level of research and value for the customer in documentation. In practice, the acceptance testing starts already at the end of phase T4 and ends few weeks before the end of the whole project.

At the end of the phase T2, the project presents the first demo of the software. A separate design document discusses also the testing aspects of the demo [DEMODOC].

The need for code inspections is evaluated in each phase separately. For more detailed resource allocation in testing, the project plan is revised in the beginning of each phase to describe the division of work.

5. Acceptance test cases

• Documented scenarios and use cases used in basis of design (tested by 5.1)
• Architecture design for mobile streaming proxy documented (tested by 5.2)
• Prototype of the architecture that shows that main technical ideas behind the design are valid (tested by 5.3)
• Structured and easily used documentation containing all the project documentation and especially rationale about the solutions that do or do not work. (tested by 5.4)

5.1 Scenarios and use cases

5.2 Architecture documentation

• Requirements Specification (LU-phase)
• Functional Specification (LU-phase)
• Technical Specification (LU-phase)
• Master Test Plan (LU-phase)
• Test report (LU-phase)
• User manual (LU-phase)

5.3 Prototype of the architecture

5.4 Organized documentation

6. System test cases

6.1 Introduction

The system test cases are defined in such a detail that it does not require a tremendous effort to execute them without deeper knowledge of how the SMOXY system performs. This is done bearing in mind the customer's need to verify the correctness of the system. Their second purpose is to facilitate the external testing effort performed by the opponent group in the beginning of the phase LU. System tests are human labor intensive due to the low level of automation.

System testing does not attempt to solve defects that are encountered in third-party software used in testing.

Installation is not tested due to the unavailability of implementation. This is scheduled for the last phase, LU. Therefore, the tester must be provided with a preinstalled version of SMOXY.
 

System test parameters
CLIENT	Netscape 4 or higher [browser]
SMOXY	[ip]
ADMIN_ACCOUNT	[id:passwd]
FRONTDOOR	[ip:port]
DIRECTUI	n/a [url]
PORTAL	n/a [url]
CONF_UTIL	[web pages]
USER ACCOUNT	[id:passwd]
LOGS	[collection of files]
SMALL_STREAM	[url]
MEDIUM_STREAM	[url]
LARGE_STREAM	[url]
COLOR_STREAM	[url]
INCORRECT_STREAM	[url]
AVI_STREAM	[url]

6.2 Test cases

7. Integration testing

7.1 Introduction

7.2 Test cases

The first set of integration tests the proprietary packet protocol between a Frontdoor and Smoxy Engine. This is a traditional client-server setup with Frontdoor and Engine, respectively.

The second set of tests verifies the functionality when Database is integrated with Frontdoor and Engine.

7.2.1 First set

The first set of test cases includes the testing Smoxy Engine, which consist of modules network.h, connection.h, parallel.h, stream.h. For this purpose, a test implementation of Frontdoor was created. The Frontdoor sends messages to Smoxy Engine with the packet protocol (PP). The purpose of the created FrontDoor is to test the strenght of the server side implemention of PP. Also the Smoxy's capability to handle parallel requests is tested. See chapter 4.1.1.1 in [TECHSPEC] for details.

7.2.2 Second set

Not available yet due to the pending implementation of Database in SMOXY.

8. Module testing

8.1 Introduction

8.2 Test cases

Testing is based on creating test programs using the functions defined in module header files. Both tested modules are used for communicating over TCP sockets and form the core of the SMOXY system. A straight-forward server implementation was created to act as a peer for both modules. With this setup, it was e.g. possible to test the transparency of the modules. This means, that when sending data to server that echoes all received data back, it can be verified that data remained unchanged.

The test cases are automated with scripts and presented in the Test Report [TESTREPORT] document.

9. Exit criteria

9.1 Acceptance test exit criteria

9.2 System test exit criteria

[1] There MUST NOT be system tests of priority "HIGH" failing (in chapter 6).

[2] All system test cases MUST be executed at least once in phase T4.

System testing is accepted by the project manager. This is enough, since one of the acceptance tests (approved by the customer) requires that the results of system testing are presented.

9.3 Integration test exit criteria

Phase T4: The exit criteria for integration testing is that 80% of test cases must be executed successfully. 

9.4 Module test exit criteria

Phase T4: The exit criteria for module testing is that 80% of test cases must be executed successfully.

10. Risks

10.1 Risk analysis

[2] Testing does not find critical defects.
Countermeasure: Use V-model which starts from the bottom to the up. Divide testing to manageable tasks. Communicate actively with other team members and try to identify critical parts. Use iterative approach.

[3] Testing tools are missing or testers don't know how to use them.
Countermeasure: Select tools in advance and arrange training.

[4] The test manager takes one month vacation (the risk has already materialized).
Countermeasure: The testing is rescheduled, and the testing effort in phase T4 is doubled.

10.2 Realized risks

Phase T3: The person responsible for testing (Eero Tuomikoski) was on a vacation the month of January 2001, therefore the schedule has been changed from the previous one presented at the end of phase T2. Due to this rescheduling the T4 phase is heavily loaded with testing.

Phase T4: The decision to automize all integratation and module test cases took more time than estimated in the beginning of the phase. This explains the relatively small total number of tests. Especially on integration testing, effort has been concentrated on issues which are expected to more likely fail than success. This explains the lack of test cases for functionalities, which are well-proofed to be correct. These are addressed again in phase LU. The second realized risk was that Database design and implementation was not available early enough to give time to prepare the test cases for that.

11. References