This document describes the SCHEMESTATION system testing unit. The purpose of the system testing is to ensure that the SCHEMESTATION simulation works as expected as a whole. This is done by running applications on top of the environment provided by the SCHEMESTATION.
The Integration testing is used to reduce system testing. The following tests assume the relating integration tests to be successful to provide accurate system-wide results. As many system tests as possible are supposed to be written in (SCHEMESTATION) Scheme.
System startup and reboot is tested by starting the SCHEMESTATION simulator binary with and without a terminal. The startup must be possilbe to perform from UNIX shell prompt with the ss binary and the related options. After the startup, ss must be capable of interactive operation if started with terminal, but at least capable of running ss bytecode binaries, and interact with other SCHEMESTATION hosts on the network if specified. If the execution of the simulator is terminated in a controlled way, the state of the system must be able to be revived later from the persistent storage.
System shutdown is performed via terminal if one exists, and can be performed by the SIGTERM signal or special message from remote domain even if there is no interactive terminal. After shutdown, all agents and local message queues on the SCHEMESTATION domain should be stored in a persistent storage, and be capable of reviving with SCHEMESTATION startup procedure later.
Simple test runs in a running station are performed with at least the three following programs: Fibonacci number calculator, Quicksort implementation, and a simple interactive Tic-tac-toe game. Fibonacci and Quicksort tests are considered valid if the results match the precalculated results produced by non-SS system. Tic-tac-toe is considered valid if the test program is capable of interacting with the user, producing coherent interface to the user making moves and interpreting in-programmed Tic-tac-toe rules correctly.
All distributed test runs must be able to run as distributed to several domains.
Distributed tests include at least: distributed Fibonacci calculator and Game of Life implemented in a distributed manner. Fibonacci program must make at least part of the computation by sending "RPC" messages to other Fibonacci agents; GOL must be implemented using single cells consisting of single agents.
The results are compared with precalculated results. Crashing or halting of any kind is considered a failure, like in all other tests in the system test set.
Other messaging system oriented tests are implemented using special "messaging" agents. The agents are capable of sending messages periodically, or in a single-shot manner triggered by receiving a message from an another agent. These agents are set to communicate with each other in various modes (send-receive 1000 short messages, send-receive 1000 long messages, send-receive 1000 mixed-size messages), and preset life interval. To pass the test all messages must be received, with correct contents. The order of messages between two agents must be chronological, as specified in Operating system specification.
Migration capability is tested with the distributed test programs defined above (Fibonacci, Tic-tac-toe). The agents are migrated between the active domains randomly by a special agent. The results of the tests programs must be the same as above.
Persistence of the system is tested with the startup/shutdown tests, together with the Tic-tac-toe game (see above). The test procedure:
Object Servers that store the information on services and resources in the system are tested. They must be capable of retrieving, storing and modifying information. The results of the operations must be consistent; the correct behavior is defined in Operating system specification.
External agent interfaces are tested on a system-wide scale using programs that exploit their services. File server is an another test case. There is also a TCP-server, and to test that a simple HTTP server will be implemented. To pass the tests, external agent interfaces must provide consistent data flow both inward and outward; this can be tested by following results of changes in the filesystem caused by a FS agent, as well as FS changes seen from the SCHEMESTATION client.
The tests are performed on two platforms, Intel/Linux and Alpha/Digital UNIX.
Two types of tests are performed --- ones that prove system to be able to be ported to a platform (the system startup, shutdown and simple tests in single architecture) and the distributed interoperability tests (distributed Fibonacci, Tic-tac-toe migration) that prove that the systems can co-operate.
The performance tests can be performed using the test programs described in the "Testing distributed runs" section. The performance tests are performed in the following categories:
The performance testing in addition to all the other testing provides enough stress testing.
It is tested that one can get accurate monitoring information from the system; the monitoring is implemented using debugging information output by the simulator. The monitoring should be able to distinguish the work of:
The SCHEMESTATION simulator provides optionally also an X-window to monitor the performance factor (load) of the running SCHEMESTATION.
Test cases (first part: (D)one, (U)ndone, (I)ncomplete, second part: (S)uccess, (U)nsuccess, (I)ncomplete (F)atal):
| Name | Status | Result |
|---|---|---|
| System startup | D | S |
| System shutdown | D | S |
| Fibonacci | D | S |
| Quicksort | D | S |
| Tic-tac-toe | D | S |
| Distributed Fibonacci | D | S |
| Distributed Game Of Life | D | S |
| Migrated Tic-tac-toe | D | S |
| Migrated Fibonacci | D | S |
| Messaging Agents | D | S |
| Persistence test | D | S |
| Object server test | D | S |
| External File Server test | D | S |
| External TCP Server test | D | S |
| Portability test | D | S |
| Cross-platform test | D | S |
| Messaging performance | D | S |
| Migration performace | D | S |
| VM performance | D | S |
| Fibonacci performance | D | S |
| System monitoring (debug) | D | S |
| System monitoring (load) | D | S |
| Total done | 22/22 (100%) | 22/22 (100%) |