Targets of the this testing report are defined at SCHEMESTATION Testing Plan [Project Test Plan] and related documents that define behavior and semantics of the SCHEMESTATION system.
System version related to this testing report is SCHEMESTATION snapshot dated 240498.
The differences from the testing plan are:
Implemented test coverage is same as the coverage defined in the SCHEMESTATION Testing Plan.
For uniform method of generating test run output, special Test Report Utility is written. The test run outputs are not included in this report because of their low signal/noise ratio to the non-implementors.
Tests are run on a Linux machine equipped with Intel CPU unless stated otherwise. In general, a Unix-compatible system is assumed.
| Name | Done | Succesful | Total |
|---|---|---|---|
| Unit tests | 669 (99%) | 668 (99%) | 674 |
| Integration tests | 16 (100%) | 15 (94%) | 16 |
| System tests | 22 (100%) | 22 (100%) | 22 |
The number in the tables should not be interpreted blindly; number, categorisation and importance of tests per target vary. For more accurate information, see related documents. The precentage values represent the number of successful tests of the total tests performed. They give some measure of the progress, but not very much.
Unit test summaries are below. Integration Testing Plan, System test specification and Acceptance testing specification describe the results in other testing phases.
| Name | Done | Succesful | Total |
|---|---|---|---|
| The heap | 12 (92%) | 11 (84%) | 13 |
| The addressing system | 15 (93%) | 15 (93%) | 16 |
| The networking system | 38 (97%) | 38 (97%) | 39 |
| The messaging system | 10 (91%) | 10 (91%) | 11 |
| The external agent system | 26 (96%) | 26 (96%) | 27 |
| The virtual machine | 228 (100%) | 228 (100%) | 228 |
| The scheduler | 18 (100%) | 18 (100%) | 18 |
| Compiler | 322 (100%) | 322 (100%) | 322 |
The results are described in Heap Unit Testing Plan, Addressing System Unit Testing Plan, Networking Testing Plan, Messaging System Unit Testing Plan, External Agent Interface Testing Plan, VM Unit Testing and Scheduler Unit Testing.
The failing test in the heap module can be triggered only by external maliciously generated linearisations. I can be corrected relatively easily - but is considered low priority as the goals of the project are related to conceptual research, not developing a rock-solid OS.
The compiler testing is discussed at Compiler test report.
Integration tests are described in Integration Testing Plan. Integration tests are performed and passed. One test fails with malicious input (not possible to create such with normal SS system). This can be tracked down to the heap module.
System and acceptance tests are defined in System test specification and Acceptance testing specification. System and acceptance test is performed, and passed.
Interoperability tests are done, most visibly at the heap [Heap Unit Testing Plan] that give basis for integration testing [Integration Testing Plan] success. Interoperability tests are passed in all port target systems and between them.
Performance flaws were detected on the heap and VM tests. The easily correctable flaws that caused essential problems on implementation of running system are fixed. Improving the performance further requires drastic changes in the data structures, algorithms, and possibly even in the basic ideas. Some of these ideas are tested in practice, but not included in the default version.
Standard virtual machine performance in the development Linux system is represented below.
| Run type | VM instructions/s |
| Real-world performance (messaging+heap+vm) | 250.000 |
| Nonmessaging task (heap+vm) | 400.000 |
| Maximum performance (vm) | 1.000.000 |
These results meet the original requirement specification values.
Messaging performance is a slight disappointment. Typical values for 433 MHz Alpha system are 20 messages per second - no matter is the message couple of bytes or several megabytes. Some horrifying misfeature or specification flaw is considered to reside in heap or messaging module (probably the heap).
Migration performance is mostly limited by the messaging performance - and the messaging load of the agent.
All of the above performance measurements can vary very much with slight variations - for instance, 64-bit operations being used often favor 64-bit architectures and compilers that inline these operations in 32-bit architectures (which resulted 60% performance increase!). Compiling the system debugging disabled can double the speed of several modules.
System monitoring is performed by turning debug flags on the software and reading the debug output. No explicit measurements from this data is made.
One or more system monitoring areas can be activated selectively - every opcode run on the VM can be displayed, or messaging activity can be shown, for instance.
A sort of usability testing was performed by giving a late version of the system to a group of CS students with common interest on Scheme programming language. The domain of this testing was to get comments and ideas of the product from non-developer team. Because of lack of one "true" interface to the system (SS is after all, a whole OS with unlimited set of possible interaction methods), the results reflect the outsider opinions of the system in whole, not just the UI part.
There were following results:
Problems encountered in testing are:
Practically all tests are performed now (240498). Couple of tests that are not performed or do fail are suc that cause no harm in the actual running system - they are not considered critical in the current project. If project is continued with altered requirements, these tests must naturally be re-evaluated.
Bugs have been found in all units being tested, but probably most of the harm was caused by heap that had fatal bugs that gone undetected, and the messaging system that had a memory leak and some other hard-to-see bugs in it. Considering the project size and schedule, this sort of bugs can pose a major threat. Along with misunderstood nuances of the subsystem semantics these must be considered first-priority targets.
Lack of practically flawless test suites causes the project a moderate pain. It made tracking bugs harder because of incapability of locating all the bugs straight away, but it also saved the project from burden of writing a massive testing system; even with current tests, the test cases can exceed ten thousand rows of code.